How do I stop the camera from becoming upside-down when the camera is rotated up or down too far?
You can cap the vertical rotation fairly simply by checking the Y value of the camera’s normalized viewDirection, and then add some code that checks and prevents it from passing those max/min values every frame.
In my project, I prevent the camera from looking up once it has a viewDirection with a Y Value near 0.9, and I prevent the player from looking down any further when the values around 0.15, but you may want to adjust these values to first your use case.
Or don’t use FlyByCamera since it’s really only suitable for demos/tests in the long run.
My SiO2 library has a MovementState which by default mostly acts like fly cam but clams pitch to +/- 90 degrees:
…with the added benefit of being able to swap out the handlers if you want to move a char control or a game object directly.
If you are using a Chase Camera, you can set maxVerticalRotation
(and min). you can also inspect its code to see how they handle that.