I have been straining myself to think of a way to limit the camera so that my player(a node attached to a FirstPersonCamera) cannot do backflips or run around on his head. I have tried getting and setting a direction if the y value is greater than a number, that didn't do anything. I have tried to wrap my mind around figuring out a way to use the up-vector but that never panned out. I am sure there is an easy way to do this since almost EVERY game that is first person would need to have this feature… 
I have actually thinking about this myself, but havent had time to do it.
I think you need to compare the angle between two vectors not between coordinates: for example the angle between players direction and projection of the camera direction vector to a specific axis of players local coordinate system. I hope you understand what I mean.
There probably is a methond in Vector3f class that gives you the angle between 2 vectors (Vector3f v; v.angleBetween(Vector3f), but you can also do it manually:
cos(a) = uv/(|u||v|)
and arccos(uv/(|u||v|)) should give you the angle.
Where u is the direction of the player and v is projection of cameras direction to a plane that is paralel to the players direction vector.
When you want to restrict the up and down angle then you need to project the camera direction vector to the players yz plane and get the angle between it and players getRotation().getRotationColumn(2). When you want to restrict the left and right angle then you need to project the camera direction vector to the players xz plane. And I think when you want to restrict cameras "roll" dunno if its the right term, then u need to project the cameras directon to the players xy plane.
When you get this implemented please let me know how u did it so I wouldn't have to figure it out myself 
since I'm not sure how to project 2 vectors on the same plane. Argh… shouldn't have skipped my linear algebra lessons. :(. But it shouldnt be too difficult. I think in a few hours I could figure it out.
Perhaps there already exists a method to project a vector to specific plane?
This has already been done here http://www.jmonkeyengine.com/jmeforum/index.php?topic=10502.0;topicseen There may be other links on the forum, check around.