I have 3x3 rotation matrix: Matrix3f rotation = readFromFile();
which I can apply to my geometry in order to rotate my model, like this: geometry.setLocalRotation(rotation);

However, what I would like to do, is calculate the yaw, pitch, and roll values of the rotation.
Is there an easy way to do that in jMonkey?

If you want to calculate it by hand, you can do so by calculating the angle between the axes of the reference frame and the axes of the axes of the body frame. Keep in mind that the sequence of calculation matters, therefore it is not possible to provide a generic solution, just an example (see below)

public double[] getYawPitchRoll(CartesianFrame referenceFrame)
{

// Define the standard reference frame
if (referenceFrame==null)
referenceFrame=new CartesianFrame();

// x1 is the reference X-axis after the first rotation, it is on the xy-plane of
// the reference frame.
Vector3d x1=
referenceFrame.getXAxis().mult(bodyAxes.getXAxis().dot(referenceFrame.getXAxis())).add(
referenceFrame.getYAxis().mult(bodyAxes.getXAxis().dot(referenceFrame.getYAxis())));

x1.normalizeLocal();

// y12 is the reference Y-axis after applying the first and second rotation
// (which doesnâ€™t change in between)
Vector3d y12=referenceFrame.getZAxis().cross(x1);
y12.normalizeLocal();

Mathematically, as seen above, there are numerous ways to do itâ€¦ but for the very most common reason people WANT to do this: the answer is impossible. (And there are other solutions that start earlier in your process.)

So please explain what you are actually trying to do.

Sorry for being so quiet last two days. I was trying to gain a better understanding of the source of the issue Iâ€™m solving here.

Iâ€™m rendering an old game. All the game data: like 3D models (.obj files), positions of objects in game world - seem to be stored in left-hand coordinate system, while jMonkeyEngine uses right-hand coordinate system. Or the other way around.

At first, I thought I could easily handle this by for example negating the Z coordinate. Now Iâ€™m loading objects (some 3D models, like tree or bench) into the game world by reading game files which contain the [x,y,z] coordinates of the objects. The problem is, that the data also contains rotation information in form of rotation matrix, as some of the objects are rotated/shifted, etc.
So I was trying to somehow convert the rotation matrix from the old coordinate system, to the new one (the one jME uses). My idea was to split the rotation matrix into yaw,pitch,roll values, negate one of them, and then use it to rotate the object. Iâ€™m not sure if that would work even if I managed to calculate the three angles properly.
I donâ€™t have knowledge of how was the rotation matrix defined in the original game, what was the order of rotations, was it clockwise or counter clockwise, etc.

Converting a rotation matrix from left hand to right hand can actually be pretty easy.

â€¦the rest of the data, I donâ€™t know. Sometimes it will also flip the polygon winding.

These are solvable problems that do not require conversion to (imperfect) yaw/pitch/roll.

The other thing that might catch you is if the rotation matrix if column major versus row major.

Whether row major or column major the (rows or columns respectively) represent the three coordinate axes (x, y, z) of that rotation. So changing handedness can be as easy as flipping the signs of one of the columns/rows.

var oldToNew = new Matrix3f();
oldToNew.loadIdentity();
oldToNew.set(2, 2, -1f);
var newToOld = oldToNew.invert();
holder.setLocalRotation(oldToNew.mult(rot).mult(newToOld));

where rot is the rotation matrix stored in game files.
I think it basically does what you said: one column gets multiplied by -1. Although I do the conversion to original system, rotate, then convert back to current system.

Iâ€™m 95% sure it is OK now, I will have to run the original game tomorrow and perform detailed comparison.

Another thing is that Iâ€™m having issues with texture mapping. Iâ€™m not sure if Iâ€™m suppose to do something about the mapping, in case I negated the Z coordinate of the modelsâ€¦

Yeah, that is similar that we have to do in OpenKeeper. We just modify the matrix columns, multiply with -1. Shame you didnâ€™t start with this information

yeaâ€¦
I didnâ€™t realise that using different coordinate systems is even a thing I just thought the game Iâ€™m dealing with is being weird in its own way by storing values in opposite order [z,y,x] instead of [x,y,z] (as swapping x with z also solves the issue, in a way).

Dungeon Keeper 2 models are in DirectX coordinate system (maybe, canâ€™t remember anymore). Some values in different order and some in opposite coordinates.

Beyond toniheleâ€™s answer (which is probably right), we would have to know what â€śissuesâ€ť is.

Texture coordinates should not be affected by the vertex coordinate system. So the most likely cause is that the texture is flipped upside down which is easily remedied on load of the texture.

For video game development, a picture is worth a bajillion-kajillion-mahjillion words.

I could probably diagnose a thousand different problems with only two images â€śThis is what it looks likeâ€ť and â€śthis is what it should look likeâ€ť.