How to find the mesh info from the camera view?

General Discussion
1

Hi. I’m working on a computer vision app. I’m using the TestRendererToMemory code to render a mesh offscreen but I also want to figure out where the mesh is rendered. Blender has an output mode where it can render the face ids to a buffer. I see that jme has the following for getting the depth:
offBuffer.setDepthTarget(FrameBufferTarget.newTarget(Format.Depth));
Is there way to write the mesh info to a buffer?

I guess as a last resort I could read the 3d points from the mesh data and use the camara’s projection matrix to get the rendered location but seems like a bit of work.

Thanks!
(PS - I can’t use blender because the python logic locks up the UI and it tends to crash a lot. I’m hoping jme might be better in this regard).

2

Do you mean you are looking for the location in the scene that the mesh is located?

If so then you can use the getCenter() method on the mesh’s world bounds. Something like mesh.getBound().getCenter() or geometry.getWorldBound().getCenter()

Otherwise I don’t quite know if I understand the question entirely, so more information could be useful if I’ve interpreted your question wrong.

3

Sorry for the confusion. I will try to clarify.

When there are thousands triangles in the scene and I use the jme3test TestRendererToMemory code to create a screenshot of them, I want to know which pixels in the screenshot belongs to which triangle. Blender has a output render mode in which you can ask it to create a special image in which each “pixel” location has the id of the triangle that was used to draw that pixel.

To compute this myself, I would need to start with the mesh’s triangles and then use the camera’s projection matrix to figure out where they would end up in the screenshot but I would also have to do a lot of additional work to handle occlusion. I was hoping there was a way to get jme3 to save the id of the triangle that is visible in the screenshot since it knows this when it decides which triangle to use to compute the color of each pixel.

4

ok, now its explained better.

tho Still dont see full picture why you need to know it. Why you need figure out where they would end up in the screenshot, why you need all of it.

Myself idk, but maybe JME optimizer had this, because as i remember there was tris/pixel based thing.

But again, if we would know why you need it, there is big chance we provide much easier solution for what you need.

5

I don’t know what the face id should be.
I dont think it is possible with stock jme materials.

Bind a your own framebuffer in a sceneprocessor, use a integer, or floating point texture. use your own material and write whatever id you want in the target texture.