How to create a custom dynamic Z-Buffer

User Code & Projects
1

Hi dear jMonkeys,

I am working with Google’s Project Tango for my Bachelor Thesis. Now I want to use the point clouds it produces to update a Z-Buffer dynamically. Thereby I hope to occlude digital objects with real objects for better depth perception in Augmented Reality. After days of nearly giving up I managed to transform the point clouds correctly so they are displayed in JME3. My problem is, I don’t even know how to get started with that z-buffer thing. I guess it is not a good idea to use Open GL methods directly from JME3.

What I try to do is:

  • create new z-buffer
  • load z-buffer with z-values, calculated from distance of cloud point and camera plane
  • occlude 3d objects with z-buffer

Hope someone can help me, or at least, point me in the right direction.

Greetings,
Gem

2

to give you a better understanding of what I am trying to do, here is a screenshot of my app with the point cloud rendered as a point mesh:

Screenshot_2016-05-14-21-29-18.png1920×1200 335 KB

PS: I guess you can see my plant on the right ^^

3

I can’t offer anything useful, but I am very interested in messing with the Z buffer so I hope you make progress!

Also, [quote]After days of nearly giving up[/quote]

This is just brilliant, like a description of everything I’ve done with programming :smiley:

4

Just render the point cloud with depth and it will fill the z buffer. Use points sprites if you want the points to have a larger size. You may also want to sort them back to front but for square point sprites it shouldn’t matter. (Or you can set the point size larger I guess… never tried it.)

1 Like
5

or you could render your depth information to a texture, pass that to a simple custom shader and do your own occlusion, just don’t forget to love it.

6

Thanks, that sounds good. I just changed the point size to 5 and will try if it’s enough.

But I can’t figure out how to make the point cloud invisible after filling the z buffer. Somehow I must get noticed, when rendering to the z buffer is finished and then remove it from the scene and add it in the next render cycle again, or not? How can I do this. Can’t find anything in the wiki or somewhere else. Your help is very appreciated. :slight_smile:

Thanks,
Gem

7

Render it transparently.

8

I wrote a GeometryComparator:

public class PhantomOpaqueGeometryComparator extends OpaqueComparator {
    
    @Override
    public int compare(Geometry o1, Geometry o2) {
        
        if(o1.getName().equals("pointcloud") && o2.getName().equals("pointcloud")) {
            return 0;
        }
        
        if(o1.getName().equals("pointcloud") && !o2.getName().equals("pointcloud")) {
            return -1;
        }
        
        if(!o1.getName().equals("pointcloud") && o2.getName().equals("pointcloud")) {
            return 1;
        }
        
        return super.compare(o1, o2);
    }
}

Do I need it? I’ll do what you said and render it transparently. But in that case it won’t fill the z-buffer, or am I mistaken? I am really sorry for being so confused right now.

9

Is that correct?

private void updateMesh() {
        // occlusionNode is a child of rootNode

        Node meshNode = new Node();
        Vector3f [] vertices = pcu.vertices;    // pcu is the PointCloudUpdate
        
        Mesh mesh = new Mesh();
        mesh.setBuffer(Type.Position, 3, BufferUtils.createFloatBuffer(vertices));
        mesh.setMode(Mode.Points);
        mesh.setPointSize(5);
        mesh.updateBound();
        
        Material mat = new Material(assets, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", new ColorRGBA(0,0,0,0));
        mat.getAdditionalRenderState().setWireframe(true);
        mat.getAdditionalRenderState().setBlendMode(BlendMode.Alpha);
        mat.getAdditionalRenderState().setDepthWrite(true);
        Geometry meshGeo = new Geometry("pointcloud", mesh);
        meshGeo.setMaterial(mat);
        meshNode.attachChild(meshGeo);
        
        meshNode.setLocalRotation(pcu.referencePose.orientation.inverse());
        meshNode.setLocalTranslation(pcu.referencePose.place);
        
        this.occlusionNode.attachChild(meshNode);
        this.meshNodes.addLast(meshNode);
    }
10

Don’t do that… you shouldn’t need to.

…else it looks right to me.

1 Like
11

Thank you so much <3 occlusion seems to work now, but point size seems to have no effect at all. Even when making the points opaque I can see no difference.

12

Maybe it’s on the material now… I don’t know. I always used point sprites so I don’t know.

13

Alright, I’ll try to figure out how point sprites work, and then let you know.

14

Sorry, can’t get it to work. I use Particle as material now (for Point Sprites, correct?) But as soon as I add the texture, nothing is rendered anymore. Without the texture it renders green squares.

Mesh mesh = new Mesh();
        mesh.setMode(Mesh.Mode.Points);
        mesh.setPointSize(100);

        mesh.setBuffer(VertexBuffer.Type.Position, 3, BufferUtils.createFloatBuffer(lineVerticies));

        mesh.updateBound();
        mesh.updateCounts();

        Material mat = new Material(assetManager,
            "Common/MatDefs/Misc/Particle.j3md");
        mat.setBoolean("PointSprite", true);
        mat.setFloat("Quadratic", 10f);
        mat.setTexture("Texture", assetManager.loadTexture("Textures/invisible.png"));
        mat.getAdditionalRenderState().setDepthWrite(true);

        Geometry g = new Geometry("Point Cloud", mesh);
        g.setMaterial(mat);
        rootNode.attachChild(g);

Invisible.png is a completely transparent PNG.

15

Things you could try:

  • setting mat.getAdditionalRenderState().setColorWrite(false); so it doesn’t write to the color buffer (and getting rid of your texture)
  • setting a TextureCoord buffer, but you shouldn’t need to be using a texture at all, it’s simply not needed in this case
  • setting a Color buffer for the mesh, but be careful of [this] (jmonkeyengine/Particle.frag at master · jMonkeyEngine/jmonkeyengine · GitHub) (and getting rid of your texture)
  • waiting for @pspeed to get back and give you a proper answer
16

Unfortunately it didn’t work. But nevertheless thank you, for your answer.

17

when it’s rendering the green boxes, and you add mat.getAdditionalRenderState().setColorWrite(false);, what is the result? (no depth write / still seeing boxes / same as before)

did adding a color buffer help after removing the texture? does adding a colour buffer of blue, full opacity, change the colour of the boxes ?

have you tested using regular sprites opposed to point sprites ? depending on hardware, performance would be decreased, but issues should be reduced.

18

His issue is not hiding the render… it’s getting the points to render bigger. So it has nothing to do with setColorWrite(false).

Using the point cloud material requires more setup because it expects other buffers, I guess. Size and so on. You may have to dig into the particle emitter to see what other material parameters it sets. I’ve done it but I don’t remember exactly what I did.

The nice thing about point sprites is that you can make them bigger as they get closer to the eye (up to a point)… but there is more setup involved.

There was a test point cloud example that renders a cloud of smoke blobs. You could trace down through the code to see what it sets up on the mesh and material. I also remember some magic “Quadratic” float that controls how the particles shrink/grow with distance.

19

I used that “Quadratic” attribute in the code above. Is there a way to change the particles color. If I could change it to something transparent I might not need to use sprites.

20

Just tried it again and it seems to work correctly. I’ll let you know.