How to do outline with stencil buffer?

Hi. I’m trying to figure out how the whole stencil buffer thingy works, I read a few guides on the stencil buffer for openGL and got the gist of it, but I’m quite confused when it comes to using it in JME.

For example here’s one tutorial I found that would be a good starter for figuring out the stencil buffer: LearnOpenGL - Stencil testing
It’s pretty straight forward and I (mostly) understand the final code snippet. However I don’t understand how exactly I would replicate this in JME. Info on stencil buffers is almost non-existent in this forum, sadly.

I’m aware of the Material’s additional render state and how it works, I also have an idea of what each stencil related parameter there is for and how it translates into openGL. I guess I just don’t understand how to put it all together in JME to make stuff work.

What do I even need to set up a stencil buffer in the first place? Is “AppSettings.setStencilBits(8)” enough, or is there more to it? Pardon me for the lots of questions, but as I said info on the stencil buffer in JME is pretty much non-existent, and hopefully this thread can change that for good.

Jme’s stencil buffer implementation is basically a one to one mapping to opengl. With the exception that it does only support the separate version where front and backfacing pixels are tested with separate stencil functions.
As for the last question, i don’t know, since i only used stencil on a self build rendertarget.

What exactly can’t you convert to jme?

    /**
     * Enable stencil testing.
     *
     * <p>Stencil testing can be used to filter pixels according to the stencil
     * buffer. Objects can be rendered with some stencil operation to manipulate
     * the values in the stencil buffer, then, other objects can be rendered
     * to test against the values written previously.
     *
     * @param enabled Set to true to enable stencil functionality. If false
     * all other parameters are ignored.
     *
     * @param _frontStencilStencilFailOperation Sets the operation to occur when
     * a front-facing triangle fails the front stencil function.
     * @param _frontStencilDepthFailOperation Sets the operation to occur when
     * a front-facing triangle fails the depth test.
     * @param _frontStencilDepthPassOperation Set the operation to occur when
     * a front-facing triangle passes the depth test.
     * @param _backStencilStencilFailOperation Set the operation to occur when
     * a back-facing triangle fails the back stencil function.
     * @param _backStencilDepthFailOperation Set the operation to occur when
     * a back-facing triangle fails the depth test.
     * @param _backStencilDepthPassOperation Set the operation to occur when
     * a back-facing triangle passes the depth test.
     * @param _frontStencilFunction Set the test function for front-facing triangles.
     * @param _backStencilFunction Set the test function for back-facing triangles.
     */
    public void setStencil(boolean enabled,
            StencilOperation _frontStencilStencilFailOperation,
            StencilOperation _frontStencilDepthFailOperation,
            StencilOperation _frontStencilDepthPassOperation,
            StencilOperation _backStencilStencilFailOperation,
            StencilOperation _backStencilDepthFailOperation,
            StencilOperation _backStencilDepthPassOperation,
            TestFunction _frontStencilFunction,
            TestFunction _backStencilFunction)

this is basically the way to setup all the testing and stencil operations in one shot. In combination with setStencil*Front*Back*Mask and setStencil*Front*Back*Reference you have everything you need. Maybe the naming of the parameters used i a bit a poor:

GL_FRONT    sfail =   StencilOperation _frontStencilStencilFailOperation,
GL_FRONT  dpfail =   StencilOperation _frontStencilDepthFailOperation,
GL_FRONT  dppass =StencilOperation _frontStencilDepthPassOperation,
GL_BACK  sfail =        StencilOperation _backStencilStencilFailOperation,
GL_BACK  dpfail =     StencilOperation _backStencilDepthFailOperation,
GL_BACK  dppass =  StencilOperation _backStencilDepthPassOperation

If you do not want to use a specific action, just set the operation to KEEP.

1 Like

Hey, sorry if I made my question unclear, my main question was about how would I go about replicating the outline effect that was shown in that LearnOpenGL page I linked in the original post (this one). I think figuring out how to replicate it in JME would be a great (practical) introduction to the stencil buffer for noobs like me.

Of course to begin, I’d create the 5 geometries (like in the guide page) and add them to the root node - box 1, box 2, enlarged outline geom for box 1, for box 2, and the floor geometry. I suppose the next steps would be to set the stencil parameters for each of their materials (3 materials in total - 1 for the 2 boxes, 1 for the outlines of the 2 boxes, and 1 for the floor), which is done via their additional render states, but at that point I’m kind of lost.

Here’s my test app code in it’s current state (which fails to replicate that outline effect, as I’m also sure I did something wrong here)

public class TestStencilOutline extends SimpleApplication {
	public static void main(String[] args) {
		TestStencilOutline app = new TestStencilOutline();
		AppSettings settings = new AppSettings(true);
		settings.setStencilBits(8);
		app.setSettings(settings);
		app.setShowSettings(false);
		app.start();
	}

	@Override
	public void simpleInitApp() {
		flyCam.setMoveSpeed(24);
		flyCam.setZoomSpeed(-5);
		
		viewPort.setClearFlags(true, true, true);
		
		// Change default render state? As I suppose each material uses this render state by default. //
		
		RenderState.DEFAULT.setStencil(true,
				RenderState.StencilOperation.Keep, //front triangle fails stencil test
				RenderState.StencilOperation.Keep, //front triangle fails depth test
				RenderState.StencilOperation.Keep, //front triangle passes depth test
				RenderState.StencilOperation.Keep, //back triangle fails stencil test
				RenderState.StencilOperation.Keep, //back triangle fails depth test
        		RenderState.StencilOperation.Keep, //back triangle passes depth test
        		RenderState.TestFunction.Always, //front triangle stencil test function
        		RenderState.TestFunction.Always);    //back triangle stencil test function
		RenderState.DEFAULT.setFrontStencilMask(0);
		RenderState.DEFAULT.setBackStencilMask(0);
		RenderState.DEFAULT.setFrontStencilReference(1);
		RenderState.DEFAULT.setBackStencilReference(1);
		
		// Setup materials //
		
		Material boxMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
		boxMat.setTexture("ColorMap", assetManager.loadTexture("Common/Textures/MissingTexture.png"));
		boxMat.getAdditionalRenderState().setStencil(true,
                RenderState.StencilOperation.Keep, //front triangle fails stencil test
                RenderState.StencilOperation.Keep, //front triangle fails depth test
                RenderState.StencilOperation.Replace, //front triangle passes depth test
                RenderState.StencilOperation.Keep, //back triangle fails stencil test
                RenderState.StencilOperation.Keep, //back triangle fails depth test
                RenderState.StencilOperation.Replace, //back triangle passes depth test
                RenderState.TestFunction.Always, //front triangle stencil test function
                RenderState.TestFunction.Always);    //back triangle stencil test function
		boxMat.getAdditionalRenderState().setFrontStencilMask(-1);
		boxMat.getAdditionalRenderState().setBackStencilMask(-1);
		boxMat.getAdditionalRenderState().setFrontStencilReference(1);
		boxMat.getAdditionalRenderState().setBackStencilReference(1);
		
		Material boxOutlineMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
		boxOutlineMat.setColor("Color", ColorRGBA.Yellow);
		boxOutlineMat.getAdditionalRenderState().setDepthTest(false);
		boxOutlineMat.getAdditionalRenderState().setStencil(true,
                RenderState.StencilOperation.Keep, //front triangle fails stencil test
                RenderState.StencilOperation.Keep, //front triangle fails depth test
                RenderState.StencilOperation.Replace, //front triangle passes depth test
                RenderState.StencilOperation.Keep, //back triangle fails stencil test
                RenderState.StencilOperation.Keep, //back triangle fails depth test
                RenderState.StencilOperation.Replace, //back triangle passes depth test
                RenderState.TestFunction.NotEqual, //front triangle stencil test function
                RenderState.TestFunction.NotEqual);    //back triangle stencil test function
		boxOutlineMat.getAdditionalRenderState().setFrontStencilMask(0);
		boxOutlineMat.getAdditionalRenderState().setBackStencilMask(0);
		boxOutlineMat.getAdditionalRenderState().setFrontStencilReference(1);
		boxOutlineMat.getAdditionalRenderState().setBackStencilReference(1);
		
		Material floorMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
		floorMat.setTexture("ColorMap", assetManager.loadTexture("Common/Textures/MissingTexture.png"));
		
		// Create geometries //
		
		Geometry floor = new Geometry("Floor", new Box(10f, 0, 10f));
		floor.setMaterial(floorMat);
		rootNode.attachChild(floor);
		
		Geometry box1 = new Geometry("Box1", new Box(0.5f, 0.5f, 0.5f));
		box1.setLocalTranslation(3, 1.5f, 0);
		box1.setLocalScale(3);
		box1.setMaterial(boxMat);
		Geometry box2 = new Geometry("Box2", new Box(0.5f, 0.5f, 0.5f));
		box2.setLocalTranslation(-3, 1.5f, 0);
		box2.setLocalScale(3);
		box2.setMaterial(boxMat);
		
		rootNode.attachChild(box1);
		rootNode.attachChild(box2);
		
		// Create outlines //
		
		Geometry box1OUTLINE = new Geometry("Box1Outline", new Box(0.5f, 0.5f, 0.5f));
		box1OUTLINE.setLocalTransform(box1.getLocalTransform());
		box1OUTLINE.setLocalScale(3.2f);
		box1OUTLINE.setMaterial(boxOutlineMat);
		Geometry box2OUTLINE = new Geometry("Box2Outline", new Box(0.5f, 0.5f, 0.5f));
		box2OUTLINE.setLocalTransform(box2.getLocalTransform());
		box2OUTLINE.setLocalScale(3.2f);
		box2OUTLINE.setMaterial(boxOutlineMat);
		
		rootNode.attachChild(box1OUTLINE);
		rootNode.attachChild(box2OUTLINE);
	}
}

I would really appreciate it if you could please provide some tips to get it working like in that guide. Thank you.

As a rule in all of Java everywhere (but especially JME), all caps fields are ‘constants’ and since Java does not really support constants, it is up to the developer to have the discipline not to call setters and things on them.

You can really really mess up JME this way.

So I don’t know what the final solution will look like but it will not be corrupting the JME constants like that.

1 Like

There is still something strange going on. I had to bypass the jme’s renderstate system and use the opengl commands directly. I have to investigate with the opengl debugger to see what is going on. I can’t say when i have time to dig deeper into this.

Ok, here is a control and geometry comparator based approach:

Screenshot 2024-10-27 233248

The raw opengl hacks are still in there.

like 99.9% sure there is a bug in jme. reporting back

4 Likes

Tracked down and fixed in: Stencil buffer fix by zzuegg · Pull Request #2325 · jMonkeyEngine/jmonkeyengine · GitHub

Screenshot 2024-10-28 102503

7 Likes

Where is the link to the example?

It is part of the pull request, added to the jme3 examples module

Note: this is only a demo showcase, it works that nicely because the boxes have the origin at the center. For large part of models this is not true, so the basic scaling trick will produce different results. One possible solution is to displace the vertices based on the normals.

It works quite nicely in a rts style of game, because you can use the same technique to show units behind object, and you have also enough control to support things like enemy units should always be fully outlined or things like this

3 Likes

Ah, I had a feeling there was some sort of bug going on, to be honest I secretly was playing around with stencil related stuff earlier, what confused the crap out of me is the mask and reference parameters in particular doing absolutely nothing, so I thought instead of embarrassing myself on the forum, first I’d just try dialing it back a bit and try a simple stencil outline effect, and still couldn’t even do that, so I came here for help. Glad to see this will be fixed in the… next version of JME 3.7 I suppose? I’m not too familiar with how this github stuff works. Thank you zzuegg!

1 Like

If you need stencil now i would use a copy of the master. The current release is still fresh and there is no schedule for the next i think.

Gradle install adds it automatically to your local maven and then you have to add mavenLocal() as repository.

Building the engine takes longer then everything else.

3 Likes