Selective Post-Process Outlining - Exclude Objects from CartoonEdgeFilter

Troubleshooting graphics
1

Hello everyone,

I’m trying to achieve a selective outlining effect where a post-processing filter is applied to a subset of objects in the scene, but not to all of them. The goal is to have a “cartoon edge” outline on certain items (e.g., selectable objects) while leaving other parts of the scene (e.g., the environment) untouched.

I’ve been looking at the CartoonEdgeFilter , but it seems to apply the effect globally to everything rendered by the main viewport.

Here are some examples of the effect I’m trying to replicate from Unity:

My question is: What is the recommended or most effective way to handle this in jMonkeyEngine 3?

  • Is there a way to tell the CartoonEdgeFilter to ignore specific objects?
  • Should I be using a separate ViewPort that only renders the outlined objects, and then apply the filter to that specific viewport? If so, what is the best practice for setting up and blending this secondary viewport with the main scene?

Any guidance or a code example would be greatly appreciated. Thank you!

2

Nope. CartoonEdgePass as is can’t distinguish between geometries after they’ve been rendered.

Yes, that should work. I have not tried doing it before, but it should work something like this:

ViewPort cartoonView = renderManager.createPreView("CartoonView", viewPort.getCamera());
cartoonView.setOutputFrameBuffer(viewPort.getOutputFrameBuffer());
cartoonView.setBackgroundColor(viewPort.getBackgroundColor());
cartoonView.setClearFlags(true, true, true);
FilterPostProcessor fpp = new FilterPostProcessor(assetManager);
cartoonView.addProcessor(fpp);
fpp.addFilter(new CartoonEdgeFilter());

viewPort.setClearFlags(false, false, false);

Geometries you want rendered with the effect are attached to cartoonView instead of viewPort.

The main drawback is that you can’t safely have a node containing both geometries with and geometries without the effect (just looking at the source code, geometries attached to both will have runControlRender called twice, and you’ll have to be careful when updating those geometries).

Another option is to render a mask to determine where in screenspace the post-processing is applied. CartoonEdgeFilter already renders the scene to get the normals, so you can just add the mask texture as another color target for that render.

@Override
protected void postQueue(RenderQueue queue) {
    Texture2D maskTexture = ...
    FrameBuffer fbo = normalPass.getRenderFrameBuffer();
    if (fbo.getNumColorTargets() < 2 || fbo.getColorTarget(1).getTexture() != maskTexture) {
        if (fbo.getNumColorTargets() < 2) {
            fbo.addColorTarget(FrameBuffer.FrameBufferTarget.newTarget(maskTexture));
        } else {
            fbo.replaceColorTarget(1, FrameBuffer.FrameBufferTarget.newTarget(maskTexture));
        }
        fbo.setMultiTarget(true);
        fbo.setUpdateNeeded();
    }
    Renderer r = renderManager.getRenderer();
    r.setFrameBuffer(fbo);
    renderManager.getRenderer().clearBuffers(true, true, true);
    renderManager.setForcedTechnique("PreNormalPass");
    renderManager.renderViewPortQueues(viewPort, false);
    renderManager.setForcedTechnique(null);
    renderManager.getRenderer().setFrameBuffer(viewPort.getOutputFrameBuffer());
}

You’ll have to change all the “PreNormalPass” shaders you use to write to the mask texture.

1 Like
3

How to do outline with stencil buffer? - General Discussion - jMonkeyEngine Hub

The hard part in any solution is how to get the contour. Mostly because you have no information in the buffers outside of the object itself. a inline / mask color, or whatever you want inside the geometry itself is much easier because it does not suffer from these limitations

1 Like
4

Thank you it worked but with another trick
I modified normal.frag for Technique PreNormalPass like this

gl_FragColor = vec4(normal.xy* 0.5 + 0.5,-normal.z* 0.5 + 0.5, m_OutlineMask);

Then I used the value to exclude the coloring

2 Likes