When I add a filter to my FilterPostProcessor, the stencil buffer gets clobbered. Poking about in the source for FilterPostProcessor, it looks like the variable “depthFormat” is used to set up the frame buffer it renders the scene to. This variable is initialized to Format.Depth at declaration and never reassigned. You can change the color buffer format with “setFrameBufferFormat” but there is no corresponding method for the depth buffer. I think this means that it is impossible to use the stencil buffer and FilterPostProcessor at the same time (at least not without reflection ). I am able to get what I want by doing this before adding the filter to the FilterPostProcessor (called fpp here):
try{
Field depthFormatField = FilterPostProcessor.class.getDeclaredField("depthFormat");
depthFormatField.setAccessible(true);
depthFormatField.set(fpp, Image.Format.Depth24Stencil8);
}catch(Exception e){}
I think this may be a bug. Am I missing something here? JME’s source goes over my head lol
Edit: Should mention I’m using version 3.3.0-stable.
Yep, that’s what I’m after. I just can’t see a way to do it through the API and I’m thinking that’s an oversight.
Also, I’ve made a simple minimum reproducible example, so I’ll put that here I guess. I’m using the stencil buffer to give my characters nice looking drop shadows like in Mario 3d land:
It works until you press left click to add a ColorOverlayFilter, then the stencil operations fail. If you press right click to remove the filter, it works again. Uncommenting the reflection hack makes it work as expected.
package mygame;
import com.jme3.app.SimpleApplication;
import com.jme3.light.AmbientLight;
import com.jme3.light.DirectionalLight;
import com.jme3.material.Material;
import com.jme3.material.RenderState;
import com.jme3.math.ColorRGBA;
import com.jme3.math.FastMath;
import com.jme3.math.Quaternion;
import com.jme3.math.Vector3f;
import com.jme3.post.FilterPostProcessor;
import com.jme3.post.filters.ColorOverlayFilter;
import com.jme3.renderer.queue.RenderQueue;
import com.jme3.renderer.queue.RenderQueue.Bucket;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.scene.shape.Cylinder;
import com.jme3.scene.shape.Quad;
import com.jme3.scene.shape.Sphere;
import com.jme3.system.AppSettings;
import com.jme3.texture.Image.Format;
import java.lang.reflect.Field;
import org.lwjgl.input.Mouse;
public class StencilProblemExample extends SimpleApplication
{
FilterPostProcessor fpp;
Geometry shadowVolume;
Geometry quad;
public static void main(String[] args)
{
AppSettings settings = new AppSettings(true);
settings.setStencilBits(8);
StencilProblemExample app = new StencilProblemExample();
app.setShowSettings(false);
app.setSettings(settings);
app.start();
}
@Override
public void simpleInitApp()
{
this.flyCam.setMoveSpeed(100);
cam.setLocation(new Vector3f(0, 15, 10));
cam.setRotation(new Quaternion().fromAngles(FastMath.DEG_TO_RAD * 60, FastMath.PI, 0));
fpp = new FilterPostProcessor(assetManager);
getViewPort().addProcessor(fpp);
//reflection hack to change depth format used for fpp's internal framebuffer
/*try
{
Field depthFormatField = FilterPostProcessor.class.getDeclaredField("depthFormat");
depthFormatField.setAccessible(true);
depthFormatField.set(fpp, Format.Depth24Stencil8);
} catch(Exception e)
{
e.printStackTrace();
}*/
//add some background geometry and lights
Material mat = new Material(assetManager, "Common/MatDefs/Light/Lighting.j3md");
Geometry sphere = new Geometry("Sphere", new Sphere(32, 32, 3));
sphere.setMaterial(mat);
sphere.setLocalTranslation(2, 0, 0);
rootNode.attachChild(sphere);
Geometry box = new Geometry("Box", new Box(2, 2, 2));
box.setMaterial(mat);
box.setLocalTranslation(-2, 0, 0);
box.setLocalRotation(new Quaternion().fromAngles(1, 1, 1));
rootNode.attachChild(box);
AmbientLight light = new AmbientLight();
light.setColor(new ColorRGBA(0.1f, 0.1f, 0.1f, 1f));
rootNode.addLight(light);
DirectionalLight light2 = new DirectionalLight();
rootNode.addLight(light2);
//add a shadow volume
Material shadowMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
shadowMat.getAdditionalRenderState().setColorWrite(false);
shadowMat.getAdditionalRenderState().setDepthWrite(false);
shadowMat.getAdditionalRenderState().setFaceCullMode(RenderState.FaceCullMode.Off);
shadowMat.getAdditionalRenderState().setStencil(
true,
RenderState.StencilOperation.Keep, //front triangle fails stencil test
RenderState.StencilOperation.DecrementWrap, //front triangle fails depth test
RenderState.StencilOperation.Keep, //front triangle passes depth test
RenderState.StencilOperation.Keep, //back triangle fails stencil test
RenderState.StencilOperation.IncrementWrap, //back triangle fails depth test
RenderState.StencilOperation.Keep, //back triangle passes depth test
RenderState.TestFunction.Always, //front triangle stencil test function
RenderState.TestFunction.Always); //back triangle stencil test function
shadowVolume = new Geometry("Shadow", new Cylinder(2, 32, 1, 100, true));
shadowVolume.setMaterial(shadowMat);
shadowVolume.setLocalRotation(new Quaternion().fromAngles(FastMath.HALF_PI, 0, 0));
shadowVolume.setQueueBucket(RenderQueue.Bucket.Transparent);
rootNode.attachChild(shadowVolume);
//add a quad to look at the volume through
quad = new Geometry("Quad", new Quad(5, 5));
Material quadMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
quadMat.setColor("Color", new ColorRGBA(0, 0, 0, 0.5f));
quadMat.getAdditionalRenderState().setBlendMode(RenderState.BlendMode.Alpha);
quadMat.getAdditionalRenderState().setStencil(
true,
RenderState.StencilOperation.Keep, //front triangle fails stencil test
RenderState.StencilOperation.Keep, //front triangle fails depth test
RenderState.StencilOperation.Keep, //front triangle passes depth test
RenderState.StencilOperation.Keep, //back triangle fails stencil test
RenderState.StencilOperation.Keep, //back triangle fails depth test
RenderState.StencilOperation.Keep, //back triangle passes depth test
RenderState.TestFunction.Less, //front triangle stencil test function
RenderState.TestFunction.Never); //back triangle stencil test function
quad.setMaterial(quadMat);
quad.setQueueBucket(Bucket.Transparent);
rootNode.attachChild(quad);
}
@Override
public void simpleUpdate(float tpf)
{
//add and remove filter with left and right mouse
if(Mouse.isButtonDown(0) && fpp.getFilterList().isEmpty())
fpp.addFilter(new ColorOverlayFilter(new ColorRGBA(0.9f, 1, 1, 1)));
if(Mouse.isButtonDown(1))
fpp.removeAllFilters();
//move the shadow volume about
float time = getTimer().getTimeInSeconds();
shadowVolume.setLocalTranslation(new Vector3f(FastMath.sin(time) * 2, 0, 0));
//place the quad in front of camera so we always look through it
quad.setLocalTranslation(cam.getLocation().add(cam.getDirection().mult(1.5f)).add(cam.getLeft()).subtract(cam.getUp()));
quad.setLocalRotation(cam.getRotation().mult(new Quaternion().fromAngles(0, FastMath.PI, 0)));
}
}
Seeing that is declared as a final field seems there was an intention(?) But anyway, I think it won’t harm to add a setter for it if that helps! Please, feel free to submit a PR if you are willing.
Edit:
And by the way, for a drop shadow filter you can also take use of this:
Hmm that’s interesting. In the 3.3.0 source, it isn’t final, but in the current version it is. I wonder then if there’s a reason not to have a setter. I’ve found this thread which indicates the setFrameBufferFormat function was contributed through a PR, so there’s a precedent: Ability to change image format for FilterPostProcessor
I’ve never made a PR before, so I’ll have to read up on that. If it gets rejected, maybe at least the reason will shed some light on this topic.
EDIT: I have seen pspeed’s filter in Mythruna, and I think the stencil-based volumetric shadows fit the look I’m going for a bit better. I particularly like how they “flow” all the way down walls instead of wrapping over the edge.
Yeah, it’s a trade off and partially a matter of style. The crisp edges in yours probably fit your style better.
The tradeoff is that I suspect both approaches are not blocked by geometry. (Mine certainly isn’t.) And in the case where Mythruna’s filter was designed, there could be multiple layers of geometry so it’s important to have some kind of pretty rapid fall-off.
Another double-edged sword for Mythruna’s approach is that it’s self-shadowing so the object looks like it’s casting shadows on itself… which was exactly the effect I was going for but also may not be right for some styles.
(Edit: another down side of my drop shadow filter is that the shadows disappear when you are inside of them. I’ve always meant to go back and try to fix that somehow but it has to do with the front-face vs back-face of the box I render for each shadow volume.)
). I am able to get what I want by doing this before adding the filter to the FilterPostProcessor (called fpp here):