I’m encountering a perplexing issue when trying to combine a custom CartoonEdgeFilter with the WaterFilter in my jME3 application. My custom CartoonEdgeFilter is based on the standard OutlineFilter but includes modifications to make the outline thinner as the camera moves further away from the object.
Here’s the problem: I’m facing a conflict related to how these post-processing filters interact with the depth buffer, specifically depending on their order in the FilterPostProcessor.
My Custom CartoonEdgeFilter (Shader Modifications):
I’ve added the following lines to the Common/MatDefs/Post/CartoonEdge.frag shader to achieve the distance-based outline thinning:
float rawDepth = texture2D(m_DepthTexture, texCoord).r;
// Linearize the depth to get the actual view-space distance
// This formula converts non-linear depth [0,1] to linear view-space Z
float currentPixelDistance = m_NearClip * m_FarClip / (m_FarClip + rawDepth * (m_NearClip - m_FarClip));
// Calculate dynamic line thickness based on distance
// 'thicknessLerpFactor' goes from 0.0 (at or before FadeStartDistance) to 1.0 (at or after FadeEndDistance)
float thicknessLerpFactor = smoothstep(m_FadeStartDistance, m_FadeEndDistance, currentPixelDistance);
// Interpolate line width: from max (near) to min (far)
float dynamicEdgeWidth = mix(m_MaxEdgeWidth, m_MinEdgeWidth, thicknessLerpFactor);
// Ensure fadeFactor doesn't go below 0 (though smoothstep should handle this)
float fadeFactor = 1.0 - thicknessLerpFactor;
fadeFactor = max(0.0, fadeFactor);
vec2 edgeOffset = vec2(dynamicEdgeWidth) * g_ResolutionInverse;
// ... (rest of the shader code using edgeOffset for sampling)
// Apply the fade factor to the edge intensity
edgeAmount *= fadeFactor;
The Conflict Scenarios:
If CartoonEdgeFilter is added BEFORE WaterFilter in the FilterPostProcessor:
The outline effect is not drawn at all. It seems like the m_DepthTexture or m_NormalsTexture it’s receiving is not correctly populated, perhaps being cleared or overridden before it can be used by my outline shader.
If WaterFilter is added BEFORE CartoonEdgeFilter in the FilterPostProcessor:
The CartoonEdgeFilter works, but the water effect is drawn on top of other objects (e.g., models that should be partially submerged or behind the water), indicating a depth buffer issue where the water isn’t respecting the depth of scene geometry.
This suggests a fundamental conflict in how the depth buffer is being used or accessed by these two filters, especially with my modification to the outline shader. The original OutlineFilter (without my distance fade logic) doesn’t seem to exhibit this specific conflict.
My Question:
How can I correctly integrate my custom CartoonEdgeFilter (with distance-based thinning) and the WaterFilter in jMonkeyEngine 3 without these rendering conflicts?
Is there a specific way to manage the depth buffer or render passes when using multiple post-processing filters that rely on depth?
Are there common pitfalls when modifying shaders that sample m_DepthTexture in the context of other transparent filters?
Any insights or suggestions on how to debug or restructure the filter pipeline would be greatly appreciated!
Thanks for the quick response and the excellent suggestion! You’re absolutely right, narrowing down the differences is key.
Regarding your question about the differences between the working and non-working filter, I’ve done some more testing based on that idea:
The “working filter” in my initial post refers to the standard OutlineFilter (without any of my distance-fade modifications). When I use that filter instead of my custom one, the water renders correctly when WaterFilter is first, and the OutlineFilter works when it’s first (though obviously without the distance fade).
The “not working filter” is my CartoonEdgeFilter with the distance fade logic added to CartoonEdge.frag as described.
Here’s what I’ve observed specifically:
Original OutlineFilter vs. My Modified CartoonEdgeFilter:
The only changes are those I listed in the fragment shader for CartoonEdge.frag:
Sampling m_DepthTexture to get rawDepth.
Linearizing depth to currentPixelDistance.
Calculating thicknessLerpFactor and dynamicEdgeWidth using m_FadeStartDistance, m_FadeEndDistance, m_MaxEdgeWidth, m_MinEdgeWidth.
Applying fadeFactor to edgeAmount.
Adjusting edgeOffset with dynamicEdgeWidth.
It seems to be specifically the reading and use of m_DepthTexture within my shader modification that causes the conflict. If I comment out only the lines related to m_DepthTexture sampling and distance calculation (and revert edgeOffset and edgeAmount to static values), the filter behaves like the original OutlineFilter and avoids the conflict, albeit losing the distance fade.
Order of Filters and Depth Issues (re-confirming for clarity):
CartoonEdgeFilter (mine) BEFORE WaterFilter: CartoonEdgeFilter simply doesn’t draw outlines. This suggests its m_DepthTexture input might be empty or incorrect at this stage.
WaterFilter BEFORE CartoonEdgeFilter (mine): CartoonEdgeFilter does draw outlines (with distance fade), but the WaterFilter then draws on top of everything, ignoring depth. This implies WaterFilter’s depth input is being affected or cleared by my filter’s processing, or my filter is somehow interfering with the scene’s depth buffer before WaterFilter gets a chance to use it correctly.
You’ve hit on the exact distinction that seems to be causing the problem, and my previous debugging missed this subtlety!
My apologies, I misspoke in my last message regarding the “original one.” The crucial detail is:
When I use the built-in CartoonEdgeFilter class directly (i.e., new CartoonEdgeFilter()), everything works perfectly fine, even when combined with WaterFilter in either order.
The issue only occurs when I use my custom CartoonEdgeEnhancedFilter class, even if the fragment shader (/Materials/CartoonEdgeEnhanced.frag) used by my custom filter contains the exact same code as the original CartoonEdge.frag.
Since you mentioned edge fading, then that makes me wonder if you are also setting the Boxes into the Transluscent queue bucket with alpha blend mode enabled, which may be affecting things.
I know that a similar issue occurs with particle emitters where they will also end up rendered behind the Water filter whenever they are in the Transluscent Bucket, and adding a TransluscentBucketFilter as the last filter fixes that.
It looks like something similar is happening here, but I don’t have much experience working with filters to know if it applies the same way.
