Avoid shadows in areas covered by fog?

Troubleshooting graphics
1

Hi,

I’m using fog shader, and I want areas that are far from camera to be 100% covered with the fog, like this:

image
image1390×686 80.6 KB

(as you can see - the mountains at the background are entirely “blue”, which is the color of the fog)

Now when I’m adding shadow:

var dlsr = new DirectionalLightShadowRenderer(assetManager, SHADOWMAP_SIZE, NO_MAPS);
dlsr.setLight(light);
dlsr.setLambda(0.75f);
dlsr.setShadowIntensity(0.8f);
dlsr.setEdgeFilteringMode(EdgeFilteringMode.Nearest);

this is how it looks:

image
image1405×679 91.2 KB

the fog effect looks bad, as the shadow divides the foggy area into shaded vs lighted. The shadows should not be visible on mountain, as you wouldn’t be able to see mountain from behind the fog.

I tried to mitigate the issue by limiting the area of shadow calculations:
dlsr.setShadowZExtend(200);
and by softening the shadows as they are close to the border of shadow-related area:
dlsr.setShadowZFadeLength(100);
and I can see some improvement:

image
image1400×681 86.5 KB

However, this is still not perfect.
I would be grateful for some advice.

I also tried DirectionalLightShadowFilter as opposed to Shadow Renderer, with the same result.
What I believe would be a perfect solution, is to have things calculated in appropriate order, that is: first calculate the shadows, then calculate the fog. I cannot find the shadow function in the Lightning.frag though. I read on wiki that the Shadow Filter is some post-processing, but I hoped that maybe the Shadow Renderer is inside the shader, so that I could maybe move things around in the GLSL?

I did some test: covered entire area with 100% intensity fog (set that in the shader), and the shadow were still there, so it seems shadows are calculated later, after the fog.

I know that there is a fog filter in jME, perhaps that works better together with the shadows. I haven’t tried it yet. In general, I tweaked the jME shader fog on GLSL level to suit my needs, so I would like to be able to use that, without needing to switch to another fog mechanism.

1 Like
2

With stock JME fog/shadows, I think your options are:

  1. make sure that shadows don’t render at fog distance (kind of what you are doing with your ‘better’ version)
  2. use the fog filter since it’s a post processing effect.

The problem with post-process fog is that it brings its own set of limitations. It will ‘fog’ anything in the z-buffer regardless of transparency or desire to be foggy. (For example, you could not have distant bright things that are unfoggy and things with transparency may look weirdly fogged.)

Edit: the non-stock-JME option I guess would be to develop ‘in shader’ shadows instead of rendering them as a separate pass. Non-trivial, I think.

3

So both jME shadows are indeed post-processed, as I suspected. Thank you for clarification, and thanks for the advice <3

1 Like
4

I think the most simple solution is to redo the fog calculcation in the shadow shader and apply shadowing depending on the fog value. At least i think that would require the lowest amount of changes.

2 Likes
5

Oh, that’s a good idea. I hadn’t thought of that.

It’s still “shader hacking” but it’s way easier shader hacking.

6

Do you know where are the shadow shader calculations?
I can see something in jmonkeyengine/jme3-core/src/main/resources/Common/MatDefs/Shadow/PostShadow.frag at 9a105913fc6941efa47990c8503d656ddf6f020b · jMonkeyEngine/jmonkeyengine · GitHub but this seems to be related to a “Basic Post Shadow” material definition which I don’t use…

Edit: or maybe this is a material used under the hood by shadow system?