BetterGroundFogFilter


#1

Hello monkeys,

I always had a problem with the FogFilter in that it might look good for close view games, but looks strange when using for bigger landscapes beause its not height based.

I searched for different ways to approach it and finally decided to follow this paper: http://www.terathon.com/lengyel/Lengyel-UnifiedFog.pdf
However, this is a postprocessing effect, so you can plug it in and it should work.
Then I added the option to set a directional light used as mainlight to do some color changes when looking into the direction of the sun / moon / whatever.
It also supports multisampling.
Additionally you can set a lowest level to apply fog to (that is when the shader finds a fragment that had a worldposition below this level, its pushed back along the viewDirection until its at this level.
Imagine you want fog to appear up to a height of y=60 but you use the postprocessing water at height y=0. You usually dont want fog to be applied underwater, but the water didnt write into the depth buffer which is used to calculate back the fragments worldposition, thus you can set the fogs groundLevel to 0 and it will not apply fog when underwater

Files you need to make it work:
-BetterGroundFog.j3md
-BetterGroundFog.frag
-BetterGroundFogFilter.java
-(optionally) BetterGroundFogState.java

And some screenshots so you can see if its worth checking it out:
first one shows fog when looking into the light at night (cou can see its brighter around the moon)
second one shows fog when standing in the fog, looking through it
third one shows inside fog (that is inside a cave in this case)
last one shows the actual “ground”-fog thing
(click it, its an album)

quick tutorial about how to make it work with TerrainTestAdvanced: https://github.com/jMonkeyEngine/jmonkeyengine/blob/master/jme3-examples/src/main/java/jme3test/terrain/TerrainTestAdvanced.java
-copy and paste the whole TerrainTestAdvanced.java
-at the bottom of simpleInitApp() add the following (make sure to add proper imports):

final BetterGroundFogState fog = new BetterGroundFogState();
fog.setSun(light); //put reference to sunlight, so it automatically updates
fog.getFilter().setGroundLevel(-128); 
fog.getFilter().setFogDensity(0.0325f);
fog.getFilter().setFogBoundary(new Vector4f(0, 1, 0, 60)); //x y and z need to be unitvector pointing away from the fog, w is the distance from the origin, so this is horizontal fog reaching up to y=60
fpp = new FilterPostProcessor(assetManager); //setup filterpostprocessor
int samples = settings.getSamples();
if (samples > 1) {
    fpp.setNumSamples(samples);
}
fpp.addFilter(fog.getFilter()); //add the filter at the position you want it to be in your fpp
stateManager.attach(fog);
viewPort.addProcessor(fpp);

‘fog’ is declared final because of the following lines i recommend to add

flyCam.setMoveSpeed(150); //otherwise its too fast
light.setDirection((new Vector3f(0.1f, -0.1f, 0.1f)).normalize()); //that thing on the skybox actually points towards this direction
//add some inputs to test day / night / indoors / groundlevel
inputManager.addMapping("toggleNight", new KeyTrigger(KeyInput.KEY_E));
inputManager.addMapping("toggleIndoor", new KeyTrigger(KeyInput.KEY_R));
inputManager.addMapping("toggleGround", new KeyTrigger(KeyInput.KEY_F));
inputManager.addListener(new ActionListener() {
    @Override
    public void onAction(String name, boolean isPressed, float tpf) {
        if (!isPressed) {
           return;
        }
        switch (name) {
            case "toggleNight":
                fog.setIsIndoors(!fog.isIndoors());
                break;
            case "toggleIndoor":
                fog.setIsNight(!fog.isNight());
                break;
            case "toggleGround":
                float c = fog.getFilter().getGroundLevel();
                if (c > -Float.MAX_VALUE) {
                    fog.getFilter().setGroundLevel(-Float.MAX_VALUE);
                } else {
                    fog.getFilter().setGroundLevel(-128);
                }
               break;
           }
        }
     }, "toggleNight", "toggleIndoor", "toggleGround");

So here is the code for the files needed:
-BetterGroundFog.j3md

MaterialDef BetterGroundFog {
    MaterialParameters {
        Int NumSamples
        Int NumSamplesDepth
        Texture2D Texture
        Texture2D DepthTexture

        Vector3 SunDirection
        Vector4 SunColor
        Float SunShininess

        Vector4 FogColor
        Float FogDensity

        Float GroundLevel
        Vector4 FogBoundary

        Float K
        Float CF
    }

    Technique {
        VertexShader GLSL150: Common/MatDefs/Post/Post.vert
        FragmentShader GLSL150: Shaders/BetterGroundFog.frag

        WorldParameters {
            ViewProjectionMatrixInverse
            CameraPosition
        }
        
        Defines {
            RESOLVE_MS : NumSamples
            RESOLVE_DEPTH_MS : NumSamplesDepth
            SUN : SunDirection
            GROUND : GroundLevel
        }
    }
}

-BetterGroundFog.frag

#import "Common/ShaderLib/MultiSample.glsllib"

// Fog pixel shader
// by Alexander Kasigkeit (samwise) for JMonkeyEngine 3.2
// based on by Eric Lengyel - Terathon Software - lengyel@terathon.com http://www.terathon.com/lengyel/Lengyel-UnifiedFog.pdf
// position calculation and multisampling taken from Remy Bouquet (nehon)'s Water pixel shader https://github.com/jMonkeyEngine/jmonkeyengine/blob/master/jme3-effects/src/main/resources/Common/MatDefs/Water/Water.frag

uniform mat4 g_ViewProjectionMatrixInverse;
uniform vec3 g_CameraPosition;

uniform COLORTEXTURE m_Texture;
uniform DEPTHTEXTURE m_DepthTexture;

uniform float m_FogDensity; 
uniform vec4 m_FogColor; 
uniform vec4 m_FogBoundary; 
uniform float m_K; 
uniform float m_CF;

#ifdef GROUND
    uniform float m_GroundLevel;
#endif
#ifdef SUN
    uniform vec3 m_SunDirection; 
    uniform vec4 m_SunColor; 
    uniform float m_SunShininess;
#endif

in vec2 texCoord;
out vec4 color;

vec4 main_multiSample(in int sampleNum){
    vec4 colorTex = fetchTextureSample(m_Texture, texCoord, sampleNum);
    float depth = fetchTextureSample(m_DepthTexture, texCoord, sampleNum).r;
    vec4 fragPos = g_ViewProjectionMatrixInverse * (vec4(texCoord, depth, 1.0) * 2.0 - 1.0);
    vec3 wsFragPos = fragPos.xyz /  fragPos.w;
    vec3 wsFragDelta = g_CameraPosition - wsFragPos;
    #ifdef GROUND
        //adjust frags worldspace position in case its below groundLevel
        float d = 1.0 - max(0.0, m_GroundLevel - wsFragPos.y) / wsFragDelta.y;
        wsFragPos = g_CameraPosition + d * -wsFragDelta;
        wsFragDelta = g_CameraPosition - wsFragPos;
    #endif
    vec3 wsFragDir = normalize(wsFragDelta);
    //calculate fog color based on viewDirection and sunDirection
    //this does not work for indoors, the fogs color would still change when looking into the direction of the sun
    float t = 0.0;
    #ifdef SUN
        t = pow(depth, m_SunShininess * m_SunShininess);
        t = pow(t * max( dot( wsFragDir, m_SunDirection ), 0.0 ), m_SunShininess);
    #endif
    vec3 colorFog = mix( m_FogColor.rgb, 
                         m_SunColor.rgb, 
                         t );
    //calculate amount to apply to this fragment
    float FP = dot(m_FogBoundary, vec4(wsFragPos, 1.0)); 
    float fc = min((1.0 - 2.0 * m_K) * FP, 0.0); 
    fc = -length((m_FogDensity * 0.01) * wsFragDelta) * ((m_K * (FP + m_CF)) - pow(fc, 2.0) / abs(dot(m_FogBoundary, vec4(wsFragDelta, 0.0)))); 
    fc = (1.0 - clamp(exp2(-fc), 0.0, 1.0));
    #ifdef GROUND
        fc *= step(m_GroundLevel, g_CameraPosition.y);
    #endif
    return vec4(mix(colorTex.rgb, 
                    colorFog, 
                    fc), 
                1.0); 
}

void main() {
    #ifdef RESOLVE_MS
        vec4 col = vec4(0.0);
        for (int i = 0; i < m_NumSamples; i++){
            col += main_multiSample(i);
        }
        color = col / m_NumSamples;
    #else
        color = main_multiSample(0);
    #endif
}

-BetterGroundFogFilter.java

package com.kabigames.tests.groundfog;

import com.jme3.asset.AssetManager;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.Vector3f;
import com.jme3.math.Vector4f;
import com.jme3.post.Filter;
import com.jme3.renderer.RenderManager;
import com.jme3.renderer.Renderer;
import com.jme3.renderer.ViewPort;

/**
 *
 * @author Alexander Kasigkeit <alexander.kasigkeit@web.de>
 */
public class BetterGroundFogFilter extends Filter {

    private Vector3f sunDirection = new Vector3f(-1, -1, -1).normalizeLocal();
    private ColorRGBA sunColor = new ColorRGBA(1.0f, 0.9f, 0.7f, 1.0f);
    private float sunShininess = 8.0f;

    private ColorRGBA fogColor = new ColorRGBA(0.5f, 0.6f, 0.7f, 1.0f);
    private float fogDensity = 0.025f;
    private float groundLevel = 0f;
    private Vector4f fogBoundary = new Vector4f(0, 1, 0, -50);

    private float k = 0f;
    private float cdotf = 0f;

    public BetterGroundFogFilter() {
        super("BetterGroundFogFilter");
    }

    /**
     * sets direction of the sunlight
     *
     * @param direction normalized direction pointing from the sun, not towards the sun
     */
    public void setSunDirection(Vector3f direction) {
        sunDirection = direction;
    }

    /**
     * set the fogs color when looking right into the sun through the fog, default is 1.0, 0.9, 0.7 (slightly yellowish)
     *
     * @param color fogs color
     */
    public void setSunColor(ColorRGBA color) {
        sunColor = color;
    }

    /**
     * used to adjust the amount of fog thats influenced by the sun, higher values give smaller areas ( pow(sunFactor,
     * sunShininess)), default is 8.0
     *
     * @param shininess the shininess of the sun
     */
    public void setSunShininess(float shininess) {
        sunShininess = shininess;
    }

    /**
     * set the fogs color, default is 0.5, 0.6, 0.7 (slightly blueish grey)
     *
     * @param color fogs color
     */
    public void setFogColor(ColorRGBA color) {
        fogColor = color;
    }

    /**
     * sets the fogs density, higher values give thicker fog, default is 0.025
     *
     * @param density fogs density
     */
    public void setFogDensity(float density) {
        fogDensity = density;
    }

    /**
     * sets the height of the groundLevel (can be used to prevent from underwaterfog). each fragment that is calculated
     * to be below this position is treated as if it was at this height, default is 0. set to Float.MIN_VALUE to turn
     * off
     *
     * @param groundLevel lowest level to apply fog to
     */
    public void setGroundLevel(float groundLevel) {
        this.groundLevel = groundLevel;
    }

    /**
     * sets the fogs boundary, x,y and z components should be its normal pointing away from the fog, w is the distance,
     * default is 0,1,0,-50 (that means fog lays horizontally and starts at y=50 and). After the fogs plane was updated,
     * updateCameraPosition(pos) should be called
     *
     * @param boundary the plane that seperates fogged from not fogged area
     */
    public void setFogBoundary(Vector4f boundary) {
        fogBoundary = boundary;
    }

    /**
     * should be called whenever the camera position changes, updates c-dot-f and k so it does not have to be calculated
     * for each fragment
     *
     * @param pos current position of the camera
     */
    public void updateCameraPosition(Vector3f pos) {
        if (fogBoundary == null) {
            throw new NullPointerException("cannot update camera position: fog boundary is null");
        }
        cdotf = fogBoundary.x * pos.x + fogBoundary.y * pos.y + fogBoundary.z * pos.z + fogBoundary.w;
        if (cdotf > 0f) {
            k = 0f;
        } else {
            k = 1f;
        }
    }

    /**
     * returns sunDirection
     *
     * @return sunDirection
     */
    public Vector3f getSunDirection() {
        return sunDirection;
    }

    /**
     * returns sunColor
     *
     * @return sunColor
     */
    public ColorRGBA getSunColor() {
        return sunColor;
    }

    /**
     * returns sunShininess
     *
     * @return sunShininess
     */
    public float getSunShininess() {
        return sunShininess;
    }

    /**
     * returns fogColor
     *
     * @return fogColor
     */
    public ColorRGBA getFogColor() {
        return fogColor;
    }

    /**
     * returns fogDensity
     *
     * @return fogDensity
     */
    public float getFogDensity() {
        return fogDensity;
    }

    /**
     * returns groundLevel
     *
     * @return groundLevel
     */
    public float getGroundLevel() {
        return groundLevel;
    }

    /**
     * returns fogBoundary
     *
     * @return fogBoundary
     */
    public Vector4f getFogBoundary() {
        return fogBoundary;
    }

    @Override
    public boolean isRequiresDepthTexture() {
        return true;
    }

    @Override
    public void initFilter(AssetManager assets, RenderManager renderManager,
            ViewPort vp, int w, int h) {
        material = new Material(assets, "MatDefs/BetterGroundFog.j3md");
    }

    @Override
    public Material getMaterial() {
        if (sunDirection != null) {
            material.setVector3("SunDirection", sunDirection);
            material.setColor("SunColor", sunColor);
            material.setFloat("SunShininess", sunShininess);
        } else {
            clearIfExists(material, "SunDirection");
            clearIfExists(material, "SunColor");
            clearIfExists(material, "SunShininess");
        }
        if (groundLevel > -Float.MAX_VALUE) {
            material.setFloat("GroundLevel", groundLevel);
        } else {
            clearIfExists(material, "GroundLevel");
        }
        material.setColor("FogColor", fogColor);
        material.setFloat("FogDensity", fogDensity);
        material.setVector4("FogBoundary", fogBoundary);
        material.setFloat("K", k);
        material.setFloat("CF", cdotf);
        return material;
    }

    protected void clearIfExists(Material mat, String param) {
        if (material.getParam(param) != null) {
            material.clearParam(param);
        }
    }

    @Override
    public void cleanUpFilter(Renderer r) {
    }

}

-BetterGroundFogState.java:


package com.kabigames.tests.groundfog;

import com.jme3.app.Application;
import com.jme3.app.state.BaseAppState;
import com.jme3.light.DirectionalLight;
import com.jme3.math.ColorRGBA;

/**
 *
 * @author Alexander Kasigkeit <alexander.kasigkeit@web.de>
 */
public class BetterGroundFogState extends BaseAppState {

    protected final ColorRGBA FOG_DAY = new ColorRGBA(0.5f, 0.6f, 0.7f, 1.0f);
    protected final ColorRGBA FOG_NIGHT = new ColorRGBA(0.25f, 0.3f, 0.35f, 1.0f);
    protected final ColorRGBA FOG_INDOORS = new ColorRGBA(0.10f, 0.10f, 0.14f, 1.0f);

    protected final ColorRGBA SUN_DAY = new ColorRGBA(1.0f, 0.9f, 0.7f, 1.0f);
    protected final ColorRGBA SUN_NIGHT = new ColorRGBA(0.35f, 0.45f, 0.5f, 1.0f);
    protected final ColorRGBA SUN_INDOORS = FOG_INDOORS;

    protected final ColorRGBA FOG_COLOR = new ColorRGBA(FOG_DAY);
    protected final ColorRGBA SUN_COLOR = new ColorRGBA(SUN_DAY);
    protected final ColorRGBA TARGET_FOG_COLOR = new ColorRGBA();
    protected final ColorRGBA TARGET_SUN_COLOR = new ColorRGBA();

    protected BetterGroundFogFilter filter = new BetterGroundFogFilter();

    protected DirectionalLight sun = null;
    protected float fadeTime = 2f;
    protected boolean isIndoors = false, isNight = false;

    @Override
    protected void initialize(Application app) {
        if (sun != null) {
            filter.setSunDirection(sun.getDirection());
        }
        filter.updateCameraPosition(app.getCamera().getLocation());
    }

    @Override
    protected void cleanup(Application app) {
    }

    @Override
    protected void onEnable() {
    }

    @Override
    protected void onDisable() {
    }

    @Override
    public void update(float tpf) {
        filter.updateCameraPosition(getApplication().getCamera().getLocation());
        if (sun != null) {
            filter.setSunDirection(sun.getDirection());
        }

        if (isIndoors) {
            TARGET_SUN_COLOR.set(SUN_INDOORS);
            TARGET_FOG_COLOR.set(FOG_INDOORS);
        } else {
            if (isNight) {
                TARGET_SUN_COLOR.set(SUN_NIGHT);
                TARGET_FOG_COLOR.set(FOG_NIGHT);
            } else {
                TARGET_SUN_COLOR.set(SUN_DAY);
                TARGET_FOG_COLOR.set(FOG_DAY);
            }
        }
        float perc = tpf / fadeTime;
        FOG_COLOR.interpolateLocal(TARGET_FOG_COLOR, perc);
        SUN_COLOR.interpolateLocal(TARGET_SUN_COLOR, perc);
        filter.setSunColor(SUN_COLOR);
        filter.setFogColor(FOG_COLOR);
    }

    /**
     * saves a reference to the directional light used as sunlight, to send uptodate direction to the shader per frame
     *
     * @param sun DirectionalLight used as sun
     */
    public void setSun(DirectionalLight sun) {
        this.sun = sun;
    }

    /**
     * sets the duration in seconds it takes to fade from indoors to outdoors / night to day
     *
     * @param fadeTime time in seconds, default 2.0f
     */
    public void setFadeTime(float fadeTime) {
        this.fadeTime = fadeTime;
    }

    /**
     * used to set indoors state, in indoors state fog color and sunlight color are the same to prevent from light
     * scattering inside
     *
     * @param isIndoors true for setting to indoor, default is false
     */
    public void setIsIndoors(boolean isIndoors) {
        this.isIndoors = isIndoors;
    }

    /**
     * at night you usually want darker fog and no yellowish lightscattering so you can define night colors and then
     * fade to them by activating night mode
     *
     * @param isNight true for settings to night, default is false
     */
    public void setIsNight(boolean isNight) {
        this.isNight = isNight;
    }

    /**
     * sets the fog color used for fog at daytime that is not influenced by sunlight
     *
     * @param color fogs color, default is (0.5f, 0.6f, 0.7f, 1.0f) slightly blueish gray
     */
    public void setFogColorDay(ColorRGBA color) {
        FOG_DAY.set(color);
    }

    /**
     * sets the color used for 'lightscattering' at day, that is the color the fog fades into when looking into the
     * direction of the sun
     *
     * @param color sun color, default is (1.0f, 0.9f, 0.7f, 1.0f) slightly yellowish grey
     */
    public void setSunColorDay(ColorRGBA color) {
        SUN_DAY.set(color);
    }

    /**
     * sets the fog color used for fog at nighttime that is not influenced by sunlight
     *
     * @param color fogs color, default is (0.25f, 0.3f, 0.35f, 1.0f) slightly blueish darker gray
     */
    public void setFogColorNight(ColorRGBA color) {
        FOG_NIGHT.set(color);
    }

    /**
     * sets the color used for 'lightscattering' at night, that is the color the fog fades into when looking into the
     * direction of the moon
     *
     * @param color moon color, default is (0.35f, 0.45f, 0.5f, 1.0f) slightly blueish grey
     */
    public void setSunColorNight(ColorRGBA color) {
        SUN_NIGHT.set(color);
    }

    /**
     * sets the fog color used for fog indoors that is never influenced by sunlight
     *
     * @param color fogs color, default is (0.10f, 0.10f, 0.14f, 1.0f) slightly blueish darker gray, useful for caves to
     * prevent them from seeming brighter due to grey fog
     */
    public void setFogColorIndoors(ColorRGBA color) {
        FOG_INDOORS.set(color);
    }

    /**
     * sets the color used for 'lightscattering' indoors, that is the color the fog fades into when looking into the
     * direction of the light, default is the same as indoors fog color so you get no lightscattering effect indoors
     *
     * @param color light color, default is (0.10f, 0.10f, 0.14f, 1.0f), same as indoors fog color
     */
    public void setSunColorIndoors(ColorRGBA color) {
        SUN_INDOORS.set(color);
    }

    /**
     * returns a reference to the directional light used as sun
     *
     * @return sun, or null if not set
     */
    public DirectionalLight getSun() {
        return sun;
    }

    /**
     * get the time used to fade between different fog colors
     *
     * @return fadeTime in seconds
     */
    public float getFadeTime() {
        return fadeTime;
    }

    /**
     * returns current indoors state
     *
     * @return true if fog is in indoors state
     */
    public boolean isIndoors() {
        return isIndoors;
    }

    /**
     * returns current night state
     *
     * @return true if fog is in night state
     */
    public boolean isNight() {
        return isNight;
    }

    /**
     * returns a reference to the BetterFogFilter, you need that to get a reference to add to your FilterPostProcessor
     * at the position you prefer (should be after shadows and postprocessing water at least)
     *
     * @return the filter this state updates
     */
    public BetterGroundFogFilter getFilter() {
        return filter;
    }

}

I had to finish the post in a rush a little because I’ve got to leave now, but in case you got questions or any improvements or crashes or similar, ask them, it just might take some hours for me to answer
just to have it mentioned, so far i only tested it on my machine, nvidia card etc…

Greetings from the shire,
samwise