[SOLVED] DirectionalLightShadowFilter causing pixelation of geometry edges

JamesRichardson · August 1, 2015, 5:07pm

Hello!

I’ve been experimenting with DirectionalLightShadowRenderer and DirectionalLightShadowFilter, trying to weigh up which is the best one to use.

DirectionalLightShadowFilter seems to give me more fps. However, I notice that with DirectionalLightShadowFilter, the edges of geometries in the distance seem to become pixelated, whereas with DirectionalLightShadowRenderer, the edges of geometries are always smooth. This effect does seem to only be noticeable on geometries in the distance. Up close, everything looks smooth whichever class I use.

I haven’t been able to find this issue mentioned anywhere, so, I thought I’d post a topic.

I’ve written two simple programs that demonstrate this effect, and have attached the screenshots for comparison.

Using DirectionalLightShadowRenderer, the edges are nice and smooth:

Using DirectionalLightShadowFilter, the edges are pixelated:

The settings I used to produce these screenshots were 640 x 480, Vzync enabled, 24 bpp color depth, and 16x anti-aliasing. However, whatever settings I use, I get the same pixelating effect on geometries in the distance.

The difference is subtle in those screenshots, but, in the game I’m working on, and when adding shadows to that town.zip scene, it’s quite noticeable and looks quite ugly.

The code that uses DirectionalLightShadowRenderer is:

package mygame;

import com.jme3.app.SimpleApplication;
import com.jme3.light.DirectionalLight;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.FastMath;
import com.jme3.math.Vector3f;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.shadow.DirectionalLightShadowRenderer;

public class DirectionalLightShadowRendererTest extends SimpleApplication {

    public static void main(String[] args) {
        DirectionalLightShadowRendererTest app = new DirectionalLightShadowRendererTest();
        app.start();
    }

    @Override
    public void simpleInitApp() {
        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);
        geom.rotate(0, FastMath.QUARTER_PI, 0);

        Material mat = new Material(assetManager, "Common/MatDefs/Light/Lighting.j3md");
        mat.setBoolean("UseMaterialColors", true);
        mat.setColor("Diffuse", ColorRGBA.White);
        geom.setMaterial(mat);

        rootNode.attachChild(geom);
        
        DirectionalLight sun = new DirectionalLight();
        sun.setDirection(new Vector3f(0, 0, -1));
        rootNode.addLight(sun);
        
        DirectionalLightShadowRenderer dlsr = new DirectionalLightShadowRenderer(assetManager, 2048, 4);
        dlsr.setLight(sun);
        viewPort.addProcessor(dlsr);
    }
}

The code that uses DirectionalLightShadowFilter is:

package mygame;

import com.jme3.app.SimpleApplication;
import com.jme3.light.DirectionalLight;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.FastMath;
import com.jme3.math.Vector3f;
import com.jme3.post.FilterPostProcessor;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.shadow.DirectionalLightShadowFilter;

public class DirectionalLightShadowFilterTest extends SimpleApplication {

    public static void main(String[] args) {
        DirectionalLightShadowFilterTest app = new DirectionalLightShadowFilterTest();
        app.start();
    }

    @Override
    public void simpleInitApp() {
        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);
        geom.rotate(0, FastMath.QUARTER_PI, 0);

        Material mat = new Material(assetManager, "Common/MatDefs/Light/Lighting.j3md");
        mat.setBoolean("UseMaterialColors", true);
        mat.setColor("Diffuse", ColorRGBA.White);
        geom.setMaterial(mat);
        
        rootNode.attachChild(geom);
        
        DirectionalLight sun = new DirectionalLight();
        sun.setDirection(new Vector3f(0, 0, -1));
        rootNode.addLight(sun);
        
        DirectionalLightShadowFilter dlsf = new DirectionalLightShadowFilter(assetManager, 2048, 4);
        dlsf.setLight(sun);
        dlsf.setEnabled(true);
        
        FilterPostProcessor fpp = new FilterPostProcessor(assetManager);
        fpp.addFilter(dlsf);
        viewPort.addProcessor(fpp);
    }
}

Has anyone else come across this issue, or is it just me? If it’s a known issue, is there something I can do about it, or is it just a feature of DirectionalLightShadowFilter?

Of course, I can just use DirectionalLightShadowRenderer, but, DirectionalLightShadowFilter seems to give me a higher fps, so, I’d use DirectionalLightShadowFilter if I could find a way to sort out this pixelating effect.

James

nehon · August 1, 2015, 5:13pm

You need to set the number of samples to the fpp, it’s not automatic.

fpp.setNumSamples(16);

JamesRichardson · August 1, 2015, 6:09pm

Looks like that’s it. For some reason, that doesn’t have any noticeable effect in DirectionalLightShadowFilterTest, but in the town.zip scene, it certainly does make a difference. A value of 4 just about does the trick. With 16, my fps goes down to about 20.

With my own game, calling setNumSamples with any value greater than 1 causes an exception:
org.lwjgl.opengl.OpenGLException: Invalid enum (1280) :chimpanzee_facepalm:

However, that’s another matter that I’ll look into tomorrow.

Thanks for your help!

pspeed · August 2, 2015, 5:04am

…then you should also use 4x for anti-aliasing as that’s effectively all you are getting by setting post-proc samples to 4.

JamesRichardson · August 2, 2015, 4:18pm

OK, I see. I hadn’t made the connection between the anti-aliasing setting and setNumSamples. Without calling setNumSamples, the FilterPostProcessor was effectively disabling anti-aliasing, which is why edges looked pixelated, just as they do when I use a DirectionalLightShadowRenderer and disable anti-aliasing. So, if I use a FilterPostProcessor, I need to call setNumSamples with the required anti-aliasing value, which is what nehon meant when he said “it’s not automatic”. Got it!

It seems to me that at 2 numSamples, DirectionalLightShadowFilter is faster than DirectionalLightShadowRenderer with 2x anti-aliasing. However, at 16 numSamples, DirectionalLightShadowFilter is much slower than DirectionalLightShadowRenderer with 16x anti-aliasing. Is that normal, or does it depend on the graphics card?

Turns out this was because I had -enableassertions set. So, there must be an assertion in there somewhere (presumably in org.lwjgl.opengl.Util.checkGLError). However, if I remove -enableassertions, it seems to run OK, so, it would seem that whatever assert was failing was doing so unnecessarily.

Thanks for your help, guys.

nehon · August 2, 2015, 6:58pm

The Filter is faster than the Renderer when you have more than a certain amout of geometries (the number can depend on your hardware, but got around 30 on my card). However antialiasing add additional treatments to the filter pass and tends to gets slower the more you add samples. So it may still be faster when you have only 2 samples but can be slower with 16 for sure.
16 is a lot though.

You may consider using the FXAA filter for a faked antialiasing when you use filters. It’s not as good as classic anti aliasing (multi sampling, hence the numSamples), but it’s way faster.

It’s been a long time since I’d like to make the FPP grab what was set in the application setting for numSamples… maybe it’s time.

JamesRichardson · August 2, 2015, 8:02pm

Yeah, the FXAAFilter certainly does add rough but relatively inexpensive anti-aliasing. Thanks for the pointer!

Am I right in thinking that if you’re using filtering, while the numSamples certainly has an effect, the anti-aliasing value (the one on the initial SettingsDialog) has no effect on the graphics quality, other than reducing the fps, so, if you’re using filtering, you might as well disable anti-aliasing (on the initial SettingsDialog)? That seems to be what I’m seeing.

pspeed · August 3, 2015, 2:23am

Filtering will use numSamples to pay attention to anti-aliasing you already have set. If you set AA and not numSamples then you get no AA… so yeah, then you are wasting CPU.

But if you set numSamples then having AA on makes a difference… because without that you will get no AA no matter how you set numSamples.

nehon · August 3, 2015, 9:43am

I’m not completely sure about that. The AA setting governs the number of samples of the default frame buffer. For post processing we have a custom frame buffer to render the scene. That’s the one needing the numSample param set to the post processor.

So in case of post processing the AA setting whould just aplly AA on the final image rendered on a fullscreen quad. not sure it has a visible effect.

pspeed · August 3, 2015, 2:01pm

Ah, didn’t know that. Makes sense, though.

…though that does mean that if num samples is set on FPP then setting the main AA setting is just a (slight) performance drain for no benefit at all.

nehon · August 3, 2015, 3:07pm

yep exactly

JamesRichardson · August 3, 2015, 5:27pm

OK, that’s cool. Thanks for your help!

Momoko_Fan · August 3, 2015, 7:26pm

There’s no way of synchronizing AppSettings num samples and FilterPostProcessor … the information is not passed around. I was thinking of adding a flag like AppSettings.setIgnoreSamples(true) to indicate that the application is using post-processing so the user-selected anti-aliasing setting should be ignored when creating the context. Then the user can pass num samples from the AppSettings to the FilterPostProcessor, but it is a manual process.

pspeed · August 3, 2015, 7:33pm

Yeah, I was more speaking from the technical side. I suspect most real games won’t use the ugly JME app settings dialog anyway and instead write their own in-game settings screen.