Volumetric Lighting Filter [WIP]

Hi Monkeys,

It’s been a while since I’ve had anything I felt worth worthy to contribute, now on my 3rd attempt at Volumetric Lighting I have finally arrived at something I’m happy with. I learned a bunch while working on this project, some of which I would like to share with those who are interested.

First things first:

The Project

The source is on google code. I’ve also created a full download of the source and test scene zipped up that comes in at a whopping 27.4 KB.

Video of The Test Scene

[video]http://www.youtube.com/watch?v=1fXvX10SJXs[/video]

3 “Volumetric Spot Lights” buzzing around… There is no real lighting in the scene, nor are there any real shadows, it is simply showing off the volumetric light.

Earlier video testing in a real scene (Village Showcase courtesy of @destroflyer) using shadows and a real spot light

[video]http://www.youtube.com/watch?v=JnvidKsiFtU[/video]

My Philosophy on Open Source Software

Here is what I made, it's open source, this is how I made it, there is nothing to hide. Now you take it away, learn from it, add to it, improve it, then bring it back and show me what you did, so I can learn too.

Background

I played Allan Wake recently, I was blown away by the graphics and story telling, but especially loved the flash light effect, so I wanted to recreate it using real-time Volumetric Lighting.

My first 2 failed attempts:
A few years ago I had a rough ray marching approach in my head, gave it a crap attempt, which failed, so trashed that and moved on.

Then within the last few months, I attempted to use what I’ve been calling the “Mitchel Light Shafts” approach, pioneered by Dobashi and Nishita, and very nicely summed up in this pdf.

This method basically uses screen aligned billboard that fill the light frustum, which looks like this:

… I got this up and running, but my attempt looked crap and was not what I was after… when the light shaft is thin, like a flash light, and many angles there are simply not enough billboards to fill the volume efficiently and it left many gaps:

The blue box is the light source.

I did learn a lot about matrices and moving geometry around relative to a camera and bounding volume, so it wasn’t a complete loss.

So I did loads of research, read roughly 20 mind bending papers all covering different aspects of creating real time volumetric lighting, mashed everything I learned together and came up with a kind of hybrid approach, which is roughly described in the Real Time Volumetric Shadows using Polygonal Light Volumes paper.

So How Does It Work

First thing is to make a “Light Volume” or a closed mesh that represents all the air we want to illuminate. I treated the light volume as if it were a camera frustum, with a subdivided plane as the far plane. A depth map is rendered from the light’s perspective (the same way JME shadows work) in

VLRenderer.java, which is a cut down version of BasicShadowRenderer.java, utilising just the depth map gathering portion (if the Volumetric Lighting is integrated into the existing lighting systems, this step could be bypassed) and the subdivide far plane is “vacume formed” over that depth map in the vertex shader, which means each vertex is displaced along the z access by the depth texture …

This is a really neat way of forming a lighting volume (like a reverse shadow volume) that doesn’t rely on manipulating the scene geometry, while not as accurate as other methods, accuracy is not super important given the nature of volumetric light. So we start with a custom mesh that looks like this (super low resolution mesh 8x8):

and end up with a mesh that looks roughly like this (crappy example I know, get over it) :

Tip: Optimizing Custom mesh Generation :: Avoid Expensive Lookups! When creating my custom frustum mesh, I initially stored my plane vertex indices in a 2D ArrayList, so I could look up their values later on, using arrayList.get(column).get(row), after some initial testing I found the mesh generation to be painfully slow, and after some investigation I found that reading from this lookup was very expensive, when I switched to using a simple integer calculation (roughly index = columnNumber * NumberOfColumns + rowNumber), and it was 231x faster!!. I found it interesting that reading values from ArrayLists to be so slow, perhaps someone out there has some insight?

Because spotlights are round, there is no need to deal with any vertices that fall outside the light cone, so came up with a neat little trick to catch those …
if (length(posInPLS.xy) < posInPLS.w) {
which roughly translates to: if when looking from the point of view of the light (Projected Light Space) the distance of the point from the middle middle of the view , length(posInPLS.xy), is greater than half the width (or height) of the view, posInPLS.w, then exclude it… (.w is a magical value I don’t completely understand, but it behaves like that in this case).

Tip: when working with a depthTexture, you will often need to linearise the depth. So you must Learning to Love your Z-buffer. !!!

I did some of the calculations on the CPU once and passed them to my shaders, to avoid needing to perform this calculation on the GPU every frame, or worse every vert, or even worse every fragment, in my shaders.

[java]
public Vector2f getLinearDepthFactors(Camera cam) {
float near = cam.getFrustumNear();
float far = cam.getFrustumFar();
float a = far / (far - near);
float b = far * near / (near - far);

    return new Vector2f(a,b);
}

[/java]

Now that there is a closed mesh representing the light volume, we need to calculate how much light this volume will contribute to the scene, this is figured out by solving the scattered air light integral, or, how much light bounces off the little bits of dust and crap inside the volume, that we see as light shafts. This calculation has some brutal maths, the best approach I have found is A Practical Analytic Single Scattering Model for Real Time Rendering and it makes my brain hurt! I found a super useful blog post from Miles Macklin, which really helped me get my head around the complex mathematics involved in solving the integral, and presented a much simpler model. But first, taking a step back, we need to figure out the thickness of the light volume at any given point. In my solution, this needs to be done on the GPU since the volume mesh has been manipulated by the vertex shader, so the CPU does not know about the changes made to the volume, so it will be both far quicker than having to transmit this information to the GPU every frame, and the GPU is optimised for these kinds of calculations.

Calculating Geometry "Thickness" on the GPU

This is done with a simple bit of vector maths, which I can best described using the following scenario (please excuse the crappy diagrams):

This is looking at the side of a simple scene, we are interested in calculating the depth of the volume along the ray coming from the eye.

We assign the following points i A B C D and E to the points where the ray starts, intersects the volume and finishes.

These vectors are labelled iA iB iC etc ... iE can be ignored as it has no effect on the calculations. We are interesting in calculating the length of AB + CD.

We can figure out the thickness by subtracting iA and iC from iB and iD. Interestingly, iB and iD are both back faces of the mesh, and iA and iC are both front faces. Now we can simply solve the air light scattering integral at at all the back faces, and subtract the solved values at the front faces!

To do this with the volume mesh, we need to setup the light volume’s material, and by setting FaceCullMode to None, and setting Depth Testing to Off:

Tip : setting Depth Testing to Off will force Depth Writing to always be Off! Depth Writing will always be off regardless of its value. This is part of the OpenGL specification and makes perfect sense when you think about it... so don't bug @Momoko_Fan about this since this is the desired behaviour and if you're lucky, he will spare your life for wasting his time.

All faces of the volume mesh will be rendered. It is now possible to calculate the thickness at each face for each “light shaft”, and incidentally add or subtract them from the a texture using BlendMode.Additive.

The volume light is rendered in a post Filter.Pass after the scene has been rendered…

[java]lightVolumePass.init(renderManager.getRenderer(), w, h, Format.RGBA32F, Format.Depth, 1, true);[/java]

This pass needs to use a non standard texture type: Format.RGBA32F, which allows us to store larger and negative values.

Tip : Front or Back face in fragment shader: gl_FrontFacing

Rendering Selected Geometry in an Addition Filter Pass

I have figured out 2 methods to do this, the first uses materials with specific render techniques, which I explained in my post about screen space displacement filter .

I figured out a simpler method after looking at some of @pspeed 's work… have a reference to the geometry in the filter, and render only that …

[java]
renderManager.getRenderer().setFrameBuffer(lightVolumePass.getRenderFrameBuffer());
renderManager.getRenderer().clearBuffers(true, true, true);
renderManager.renderGeometry(lightVolume);
[/java]

Since needed my geometry to be positioned using LocalTranslations, I needed to add the spatial it to the rootNode, and to make sure it is ignored during the initial scene render, I set CullHint to Always.

Gotcha! Make sure to reset viewport camera in post pass filters! This gotcha tricked me on two separate occasions: The first time I thought my filter wasn't rendering at all, it turned out my filter pass was being rendered at an odd angle in the bottom left corner of the screen... because I had set the renderManager camera to something else, elsewhere in other code.

I was fooled again when I attempted to add 3 separate VolumetricLightFilters…only the first filter was being rendered, the other two appeared not to render, but were hidden in the bottom left corner of the screen.

So before rendering a pass, make sure the camera is set correct eg renderManager.setCamera(viewPort.getCamera(), false);

Current Limitations

Changing the spot Light Radius, near distance or far distance, requires the volume mesh to be regenerated on the CPU, so this is not supported. This could be performed on the GPU and allow for real-time updates, but it would also require many addition calculations per vertex, per frame, which seamed like a waste to me, so this functionality was simply not included. If there is a decent use case for this, there is no reason why it could not be added.

The “fog” that is causing the light to scatter, so the light shafts can be seen, is considered to be an isotropic or homogeneous media, which is a fancy way of saying its consistent, like water with some milk completely mixed in. The current integral calculation simply does not allow for a non homogeneous media, like a smoke coming from a fire, or a cloudy sky,. The maths to calculate this is waaaaay beyond me at this stage. It is possible to simulate this effect using particles, preferably soft particles, I may look into this a bit later.

A colored Gobo or Cookie are not possible using the current integration, more on this further down.

Future plans and thoughts

  • There is room to add a simple cookie or Gobo which would be used to control the shape of emitted light. At the moment it is assumed the shape is a circle since it is being emitted from a spot light. I will add this shortly as it's rather trivial.
  • This effect is pretty dull without an actual Spot Light to illuminate surfaces, and Spot Light often looks silly without a shadow renderer, plus the fact the shadow filter and the volumetric light filter both share the same SceneProccessor it only makes sense to make the VolumetricLightFilter and extension of the SpotShadowRenderer.
  • 6 of these filters could be linked together to create an omni or point light volumetric light, and extend the PointLightShadowFilter.
  • I would like to create a cut down version that renders a simple cone volume that does not create shafts, this would bypass the shadowRenderer scene processor and would be a lot faster since there is no addition depth render, and there would never be more than 2 faces to render per fragment.
  • I was wondering if it was possible to use the LightViewProjectionMatrix to figure out the light position within the shader. I tried multiplying vec4(0.0) and vec4(1.0) by it but that didn't work, surely this is possible, and it would avoid the need to pass in light and camera positions... anyone ?
  • I bit of blur would help mask any aliasing artefacts from low volume resolutions.
  • The often horribly abused post effect Bloom is an ideal partner for this filter, as it could be used to simulate the scattering of diffuse light from brightly lit surfaces.

So here is my Volumetric Lighting Filter, that is how I made it, links to the source are listed above, learn from it, add to it, improve it, then hit up this thread with your ideas and improvements so we can add a slick new filter to the project =)

Cheers
James

Pro Tip: Editing Posts For whatever reason I couldn't find a way to edit my post, which made me a very sad panda.. so I added /edit to the end of this pages url and it gave me editing access, may work for you.
43 Likes

This is really awesome work! Good job! =)

Amazing, simply amazing

Wow, great work, and excellent explanation.

One thing I wonder about, since you use the basicShadow renderer as a basis, and a similar logic, does it have stabilisation issues as well ?
@nehon Do you know if the stabilisation is also in the Basic one or only in the PSSM one??
It might be a idea, to use the other shadow renderer as a basis, and abuse the smoothing it has, for softer light corners.


…and the explanation.

Wow!!

Just… :-o thanks!

Wow… this is the kind of contribution we all need . Just keep it rocking man!

That’s really great!! We need to talk
@erlend_sh we need a tech blog for this kind of post…it would be too bad that it get lost in the abyss of the forum…

1 Like
@nehon said: That's really great!! We need to talk @erlend_sh we need a tech blog for this kind of post...it would be too bad that it get lost in the abyss of the forum...
I don't think it will XD

Woooahh, just wow :-o . Most thorough jME post I’ve read. Thanks and good job

Ok.

Now I really want to make an occulus rift horror game :smiley:

Good job - impressive visuals :slight_smile:

I re-read this post a few times, wish I could give a thumbs up for every time :slight_smile:

Reminds me of the work by Ulf Assarson
http://www.realtimeshadows.com/
Here is a PPT
http://www.realtimeshadows.com/sites/default/files/sig2013-course-volumetricshadows_3_online.pptx_.zip

Anyone successful in using the VLF in his/her own project?

My result is a little bit weird:

From my understanding i just have to add the VLF:
[java]
VolumeLightFilter vsf = new VolumeLightFilter(spot, 128, rootNode);
filterPostProcessor.addFilter(vsf);
[/java]

Probably i’m missing something. At least the “demo” works like intended.

Edit:
Looks better now that i deactivated my shadows. So i probably misconfigured them.

Hi, congratulations and thank you for this great work. Tough, I’ve wanted to post this for a couple months now: I’ve tested this under a large variety of graphics cards in all brands, all budgets and the only graphics card not supporting it so far is the Intel® HD Graphics Family / 3.0.0

JME3 reports that card as: FrameBuffer, FrameBufferMRT, FrameBufferMultisample, OpenGL20, OpenGL21, OpenGL30, ARBprogram, GLSL100, GLSL110, GLSL120, GLSL130, VertexTextureFetch, TextureArray, FloatTexture, FloatColorBuffer, FloatDepthBuffer, PackedFloatTexture, SharedExponentTexture, PackedFloatColorBuffer, NonPowerOfTwoTextures, MeshInstancing, VertexBufferArray, Multisample, PackedDepthStencilBuffer

Here’s the stack trace error, can you help fix this, @thetoucher ?

[java]

Uncaught exception thrown in Thread[LWJGL Renderer Thread,5,main]
com.jme3.renderer.lwjgl.LwjglRenderer.checkFrameBufferError(LwjglRenderer.java:1335)
com.jme3.renderer.lwjgl.LwjglRenderer.setFrameBuffer(LwjglRenderer.java:1606)
mygame.VLRenderer.renderShadowMap(VLRenderer.java:160)
mygame.VLRenderer.postQueue(VLRenderer.java:133)
mygame.VolumeLightFilter.postQueue(VolumeLightFilter.java:151)
com.jme3.post.FilterPostProcessor.postQueue(FilterPostProcessor.java:215)
com.jme3.renderer.RenderManager.renderViewPort(RenderManager.java:979)
com.jme3.renderer.RenderManager.render(RenderManager.java:1029)
com.jme3.app.SimpleApplication.update(SimpleApplication.java:252)
com.jme3.system.lwjgl.LwjglAbstractDisplay.runLoop(LwjglAbstractDisplay.java:151)
com.jme3.system.lwjgl.LwjglDisplay.runLoop(LwjglDisplay.java:185)
com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:228)
java.lang.Thread.run(Unknown Source)

[/java]

Seems like you left the exception out. And if it was a shader compile error then you left out the shader source dump, too.

Oops sorry, it was: IllegalStateException: Incomplete Draw Buffer

It didn’t say anything about the shader compile error. Just a regular exception.

EDIT: I should also add that on a HP laptop with a Intel® HD Graphics / 2.1.0 card it works fine (it’s slow, but it works without crashing even for an extended period of render time).

EDIT #2: Same thing for another laptop with a Intel® HD Graphics / 3.1.0 card, it works fine.

Hi @thetoucher
Thanks for article and thanks for sharing.
Just downloaded source code but have a few problems running test on JME 3.2.
Can you help me to figure it out or may you provide download link for updated source which works with JME 3.1+ ? Thanks

first .getShadowQueueContent(RenderQueue.ShadowMode.Cast); is not available in JME 3.1+ I changed it to :

for (Spatial scene : viewPort.getScenes()) {
            ShadowUtil.getGeometriesInCamFrustum(scene, viewPort.getCamera(), ShadowMode.Cast, occluders);
        }
        
        for (Spatial scene : viewPort.getScenes()) {
            ShadowUtil.getGeometriesInCamFrustum(scene, viewPort.getCamera(), ShadowMode.Receive, sceneReceivers);
        }

now when running i get :

run:
Nov 29, 2016 5:10:47 PM com.jme3.system.JmeDesktopSystem initialize
INFO: Running on jMonkeyEngine 3.2-1
 * Branch: master
 * Git Hash: cd70630
 * Build Date: 2016-10-04
Nov 29, 2016 5:10:47 PM com.jme3.system.lwjgl.LwjglContext printContextInitInfo
INFO: LWJGL 2.9.3 context running on thread jME3 Main
 * Graphics Adapter: null
 * Driver Version: null
 * Scaling Factor: 1
Nov 29, 2016 5:10:47 PM com.jme3.renderer.opengl.GLRenderer loadCapabilitiesCommon
INFO: OpenGL Renderer Information
 * Vendor: Intel Open Source Technology Center
 * Renderer: Mesa DRI Intel(R) Sandybridge Mobile 
 * OpenGL Version: 3.0 Mesa 13.1.0-devel
 * GLSL Version: 1.30
 * Profile: Compatibility
Nov 29, 2016 5:10:47 PM com.jme3.asset.AssetConfig loadText
WARNING: Cannot find loader com.jme3.scene.plugins.blender.BlenderModelLoader
Nov 29, 2016 5:10:47 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio Renderer Information
 * Device: OpenAL Soft
 * Vendor: OpenAL Community
 * Renderer: OpenAL Soft
 * Version: 1.1 ALSOFT 1.15.1
 * Supported channels: 64
 * ALC extensions: ALC_ENUMERATE_ALL_EXT ALC_ENUMERATION_EXT ALC_EXT_CAPTURE ALC_EXT_DEDICATED ALC_EXT_disconnect ALC_EXT_EFX ALC_EXT_thread_local_context ALC_SOFT_loopback
 * AL extensions: AL_EXT_ALAW AL_EXT_DOUBLE AL_EXT_EXPONENT_DISTANCE AL_EXT_FLOAT32 AL_EXT_IMA4 AL_EXT_LINEAR_DISTANCE AL_EXT_MCFORMATS AL_EXT_MULAW AL_EXT_MULAW_MCFORMATS AL_EXT_OFFSET AL_EXT_source_distance_model AL_LOKI_quadriphonic AL_SOFT_buffer_samples AL_SOFT_buffer_sub_data AL_SOFTX_deferred_updates AL_SOFT_direct_channels AL_SOFT_loop_points AL_SOFT_source_latency
Nov 29, 2016 5:10:47 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
WARNING: Pausing audio device not supported.
Nov 29, 2016 5:10:47 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio effect extension version: 1.0
Nov 29, 2016 5:10:47 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio max auxiliary sends: 4
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.animation.SkeletonControl controlRender
INFO: Hardware skinning engaged for JaimeGeom-ogremesh (Node)
Nov 29, 2016 5:10:50 PM com.jme3.renderer.opengl.GLRenderer updateShaderSourceData
WARNING: Bad compile of:
1	#version 110
2	#define FRAGMENT_SHADER 1
3	uniform vec4 g_ViewPort;
4	
5	uniform sampler2D m_SceneDepthTexture;
6	uniform sampler2D m_ShadowMap;
7	uniform sampler2D m_ShadowDepthMap;
8	uniform vec2 m_LinearDepthFactorsCam;
9	uniform vec3 m_CameraPos;
10	uniform vec3 m_LightPos;
11	
12	uniform float m_LightIntensity;
13	uniform vec4 m_LightColor;
14	
15	varying vec4 projCoord;
16	varying vec4 posInCS;
17	varying vec4 posInWS;
18	
19	
20	
21	float ScatteringIntegral(vec3 cameraPos, vec3 lightPos, vec3 direction, float thickness) {
22	    
23	    vec3 lightToCam = cameraPos - lightPos;
24	
25	    // coefficients
26	    float direct = dot(direction, lightToCam);
27	    float scattered = dot(lightToCam, lightToCam);
28	    //c = length(lightToCam)*length(lightToCam);
29	
30	    // evaluate integral
31	    float scattering = 1.0 / sqrt(scattered - direct*direct);
32	    return scattering*(atan( (thickness+direct)*scattering) - atan(direct*scattering));
33	}
34	
35	void main() {
36	
37	    vec2 uv = vec2(gl_FragCoord.x/g_ViewPort.z, gl_FragCoord.y/g_ViewPort.w);
38	
39	    float depthInCS = (m_LinearDepthFactorsCam.y / (texture2D(m_SceneDepthTexture, uv).r - m_LinearDepthFactorsCam.x));
40	    vec3 viewRay = posInWS.xyz - m_CameraPos;
41	
42	    float volumeDepth = length(viewRay);
43	    viewRay /= volumeDepth;
44	
45	    volumeDepth = min(volumeDepth, depthInCS); // clamp to scene depth
46	
47	    float scatteringCoefficient = ScatteringIntegral(m_CameraPos, m_LightPos, viewRay, volumeDepth)*m_LightIntensity;
48	
49	    if (gl_FrontFacing) {
50	        scatteringCoefficient*=-1;
51	    }
52	
53	    gl_FragColor = m_LightColor * scatteringCoefficient;
54	}

Nov 29, 2016 5:10:50 PM com.jme3.app.LegacyApplication handleError
SEVERE: Uncaught exception thrown in Thread[jME3 Main,5,main]
com.jme3.renderer.RendererException: compile error in: ShaderSource[name=Shaders/VolumetricLight.frag, defines, type=Fragment, language=GLSL100]
0:50(2): error: could not implicitly convert operands to arithmetic operator
0:50(2): error: could not implicitly convert error to float

	at com.jme3.renderer.opengl.GLRenderer.updateShaderSourceData(GLRenderer.java:1203)
	at com.jme3.renderer.opengl.GLRenderer.updateShaderData(GLRenderer.java:1230)
	at com.jme3.renderer.opengl.GLRenderer.setShader(GLRenderer.java:1294)
	at com.jme3.material.logic.DefaultTechniqueDefLogic.render(DefaultTechniqueDefLogic.java:94)
	at com.jme3.material.Technique.render(Technique.java:166)
	at com.jme3.material.Material.render(Material.java:970)
	at com.jme3.renderer.RenderManager.renderGeometry(RenderManager.java:616)
	at mygame.VolumeLightFilter.postFrame(VolumeLightFilter.java:175)
	at com.jme3.post.FilterPostProcessor.renderFilterChain(FilterPostProcessor.java:273)
	at com.jme3.post.FilterPostProcessor.postFrame(FilterPostProcessor.java:320)
	at com.jme3.renderer.RenderManager.renderViewPort(RenderManager.java:1111)
	at com.jme3.renderer.RenderManager.render(RenderManager.java:1154)
	at com.jme3.app.SimpleApplication.update(SimpleApplication.java:253)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.runLoop(LwjglAbstractDisplay.java:151)
	at com.jme3.system.lwjgl.LwjglDisplay.runLoop(LwjglDisplay.java:197)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:232)
	at java.lang.Thread.run(Thread.java:745)

I am not a shader guy :confused:, can you please help to fix this ?

Where did you find the source codes? Links at code.google.com are dead as well as that one on dropbox.

http://dropbox.mtheorygame.com/

2 Likes