Light and Shadow Blending.. neat idea with awesome results

This is more for @nehon than anyone else, however… others may find the idea useful… or at least get you thinking about how to get more realistic looking shadows without killing frame rate on multiple shadow maps + a million lookups for dithering.

A couple things I noticed while playing around with deferred rendering:

  1. Shadow mapping KILLS specular lighting.
  2. Shadow mapping KILLS specular lighting.
  3. I think you get the idea.

Sooooo… the solution I cam up with was this:

  1. As lighting is being applied, blend back in the diffuse sum into the shadow map.
  2. Do NOT add the shadows to ambient lighting… this sort of defeats the purpose of ambient light.
  3. Apply the shadowdiffuse sum to diffusediffuse sum.
  4. Apply the shadowdiffuse sum to specularspecular sum.

There is a vid at the end, which will hopefully show how dramatic the difference is specular lighting… as well as overall ambient lighting.

However, I was thinking that this same technique could be incorporated into JME with or without the use of deferred rendering as shadow rendering could potentially be moved to happen prior to scene rendering. and the outputs of these could be handed to the lighting material. With a single added texture lookup the results are AMAZING!

Anyways… enough of my rambling. Hopefully the vid at fullscreen will show what I am talking about.

Oh… ignore the SSAO. It is being blended over the entire scene as I haven’t completely decided the best way of blending it into the lighting. Obviously it should effect ambient lighting… however, the diffuse lighting should be blended back into the AO before this… and likely be applied to diffuse lighting as well… not sure… soooo ignore it!

Here is the output of the idea. It went from what seemed like ZERO specular reflection back to what it looked like prior to applying shadow maps.

On a side note… the shadow maps are half the resolution they were in the previous vid with (what I think) is better results.

[video]http://youtu.be/kKUM5PAcLgk[/video]

Anyways… if you have some time to think over the idea, let me know if you think this is possible outside of deferred rendering.

3 Likes

Not sure what you mean by “KILLS specular lighting”. If it does and it shouldn’t , or if it doesn’t and it should :p.

Right now in JME, specular is modulated by shadows as diffuse and ambient are. It’s wrong, but it’s due to how shadows are rendered, with an additional pass after the lighting pass has been done.
For example, you can see specular lighting in a shadowed area. It’s darker, but you can definitely see it.
Your solution seems to be applicable to the forward rendering process yeah, also @thetoucher did some tests with feeding the lighting pass with the shadow information and render the shadows during the lighting pass. This allows to do the same kinds of things you do, by properly modulating specular with the shadow factor.

I’m not sure you considered one case though. You should not see specular of a light in it’s own shadow. I mean if a light casts shadows, it’s specular should be voided in the shadows it casts (not sure I’m clear).
Though to do this, you need the information of what light casts what shadow, and I guess that information is lost when you pre render the full screen shadow map. not sure how you could pass the information…maybe by storing the light index in the alpha channel of the shadow map…

One sure thing though is that your lighting looks damn good. that’s great work really.

I’m gonna think about it , and how to integrate this in the forward rendering.

Thanks

@nehon said: I'm not sure you considered one case though. You should not see specular of a light in it's own shadow. I mean if a light casts shadows, it's specular should be voided in the shadows it casts (not sure I'm clear). Though to do this, you need the information of what light casts what shadow, and I guess that information is lost when you pre render the full screen shadow map. not sure how you could pass the information...maybe by storing the light index in the alpha channel of the shadow map....

Right now… this is the major holdup on finishing point light shadows.

One thought I had was to render 4 pointlights shadows per shadow map (one in each channel) in the order they are stored in the light manager. And actually… the same would work for directionals and spots, however I just nixed the directional lights effecting other directional light’s shadows, as the chances of people having more than a single sun in there scene is less likely than multiple points or spots.

I’ve been trying to clean up loose ends with the stuff I have done thus far… actually… any excuse to put off finishing up the point light shadow mapping because of just this problem (I think I have rewritten the ssao filter 5 times now >.< ). I’m hoping the single channel per light idea works.

It’s a lot of empty space if there are less than 4 point lights casting shadows, but I guess no more than one shadow map per image.

@nehon
Oh yeah… one last thought. You said “Right now in JME, specular is modulated by shadows as diffuse and ambient are. It’s wrong, but it’s due to how shadows are rendered, with an additional pass after the lighting pass has been done”

Everything is an approximation anyways (i.e. per pixel phong shading is wrong from the get go as it has no idea that a normal facing towards the light could be self shadowed), so… even if some is completely inaccurate in terms of realism… if it produces a more desirable effect (or even… just a cooler effect), you’re better off doing it wrong!

@nehon
I just ran a quick test with the idea above like so: (simplified to 4 lights for explanations sake)

  1. During the preShadow pass, I pass in a channel index and render the depth from the light to the single channel specified in the depth buffer.
  2. During the post shadow process (and this is were it got REALLY cool), dithering the edges effected 4 shadow maps at the cost of a single shadow map. Each lookup was pulling 4 individual maps.
  3. Outputted the results to a single channel within color buffer.
  4. Passed this into the lighting process and read each channel as it’s own shadow map, allowing me to negate the light that it was cast by.

There is still a small issues that needs to be handled, an array of indexes would need to be passed into the lighting shader telling which lights had shadow casting enabled. The above test assumed all lights were casting. But this allows for a better dithering process in the post shadow shader, still negating a lot of the cost of multiple shadow casters.

@t0neg0d said: @nehon Oh yeah... one last thought. You said "Right now in JME, specular is modulated by shadows as diffuse and ambient are. It’s wrong, but it’s due to how shadows are rendered, with an additional pass after the lighting pass has been done"

Everything is an approximation anyways (i.e. per pixel phong shading is wrong from the get go as it has no idea that a normal facing towards the light could be self shadowed), so… even if some is completely inaccurate in terms of realism… if it produces a more desirable effect (or even… just a cooler effect), you’re better off doing it wrong!


Yep that was exactly the point. though is some case it can be very obvious that it’s wrong, and can catch the eye of the player. Right now for example, the shadows are not lighten by other light source (I guess you’re aware of that, since that’s one of the feature of your deferred process).

@t0neg0d said: @nehon I just ran a quick test with the idea above like so: (simplified to 4 lights for explanations sake)
  1. During the preShadow pass, I pass in a channel index and render the depth from the light to the single channel specified in the depth buffer.
  2. During the post shadow process (and this is were it got REALLY cool), dithering the edges effected 4 shadow maps at the cost of a single shadow map. Each lookup was pulling 4 individual maps.
  3. Outputted the results to a single channel within color buffer.
  4. Passed this into the lighting process and read each channel as it’s own shadow map, allowing me to negate the light that it was cast by.

There is still a small issues that needs to be handled, an array of indexes would need to be passed into the lighting shader telling which lights had shadow casting enabled. The above test assumed all lights were casting. But this allows for a better dithering process in the post shadow shader, still negating a lot of the cost of multiple shadow casters.


Sounds pretty cool.
only issue I see with the 4 shadow maps in one texture is that you have only 8bit shadow maps (assuming you used a RGBA8), which is very poor regarding precision for a depth buffer. You’ll have issue comparing depth when the light source is far from the caster and receiver objects and specially when the caster and the receiver are close to each other. You may have to use a RGBA16f or even RGBA32f. But if you only have one map, that’s a lot of wasted memory…
Maybe you could have a system that choose the right format depending of the number of shadow casting lights.
This should be tested though, maybe I’m wrong. If the issue occur you should see some bandings in the shadows that are far away from the light source.

@nehon said: Yep that was exactly the point. though is some case it can be very obvious that it's wrong, and can catch the eye of the player. Right now for example, the shadows are not lighten by other light source (I guess you're aware of that, since that's one of the feature of your deferred process).

Sounds pretty cool.
only issue I see with the 4 shadow maps in one texture is that you have only 8bit shadow maps (assuming you used a RGBA8), which is very poor regarding precision for a depth buffer. You’ll have issue comparing depth when the light source is far from the caster and receiver objects and specially when the caster and the receiver are close to each other. You may have to use a RGBA16f or even RGBA32f. But if you only have one map, that’s a lot of wasted memory…
Maybe you could have a system that choose the right format depending of the number of shadow casting lights.
This should be tested though, maybe I’m wrong. If the issue occur you should see some bandings in the shadows that are far away from the light source.

Looks as if Depth lets the video hardware choose the precision.

This is used by the post shadow shader to compare depth values to the original scenes depth texture which is (hopefully) the same format.

It’s only written to a color buffer at the point the depth comparison is done and the shader determines if it is a shadow or self-shadow.

Instead of packing the depth into another image, I cut corners in other areas and left depth alone as it is used by countless post process effects.

@t0neg0d said: It's only written to a color buffer at the point the depth comparison is done and the shader determines if it is a shadow or self-shadow.
ohhh ok, got it now! yeah so forget what I said :p