Here are the shader replacements. It produces a much faster, slightly more accurate shadow/self-shadow map by removing all of the multisample comparisons and border checks… this leaves a lot of room for border blurring and cleanup.
EDIT: Added a quick edit to PostShadow.frag to knock out noise… left the original line commented above the edit.
Here are the shaders…
PreShadow.vert:
[java]attribute vec4 inPosition;
attribute vec2 inTexCoord;
uniform mat4 g_WorldViewProjectionMatrix;
uniform mat4 g_WorldViewMatrix;
varying vec2 texCoord;
varying vec4 v_position;
void main(){
gl_Position = g_WorldViewProjectionMatrix * inPosition;
texCoord = inTexCoord;
v_position = gl_Position;
}[/java]
PreShadow.frag
[java]varying vec4 v_position;
void main() {
float depth = v_position.z / v_position.w ;
depth = depth * 0.5 + 0.5;
float moment1 = depth;
float moment2 = depth * depth;
float dx = dFdx(depth);
float dy = dFdy(depth);
moment2 += 0.25*(dxdx+dydy) ;
gl_FragColor = vec4( moment1,moment2, 0.0, 0.0 );
}[/java]
PostShadow.vert - don’t think this one changed
[java]uniform mat4 m_LightViewProjectionMatrix;
uniform mat4 g_WorldViewProjectionMatrix;
uniform mat4 g_WorldMatrix;
varying vec4 projCoord;
attribute vec3 inPosition;
const mat4 biasMat = mat4(0.5, 0.0, 0.0, 0.0,
0.0, 0.5, 0.0, 0.0,
0.0, 0.0, 0.5, 0.0,
0.5, 0.5, 0.5, 1.0);
void main(){
gl_Position = g_WorldViewProjectionMatrix * vec4(inPosition, 1.0);
vec4 worldPos = g_WorldMatrix * vec4(inPosition, 1.0);
vec4 coord = m_LightViewProjectionMatrix * worldPos;
projCoord = biasMat * coord;
}[/java]
PostShadow.frag
[java]uniform sampler2D m_ShadowMap;
varying vec4 projCoord;
varying vec4 color;
vec4 ShadowCoordPostW;
float chebyshevUpperBound(in float dist) {
float blur = 0.000175;
vec2 moments = texture2D(m_ShadowMap,ShadowCoordPostW.xy).rg;
if (dist <= moments.x)
return 1.0 ;
float variance = moments.y - (moments.xmoments.x);
variance = max(variance,0.002);
float d = dist - moments.x;
// float p_max = variance / (variance + dd);
// Quick edit to knock out noise
float p_max = smoothstep(0.0,0.99999999,(variance / (variance + d*d)));
return p_max;
}
void main()
{
ShadowCoordPostW = projCoord / projCoord.w;
float shadow = chebyshevUpperBound(ShadowCoordPostW.z);
if (shadow < 1.0) shadow *= 0.75;
// for solid shadows use: if (shadow < 1.0) shadow = 0.75;
gl_FragColor = vec4(vec3(shadow),1.0);
}[/java]
You can see that the post shadow frag has no need for Shadow.glsl and is incredibly simplified all the way around producing better results only taking one sample.
@nehon @Momoko_Fan
Yep… I’m sure you’re sick of the subject, but I have a few more findings I thought you may be interested in.
If you turn Front face culling on in the PreShadow.j3md and do the distance check like:
[java]
if (depth - 0.0018 <= lookupDepth && depth + 0.0018 >= lookupDepth)
//and
if (depth - 0.0018 > lookupDepth)
[/java]
to apply the shadow, this will fill in the holes projected by front face rendering for the shadow depth texture (i.e. no more line in the self shadow for where the lid of the teapot is separated from the body). This does not effect quads, or open meshes.
Couple more findings while looking into making this more usable for all different scale models. A better way of eliminating Moiré pattern in the chebyshevUpperBound is to add a slight bias to the distance check against shadow samples (and only this check):
EDIT: heh… just noticed I left a variable in there from testing blur… my bad 
[java]float chebyshevUpperBound(in float dist) {
vec2 moments = texture2D(m_ShadowMap,ShadowCoordPostW.xy).rg;
if (dist - 0.005 <= moments.x) // THIS LINE : 0.005 being the pattern filter bias
return 1.0 ;
float variance = moments.y - (moments.xmoments.x);
variance = max(variance,0.002);
float d = dist - moments.x;
// Reverting this change as the above is a better solution
float p_max = variance / (variance + dd);
// Quick edit to knock out noise
// float p_max = smoothstep(0.0,0.99999999,(variance / (variance + d*d)));
return p_max;
}[/java]
In a best of all worlds approach, the BSR would allow the pattern filter bias to be adjustable, making it completely useful in all model scales.
In the test using the crappy teapot model… a bias of 0.005 works wonderful.
In my game… the bias needs to be adjusted to 0.0005 to eliminate the pattern noise without destroying half of the shadows on small objects.
You might be able to obtain more precision on the variance map by using 16 bit or 16 bit float, by setting the color texture on the shadow framebuffer to use RGBA16 or RGBA16F formats, I can’t imagine it working correctly with only 8 bits of precision.
Also the improvement that should be more apparent from this algorithm is the free PCF. I would imagine the border checks are trivial as the shadow shaders already don’t do much ALU. As for the multisampling, I am not sure what you mean, since the shadow shader does not resolve MS.
@Momoko_Fan said:
You might be able to obtain more precision on the variance map by using 16 bit or 16 bit float, by setting the color texture on the shadow framebuffer to use RGBA16 or RGBA16F formats, I can't imagine it working correctly with only 8 bits of precision.
Also the improvement that should be more apparent from this algorithm is the free PCF. I would imagine the border checks are trivial as the shadow shaders already don't do much ALU. As for the multisampling, I am not sure what you mean, since the shadow shader does not resolve MS.
Hahahaa... I have NO CLUE what you just said!
But... the BSR calls Shadow_GetShadow which calls Shadow_DoDither_2x2 which calls Shadow_DoShadowCompareOffset x4 which does texture sampling. Isn't this multi-sampling?
It, however, does not call the border check routine... which I thought it did.
I can post a video of the above example... the artifacting is considerably less than the original at 8 bit. However, the ability to adjust the noise filter bias would be semi-critical IMO, as it allows you to fine tune the output for your purposes.
I integrated this with the changes into the game I am working on atm... and I am more pleased with the results then I was with the PSSM. I really would like to see blurring as an option, but aside from this little tidbit, I was able to turn CastAndRecieve on for all objects in the scene and get nice looking results (after fine tuning the bias).
I tried the original and the results were not very pleasing to the eye, unfortunately :(
Btw… can you give me a little hint as to limitations and the use of 8bit (more of a general why it is used)?
And PCF & ALU? << what are there 
I think this would be a really cool addition to the current PssmShadowRenderer, and I am sure @nehon probably already has VSM in his to-do list so this would knock one item from it ;). From API perspective it would be easier to expose another FilterMode that would activate VSM - basically a higher precision color map for the shadow map and this new filter mode in the Pre/Post shader.
Regarding the 8bit vs 16bit - it seems you don’t really understand what’s going on, and actually in my previous post I made some assumptions that were wrong so let’s start again …
In your PostShadow.frag shader, what is m_ShadowMap? Is it a depth map (with Format.Depth) or a color map (with Format.RGB***)?
@Momoko_Fan said:
I think this would be a really cool addition to the current PssmShadowRenderer, and I am sure @nehon probably already has VSM in his to-do list so this would knock one item from it ;). From API perspective it would be easier to expose another FilterMode that would activate VSM - basically a higher precision color map for the shadow map and this new filter mode in the Pre/Post shader.
Regarding the 8bit vs 16bit - it seems you don't really understand what's going on, and actually in my previous post I made some assumptions that were wrong so let's start again ..
In your PostShadow.frag shader, what is m_ShadowMap? Is it a depth map (with Format.Depth) or a color map (with Format.RGB***)?
Regarding the 8bit vs 16bit - it seems you don't really understand what's going on, and actually in my previous post I made some assumptions that were wrong so let's start again .. (This is freakin' hysterical) I actually was confused by your original post...
@Momoko_Fan said:
I can't imagine it working correctly with only 8 bits of precision.
However, it does produce desirable (and what seem to be accurate) results. Now... whether or not 16bit precision would produce different results, is a different story. I hope this helps clarify the question after.
As for m_ShadowMap... I changed nothing inside of the BasicShadowRenderer class or the j3md files. so it is still a depth texture.
The questions mostly stem from my lack of speaking acronym-ese. I avoid them when I can :)
I think we skipped this point:
But… the BSR calls Shadow_GetShadow which calls Shadow_DoDither_2x2 which calls Shadow_DoShadowCompareOffset x4 which does texture sampling. Isn’t this multi-sampling?
Am I not understanding this correctly?
@t0neg0d said:
However, it does produce desirable (and what seem to be accurate) results. Now... whether or not 16bit precision would produce different results, is a different story. I hope this helps clarify the question after.
As for m_ShadowMap... I changed nothing inside of the BasicShadowRenderer class or the j3md files. so it is still a depth texture.
Can you post a screenshot with the teapot, showing the "desirable" results? The whole point of VSM is to give free PCF, otherwise you're just losing performance for nothing.
@t0neg0d said:
But… the BSR calls Shadow_GetShadow which calls Shadow_DoDither_2x2 which calls Shadow_DoShadowCompareOffset x4 which does texture sampling. Isn’t this multi-sampling?
It's not multisampling. There are two methods of "blurring" the shadows currently used by the shadow renderer, one is to use dithering (what you're referring to) and the other is to use PCF. Multisampling refers to something completely different that has nothing to do with shadows.
The original: (Note all the noise on the side of the teapot with no shadow. It looks like Phong shading)
The code above: (Note no noise on the side of the teapot with no shadow) And… yes… I blotted the SoftShadows for this screenshot to give a better comparison.
Ah… MS is used in anti-aliasing… I guess “without multiple texture lookups” would have been more accurate.
@Momoko_Fan said:
Can you post a screenshot with the teapot, showing the "desirable" results? The whole point of VSM is to give free PCF, otherwise you're just losing performance for nothing.
Screenshots above.
I’m still really confused how 3 less texture lookups could be a performance hit.
@t0neg0d said:
I’m still really confused how 3 less texture lookups could be a performance hit.
The screenshot with your VSM implementation has no PCF, which is what it is supposed to accomplish. The lack of noise is probably because you introduced an offset to the depth, which was already done for you with the PolyOffset render state. The original BSR does not have the artifacts you show in the screenshot.
@Momoko_Fan said:
The screenshot with your VSM implementation has no PCF, which is what it is supposed to accomplish. The lack of noise is probably because you introduced an offset to the depth, which was already done for you with the PolyOffset render state. The original BSR does not have the artifacts you show in the screenshot.
Whaaaaaaaa??? That is a screen shot of the original BSR. Does not have artifacts??? So you're saying the screen shot is a fake? lol
And did you read what I said? I blotted out the softshadows to show a comparison to the old version.
I could care less whether you find this code useful or not... I just threw it out there... but to deny that the original has the artifacting that is CLEARLY in the screenshot makes me wonder whether what you are saying isn't ego driven (which is not helpful at all and will lead to making EXTREMELY poor choices in anything you pursue). Please tell me you misspoke yourself and didn't realize that the first screenshot IS the original.
If not... did you bother testing the original at higher resolution texture sizes?
Well to be honest, that are artifacts you can see with the bsr, though most of the time, they are not taht visible
@Momoko_Fan
@EmpirePhoenix
I thought maybe a comparison in application might point out why I was trying to fix this to begin with. I took a screen shot of the original and the code above used in my game. Do note I have no implementation of blurring, dithering, etc in mine… as I was just posting the finding for someone else to make use of.
THE ORIGINAL… AKA NOT MINE… AKA THE BSR WITH NO ARTIFACTING >.< Just clarifying so it isn’t missed again. Note the large amount of artifacting on the model even though we are viewing the side with no self shadowing.
The code above
If you open each image in a new window/tab… you will be able to see it at the actual size and get a better comparison
@t0neg0d said:
Whaaaaaaaa??? That is a screen shot of the original BSR. Does not have artifacts??? So you're saying the screen shot is a fake? lol
And did you read what I said? I blotted out the softshadows to show a comparison to the old version.
I could care less whether you find this code useful or not... I just threw it out there... but to deny that the original has the artifacting that is CLEARLY in the screenshot makes me wonder whether what you are saying isn't ego driven (which is not helpful at all and will lead to making EXTREMELY poor choices in anything you pursue). Please tell me you misspoke yourself and didn't realize that the first screenshot IS the original.
If not... did you bother testing the original at higher resolution texture sizes?
It's not fake, just a modified version of jME3. Like I said, in a previous thread you posted a modification to PreShadow that removed the polyoffset statement, I assume you still have that modification applied in your code. You can see for yourself, go to http://jmonkeyengine.com/nightly, get the 9/10 version, run TestChooser and select TestShadow. The result is as follows:
http://i.imgur.com/Par13.pngYou cited the noise on the lit parts of the teapot as the reason for implementing VSM, as you can see, the noise is not present in vanilla jME3.
@Momoko_Fan said:
It's not fake, just a modified version of jME3. Like I said, in a previous thread you posted a modification to PreShadow that removed the polyoffset statement, I assume you still have that modification applied in your code. You can see for yourself, go to http://jmonkeyengine.com/nightly, get the 9/10 version, run TestChooser and select TestShadow. The result is as follows:
You cited the noise on the lit parts of the teapot as the reason for implementing VSM, as you can see, the noise is not present in vanilla jME3.
I didn't modify the file shader in that screen cap >.< It's running the test file passing in a larger texture size than 512 (just multiple it by some number... 2, 4, etc). Try it and see what happens.
The whole reason you allow for larger texture sizes is to improve the shadow map… if 512 is the max you can use before the noise bleeds back in… there is a problem with the code.
This might be happening because Shadow.glsl hard-coded the texture size at 1024… so no matter what size you pass in, it uses 1.0/1024 …
Shadow.glsl
[java]const float texSize = 1024.0;
const float pixSize = 1.0 / texSize;
const vec2 pixSize2 = vec2(pixSize);[/java]
This might also make your dithering look like crap at anything other than 1024