I have a problem with the things I list in the topic, obviously.
To be precise, I am running a small editor that I've created with StandardGame and gamestates. I have physics, lighting and shadows enabled, and for the most part all is well. Recently I added a Sphere which I can drop on my landscape to test the physics and lighting. The thing that immediately caught my eye, is how the sphere shadow side (not the one it casts on the terrain, the dark side of the sphere itself) has an extremely edgy border to the non shadowed side. It is as if the polygons just switch between darker and brighter, two states, which makes the relatively low poly sphere look like it has a jig saw edge between the two areas.
From what I understand, dithering might be a way to smooth that out. For some reason the dithering state doesnt seem to work here though - maybe I am overriding its effects by some lighting, texture or other state setting?
So I guess my question would be, in order of relevance:
How to avoid this edgy shadowing?
What could be messing up attempts to dither, if dithering would be a possible solution?
If needed, I can provide snippets of code, with the light setup and other releant states.
I know the saw-type edge you speak of, Its a stencil shadow problem, heres a quote from wikipedia
There is also a problem with the shadow where the faces along the silhouette edge are relatively shallow. In this case, the shadow an object casts on itself will be sharp, revealing its polygonal facets, whereas the usual lighting model will have a gradual change in the lighting along the facet. This leaves a rough shadow artifact near the silhouette edge which is difficult to correct. Increasing the polygonal density will minimize the problem, but not eliminate it. If the front of the shadow volume is capped, the entire shadow volume may be offset slightly away from the light to remove any shadow self-intersections within the offset distance of the silhouette edge (this solution is more commonly used in shadow mapping).
for a workaround you can use passes and have the node cast shadows but not receive them. Its not the perfect solution but it works in some cases.
Thanks for the answers. I've been doing some other stuff lately but am now back to sort this issue out. Indeed I am using modulative setting for the shadow pass. Might play around with that, as well as trying to disable the "self shadowing" of the sphere to see if the result is less disastrous.
