I think I know the answer to this question, but still thought I might ask, is it possible to project the effect a shader can have on a high poly model onto a low poly one or it is easier to simple make the low poly model as smooth as possible. I found a very good toon shader on blender artists and used it as a base for what I’m trying to achieve. also is it possible to apply a noisy grain to the contour effects of the the shader and what would be the best way
the forum either does ‘like’ photobucket urls of I’m doing something wrong
low
http://i922.photobucket.com/albums/ad68/mcbeth316/low.jpg
smooth
http://i922.photobucket.com/albums/ad68/mcbeth316/smooth1.jpg
http://i922.photobucket.com/albums/ad68/mcbeth316/smooth2.jpg
For the noisy grain question, well, you are in luck: Film Grain. I just put that online. If you need help understanding it or adding it to a specific part in your shader just poke me a bit, I can probably even put it in for you to get the result you desire.
Other than that, it seems to me that you are mostly concerned with the slightly visible triangle edges on your model when using the toon shading? I’m not sure if this is any improvement but I just wanted to mention it. One thought that came to my mind is perhaps changing the way the light is calculated on your model, like smoothing that out, then applying the toon shading color levels after that. That way you may improve on those details.
I’ll take a look at the film grain thanks as for what I’m going for with the noise some thing like that
http://blenderartists.org/forum/attachment.php?attachmentid=125565&d=1294342571
mixed with a sin city-ish style that I put together in my blender rendering shader
http://blenderartists.org/forum/attachment.php?attachmentid=125566&d=1294343353
the shader can already do the sin city-ish…kinda thing, still needs tweaking though and the grain would be bonus
One thought that came to my mind is perhaps changing the way the light is calculated on your model, like smoothing that out, then applying the toon shading color levels after that. That way you may improve on those details.
shaders are not my thing, how do you do that
Interesting picture. Seems to me that to obtain results as close as possible to that one you will have to use two kinds of materials:
One that goes with a slightly visible posterization type of colorset until a certain luminance treshold then turn into a black blob, as seen on the characters back in the middle. That material seems to be on most characters in the scene.
The other material would be a world material, one that goes into dithering below a treshold, perhaps followed by a blob as well although it seems that the blacks on that are only for edges and such, so that’s probably best done by using black textures and edges from the cartoonedge filter.
Looking at it again, it seems the devil-ish figure has a hybrid of the two above, depends on your specific wants how you handle that I guess 
Seems an interesting style to choose, I surely would like to see your results
If you don’t mind, I might as well poke around a bit and see what I come up with in the coming days 
must have mist my quoting you but you said
One thought that came to my mind is perhaps changing the way the light is calculated on your model, like smoothing that out, then applying the toon shading color levels after that. That way you may improve on those details.
I'm not the best at shaders and wonder if you could eloborate
also this is the toon shader as it now stands the next thing for me will be figuring out the jme3 compatibility stuff after I add the grain
vert
[java]varying vec3 vLight;
varying vec4 vDiff;
varying vec4 vView;
varying vec3 vNormal;
varying vec4 vMat;
varying vec2 texCoord0;
void main()
{
vec3 vtx = (gl_ModelViewMatrix * gl_Vertex).xyz;
vLight = (gl_LightSource[0].position).xyz - vtx;
//texture
texCoord0 = gl_MultiTexCoord0.xy;
vView = gl_Vertex - gl_ModelViewMatrix[3];
vNormal = gl_NormalMatrix * gl_Normal;
vMat = gl_Color * gl_LightSource[0].diffuse;
gl_Position = ftransform();
}[/java]
frag
[java]varying vec3 vLight;
varying vec4 vDiff;
varying vec4 vView;
varying vec3 vNormal;
varying vec4 vMat;
varying vec2 texCoord0;
uniform vec4 vWidths;
uniform sampler2D colorMap0;
void main(){
vec3 vNorm = normalize(vNormal);
vec4 vColour = vMat;
vec3 vLightN = normalize(vLight);
vec3 vViewN = normalize(vView.xyz);
float ndotv = dot(vNorm, vLightN); //vViewN
float ndotl = dot(vNorm, vLightN);
vec4 base0 = texture2D(colorMap0, texCoord0);
gl_FragColor = base0 * gl_FrontMaterial.diffuse /* gl_LightSource[0].diffuse*/;
if ( ndotl < vWidths.x )
vColour *= 0.75;
if ( ndotl < vWidths.y )
vColour *= 0.5;
if ( ndotl < vWidths.z )
vColour = 0.5;
if ( ndotv < vWidths.w )
vColour = 0.0 / gl_LightSource[0].diffuse/;
gl_FragColor *= (vColour+base0) * gl_FrontMaterial.diffuse;
}[/java]
I lol’ed at:
[java]<img class=“maximage” src=“http://hub.jmonkeyengine.org/wp-includes/images/smilies/chimpanzee-sad.gif” alt=":(">[/java]
I think there is room for improvement on this forum
Anyhow, I’m going to bed soon so I won’t be trying your snippet right now, I think I’ll manage to spare some time for it tomorrow
By the way, I see you got a .vert, did you change anything in there? looks pretty default to me, don’t know them by heart yet tho. Judging from the code I suppose you wrote a material, any specific reason you wanted to go that way (perhaps my precious post? :P) or do you think a post processing shader would do well enough? I think that would be a tad easier (for me anyway, got all my experience until today only in that part, looking into materials themselves right now, quite fun).
To explain my lighting suggestion: The thing is that you first calculate light the usual way. This calculation is heavily dependent on interpolations between the triangles (edit: erm, normals in this case) and therefore you end up with the crisp edges in your shader at several locations. My suggestion is to change this lighting calculation a bit (as it may be less precise for toon shading anyway) or to smooth it out a little. That way the crisp changes in the lighting will disappear and after that, you can apply the toon shader and hopefully end up with a bit smoother look on the different color levels. I haven’t seen any implementations of that but it couldn’t hurt to try anyway.
A somehwat rediculous idea just sprung to my mind, perhaps we could smooth the colors a bit using the z buffer and apply the toon shader after that… Interesting topic for some prototyping, we may lose some crisp edges when changes occur on a single matterial tho. Hmm.
How’s this?
Filters I used:
new kind of nosiy treshold, gotta find a name for that, it basically blobs dark parts with some noise. SinCityShadowFilter?
posterization with strength down to 0.5 to get rid of too coarse edges but give the impression of color levels
cartoon filter with a slightly increased line thickness, matter of preference.
Let me know what you think of it, I’ll start cleaning it up a little now so I can share it soon
Just wondering, do you want the grain to be still or changing per frame? Steady is less tiresome for your eyes but looks a bit strange when changing the camera angle. Could also change the noise only every x msec, that way we may improve the visuals a bit without tiring your eyes.
Edit: The noisy grain area can be influenced easily, I’ll add control to change that. I put it quite wide for the image above, just imagine that if I put it lower you obtain nearly a clear black edge but some small noise occurs
Still thinking about adding another filter to make the image appear a bit washed, the colors used here are way too strong.
holy crap that’s sweet it would be interesting to see that indeed,especially to compare with my less spectacular but pleasing "to me 
" result
I dont think forum likes photobucket urls for some reason, I really wish they would find and say what the issue might be :roll: I cant even simply paste the links.
http://i922.photobucket.com/albums/ad68/mcbeth316/grain_side.jpg
http://i922.photobucket.com/albums/ad68/mcbeth316/grain_back.jpg
http://i922.photobucket.com/albums/ad68/mcbeth316/grain_front.jpg
as for name how about GraphicNovel and it would be useful to be able to disable the grain for people that may not want it
but that is cool
if I was to have a complaint is that the edges seem “hard” and “separate” from the effect a bit or is that jme’s toon edge added
I get it contrasty but blending
http://i922.photobucket.com/albums/ad68/mcbeth316/test.jpg
your result is still better than I managed though 
I’ll adopt the name you suggested
It’s a tad hard to see for me as your images appear resized thus getting the grain nearly blurred out, but that also looks good to me Just some slight grain at the shadow edges if I’m right, I think I can obtain the same look by changing my grainyness parameter
By the way, I believe you are using your high-poly model on that shot? How does it look on the low poly one?
sure thing, different model though
all I did was add your film grain effect to my shader to work on a layer of the contour effects
[java]varying vec3 vLight;
varying vec4 vDiff;
varying vec4 vView;
varying vec3 vNormal;
varying vec4 vMat;
varying vec2 texCoord0;
uniform vec4 vWidths;
uniform sampler2D colorMap0;
uniform float m_Time;
uniform float m_Strength;
void main(){
vec3 vNorm = normalize(vNormal);
vec4 vColour = vMat;
vec3 vLightN = normalize(vLight);
vec3 vViewN = normalize(vView.xyz);
float ndotv = dot(vNorm, vLightN); //vViewN
// Random, adding values to get rid of edge errors and mods that return 0
float x = (texCoord0.x+4.0) * (texCoord0.y+4.0) * (m_Time10.0);
vec4 grain = vec4(mod((mod(x, 13.0) + 1.0) * (mod(x, 123.0) + 1.0), 0.01)-0.005) * m_Strength;
float ndotl = dot(vNorm, vLightN);
vec4 base0 = texture2D(colorMap0, texCoord0);
gl_FragColor = base0 * gl_FrontMaterial.diffuse / gl_LightSource[0].diffuse*/;
if ( ndotl < vWidths.x )
vColour *= 0.75+grain;
if ( ndotl < vWidths.y )
vColour *= 0.5;
if ( ndotl < vWidths.z )
vColour *= 0.5;
if ( ndotv < vWidths.w ){
vColour *= 0.0;
}
gl_FragColor *= (vColour+base0) * gl_FrontMaterial.diffuse;
}[/java]
Looks quite good, no ugly triangle edge mistakes visible I think
I do want to point out that you use multiple if-statements which may really slow down your frame rate, while I my approach got rid of them, as if statements are really, really bad in GPU programming (altho I probably got a lot of other slow downs anyway using multiple filters hehe, perhaps I should put it all together into a single one). Perhaps you can find another way to obtain the same results without any if’s? (And yes, I did use a bunch of if statements in my crosshatching as I couldn’t come up with a work around there :P)
Edit: Oh, I also noticed you left the -0.005 in the ‘random’ function, that was added to pad out to white and black noise pixels, perhaps you can leave that calculation out?
well unfortunately the code design is not mine I simply adapted it, I think I mentioned that some where…in fact, truth be told, I don’t really know glsl 
… in the truest sense, I found a shader that looked capable of doing what I needed and toyed with the variables to figure out what each part did, then toyed with it some more , looking at it I can’t really see a way to avoid the if/s but I’m all for learning so I’ll definitely give it some thought
still may try and do a filter for mine but, I figure there is tonnes I can learn from yours and see how it looks on my models
Here you go mate
The settings to obtain as close as possible to what I think you want are:
Add this new NoisyTreshold filter (use the original image for best effects). Default options should do it, if not, well I think the paramaters are quite easy to play a with I got noiseWidth on 8 now, way better than the image I posted.
Add a Posterization filter, put strength down to 0.5 or so to blend the colors a little (optional if you got your textures etc right or got another filter to handle lighting the way you want)
Add a CartoonEdge filter to your likings.
Enjoy! Send a picture of your results or let me know if you require any specific changes If you need to speed it up (numerous shaders with more features than necessary may slow down your game) I suppose we could rewrite some stuff and end up with a single shader that does everything you need at a single pass.