Can you render to stencil buffer?

The_Leo · September 8, 2016, 8:31pm

Hello, :chimpanzee_amused:

Basically, I have made a texture

 stencilTex = new Texture2D(camW, camH, Image.Format.Depth24Stencil8);

How can I run a fragment shader that can output to this texture?
I have tried:

depthBuffer = new FrameBuffer(camW, camH, 1);
depthBuffer.addColorTexture(stencilTex);

But this throws an exception for this Texture format.
If I do:

depthBuffer.setDepthTexture(stencilTex);

I assume this binds the stencilTex to be read not written to. In that case it’s no good.

Now a little information about the problem I’m trying to solve. I have a fragment shader that outputs materialIDs per fragment. I’d like to store these ids in stencil buffer. Then for each material, there will be a pass with stencil operation to render the material’s matching fragments.

If there’s no way to render to stencil buffer, I suppose I could render to Texture of type eg. GL_R8UI, and copy its data to the stencilTex with CPU, but I’d like to avoid needless GPU-CPU transfers if possible. Also jme3 does not have Image.Format.R8UI, is there a reason why it doesnt have this type?

Thanks. :chimpanzee_amused:

david_bernard_31 · September 8, 2016, 9:41pm

One year ago, I used stencil buffer for my deferred pipeline.
iirc:

lbuffer.fb.setDepthTexture(gbuffer.depth)
FrameBufferHack.setDepthStencilAttachment(lbuffer.fb)

source: jme3_ext_deferred/Pass4LBuffer.xtend at master · davidB/jme3_ext_deferred · GitHub

AND

github.com

davidB/jme3_ext_deferred/blob/master/src/main/java/com/jme3/texture/FrameBufferHack.xtend#L6


package com.jme3.texture
import java.lang.reflect.Field
import com.jme3.texture.FrameBuffer.RenderBuffer
class FrameBufferHack {
	def static void setDepthStencilAttachment(FrameBuffer fb) {
		try {
			var Field field = typeof(FrameBuffer).getDeclaredField("depthBuf")
			field.setAccessible(true)
			var RenderBuffer depthBuf = field.get(fb) as RenderBuffer
			depthBuf.slot = -101
		} catch (Exception exc) {
			throw new RuntimeException('''wrap :«exc»'''.toString, exc)
		}

FYI, for materialId, I don’t used stencil, but 8bit (alpha of the normal texture in a Gbuffer / MRT).

github.com

davidB/jme3_ext_deferred/blob/master/src/main/resources/Shaders/deferred/gbuffer.frag#L47


		#ifdef LATC
			normal.z = sqrt(1.0 - (normal.x * normal.x) - (normal.y * normal.y));
		#endif
	#elif !defined(VERTEX_LIGHTING)
		vec3 normal = vNormal;
		#if !defined(LOW_QUALITY) && !defined(V_TANGENT)
			normal = normalize(normal);
		#endif
	#endif
    vec3 n = normalize(normal);
    out_FragData[0] = vec4(encodeNormal(normal), float(m_MatId) / 256.0);
    #ifdef COLORMAP
      vec4 albedo = texture(m_ColorMap, texCoord);
    #else
      vec4 albedo = m_Color;
    #endif
    out_FragData[1] = albedo;
    #ifdef SPECULARMAP
      vec4 specularColor = texture(m_SpecularMap, texCoord);

The_Leo · September 8, 2016, 10:01pm

Thank you for the reply. Btw what does depthBuf.slot = -101 do? is -101 some special number?

Yes I can store matID as alpha, but… Let’s say i got 10 different materials, each processing a fullscreen quad. Yes, I can compare the matID inside the fragment shader and discard, but that has two disadvantages: 1. extra frag shader code, 2. Each material processes every pixel. It’s not using an Early Fragment Test, which I wanted to use by using a stencil.

The_Leo · September 8, 2016, 10:07pm

I think I have found a viable solution, instead of using a stencil, a depth buffer will do. :chimpanzee_amused:
I can add a Image.Format.Depth to a FrameBuffer and write to it with gl_FragDepth = float(m_MatId) / 256.0;

Then when rendering materials I can move the fullscreen quad to or from cam to select which pixels a shader will render. :chimpanzee_amused: This way fragments with other matIds should be tested out by the Depth Test.