Black Frame when reading Pixels in postQueue

Hi! I implemented some classes to be able to accurately select objects by rendering them in a special way, saving the result and later reading the pixel color at the mouse position.
It works quite good but every time i render the selectable objects i get a black frame. Is there any way to prevent that?

Here is the postQueue Method of my SceneProcessor:

[java]
public void postQueue(RenderQueue rq) {

    if (renderObjects) {

        renderObjects = false;

        renderManager.getRenderer().setBackgroundColor(ColorRGBA.BlackNoAlpha);
        renderManager.getRenderer().setFrameBuffer(tempBuffer);
        renderManager.getRenderer().clearBuffers(true, true, true);

        // Render Selectable Geometries

        renderManager.getRenderer().setFrameBuffer(this.viewPort.getOutputFrameBuffer());

        renderer.readFrameBuffer(tempBuffer, copyBuffer);

        copyBuffer.get(copyArray);
        copyBuffer.clear();

    }
}

[/java]

To read the pixel color i access copyArray.

Can you post how you set up tempBuffer?

Erm… EDIT: And the camera?

Uhhh… EDIT @: And copyBuffer?

There is more than likely some discrepancy between these.

@t0neg0d said: Can you post how you set up tempBuffer?

Erm… EDIT: And the camera?

Uhhh… EDIT @: And copyBuffer?

There is more than likely some discrepancy between these.

Yes of course! Sorry i didnt before but the original class i a little big ^^

Here the complete SceneProcessor:

[java]/*

  • To change this template, choose Tools | Templates
  • and open the template in the editor.
    */
    package com.jme3.app;

/**
*

  • @author Naas
    */
    import com.jme3.app.ListenableFlyCam.CameraChangedListener;
    import com.jme3.asset.AssetManager;
    import com.jme3.material.Material;
    import com.jme3.math.ColorRGBA;
    import com.jme3.post.SceneProcessor;
    import com.jme3.renderer.RenderManager;
    import com.jme3.renderer.Renderer;
    import com.jme3.renderer.ViewPort;
    import com.jme3.renderer.queue.RenderQueue;
    import com.jme3.scene.Geometry;
    import com.jme3.texture.FrameBuffer;
    import com.jme3.texture.Image;
    import com.jme3.texture.Texture;
    import com.jme3.texture.Texture2D;
    import com.jme3.util.BufferUtils;
    import java.nio.ByteBuffer;
    import java.util.HashMap;
    import java.util.HashSet;
    import java.util.Iterator;
    import java.util.LinkedList;
    import java.util.Map;
    import java.util.Queue;
    import java.util.Set;

/**
*

  • @author Naas
    */
    public class ObjectPixelProcessor implements SceneProcessor, CameraChangedListener {

    private FrameBuffer tempBuffer;
    private ViewPort viewPort;
    private Renderer renderer;
    private RenderManager renderManager;
    private ByteBuffer copyBuffer;
    private byte[] copyArray;
    private final static int waitUpdateMillis = 100;
    private long markOutdatedTime = 0;
    private boolean forceUpdate = false;
    private Texture2D offTex;
    private int width;
    private int height;
    private boolean selectionEnabled;
    private ColorRGBA tempColor = new ColorRGBA();

    /**

    • Start Value 1 makes sure that id 0 can be used to ignore all unselectable
    • Objects
      */
      private int highestFreeId = 1;
      private Queue freeIds = new LinkedList();
      private Map selectableIdGeometryMap = new HashMap();
      private Map selectableGeometryIdMap = new HashMap();
      private Set unSelectableGeometries = new HashSet();
      private Material idMaterial;
      //private Main main;

    /**

    • Adds a Geometry which can be selected if the Geometry is already added as
    • Unselectable Geometry it is removed from the Unselectable Geometries List
    • @param e
      */
      public void addSelectableGeometry(Geometry e) {
      if (!selectableGeometryIdMap.containsKey(e)) {
      if (unSelectableGeometries.contains(e)) {
      unSelectableGeometries.remove(e);
      }
      int id = createId();
      selectableIdGeometryMap.put(id, e);
      selectableGeometryIdMap.put(e, id);
      }
      }

    public void removeSelectableGeometry(Geometry o) {
    int id = selectableGeometryIdMap.get(o);
    selectableGeometryIdMap.remove(o);
    selectableIdGeometryMap.remove(id);
    destroyId(id);
    }

    private int createId() {
    if (!freeIds.isEmpty()) {
    return freeIds.poll();
    } else {
    highestFreeId++;
    return highestFreeId - 1;
    }
    }

    private void destroyId(int id) {
    if (id == highestFreeId - 1 && id > 1) {
    highestFreeId–;
    } else {
    freeIds.add(id);
    }
    }

    /**

    • Adds a Geometry as Unselectable which means it cannot be selected and it
    • prevents other Geometries from being selected if they are behind of this
    • Geometry if the Geometry is already added as selectable Geometry it is
    • removed from the selectable Geometries List
    • @param e
      */
      public void addUnSelectableGeometry(Geometry e) {
      if (!unSelectableGeometries.contains(e)) {
      if (selectableGeometryIdMap.containsKey(e)) {
      removeSelectableGeometry(e);
      }
      unSelectableGeometries.add(e);
      }
      }

    public void removeUnSelectableGeometry(Geometry o) {
    unSelectableGeometries.remove(o);
    }

    public ObjectPixelProcessor(RenderManager rm, Renderer renderer, ViewPort viewPort, AssetManager assetManager) {

     this.renderManager = rm;
     this.renderer = renderer;
    
     this.viewPort = viewPort;
    
     viewPort.addProcessor(this);
    
     idMaterial = new Material(assetManager, "MatDefs/Colorid.j3md");
    

    }

    public boolean isNeedUpdate() {
    return (markOutdatedTime > 0 ? (System.currentTimeMillis() - markOutdatedTime > waitUpdateMillis && selectionEnabled) : false) || forceUpdate;
    }

    public void initialize(RenderManager rm, ViewPort vp) {

    }

    public void reshape(ViewPort vp, int w, int h) {

     reshape(vp);
    

    }

    public void reshape(ViewPort vp) {
    this.viewPort = vp;
    height = vp.getCamera().getHeight();
    width = vp.getCamera().getWidth();

     copyBuffer = BufferUtils.createByteBuffer(width * height * 4);
     copyArray = new byte[width * height * 4];
    
     tempBuffer = new FrameBuffer(width, height, 0);
    
     tempBuffer.setDepthBuffer(Image.Format.Depth);
     tempBuffer.setColorBuffer(Image.Format.RGBA8);
    
     offTex = new Texture2D(width, height, Image.Format.RGBA8);
     offTex.setMinFilter(Texture.MinFilter.Trilinear);
     offTex.setMagFilter(Texture.MagFilter.Bilinear);
    
     tempBuffer.setDepthBuffer(Image.Format.Depth);
     tempBuffer.setColorTexture(offTex);
    

    }

    public boolean isInitialized() {
    return true;
    }

    public void preFrame(float tpf) {
    }

    /**
    *Call when Scene or Camera has changed
    */
    public void markAsOutdated() {

     markOutdatedTime = System.currentTimeMillis(); 
    

    }

    public void postQueue(RenderQueue rq) {

     if (isNeedUpdate()) {
    
         renderManager.getRenderer().setBackgroundColor(ColorRGBA.BlackNoAlpha);
         renderManager.getRenderer().setFrameBuffer(tempBuffer);
         renderManager.getRenderer().clearBuffers(true, true, true);
    
         Material tempMat;
    
         for (Iterator it = this.selectableGeometryIdMap.keySet().iterator(); it.hasNext();) {
             Geometry selectableGeometry = it.next();
             int selectableGeometryId = selectableGeometryIdMap.get(selectableGeometry);
             tempMat = selectableGeometry.getMaterial();
             if (tempMat.getAdditionalRenderState().isWireframe()) {
                 idMaterial.getAdditionalRenderState().setWireframe(true);
             } else {
                 idMaterial.getAdditionalRenderState().setWireframe(false);
             }
             selectableGeometry.setMaterial(idMaterial);
             tempColor.fromIntRGBA(selectableGeometryId);
             idMaterial.setColor("objectIdColor", tempColor);
             renderManager.renderGeometry(selectableGeometry);
             selectableGeometry.setMaterial(tempMat);
         }
    
         tempColor.fromIntRGBA(0);
    
         idMaterial.setColor("objectIdColor", tempColor);
    
         for (Iterator it = this.unSelectableGeometries.iterator(); it.hasNext();) {
             Geometry unselectableGeometry = it.next();
             tempMat = unselectableGeometry.getMaterial();
             if (tempMat.getAdditionalRenderState().isWireframe()) {
                 idMaterial.getAdditionalRenderState().setWireframe(true);
             } else {
                 idMaterial.getAdditionalRenderState().setWireframe(false);
             }
             unselectableGeometry.setMaterial(idMaterial);
             renderManager.renderGeometry(unselectableGeometry);
             unselectableGeometry.setMaterial(tempMat);
         }
    
         renderManager.getRenderer().setFrameBuffer(this.viewPort.getOutputFrameBuffer());
    
        
    
         renderer.readFrameBuffer(tempBuffer, copyBuffer);
    
         copyBuffer.get(copyArray);
         
         copyBuffer.clear();
         markOutdatedTime = 0;
         forceUpdate = false;
    
     }
    

    }

    public void forceUpdate() {
    forceUpdate = true;
    }

    public void postFrame(FrameBuffer out) {

    }

    public Geometry getGeometryAt(int x, int y) {

     int id = getIdAt(x, y);
    
     if (id == 0) {
         return null;
     } else {
    
         Geometry geom = selectableIdGeometryMap.get(id);
    
         if (geom != null) {
             return geom;
         } else {
             throw new UnsupportedOperationException("There is no Geometry with id '" + id + "'");
         }
     }
    

    }

    private int getIdAt(int x, int y) {

     if ((x >= 0 && x = 0 && y < height)) {
    
         int index = 4 * (y * width + x);
    
         int id = ((copyArray[index + 2] & 0xFF) << 24)
                 | ((copyArray[index + 1] & 0xFF) << 16)
                 | ((copyArray[index + 0] & 0xFF) << 8)
                 | ((copyArray[index + 3] & 0xFF));
         return id;
     } else {
        
       throw new UnsupportedOperationException("Pixel Coordinates ("+x+","+y+") are out of Bounds ("+width+","+height+") viewPort:'"+viewPort+"'");        
     }
    

    }

    public void getPixelColor(int x, int y) {
    }

    public void cleanup() {
    }

    public void onCameraChanged() {

     this.markAsOutdated();
    

    }

    public boolean isSelectionEnabled() {
    return selectionEnabled;
    }

    public void setSelectionEnabled(boolean selectionEnabled) {
    this.selectionEnabled = selectionEnabled;
    }
    }
    [/java]

And how i set it up in my SimpleApplication:
[java] @Override
public void simpleInitApp() {

    objectSelectionProcessor = new ObjectPixelProcessor(renderManager, renderer, viewPort, assetManager);
    objectSelectionProcessor.reshape(viewPort);

    addTestGeometries(100);

    objectSelectionProcessor.forceUpdate();

}
[/java]

Curious… is there a reason you are using this particular approach to picking?

There was a thread recently where it was dissected pretty thoroughly, I think… if I read your original post correctly.

Forgot to mention… I think this is a cool idea =)

My first guess is here where the <<<< are.

Couple questions:
Is the black border single pixel? Or?
Which leads me to the next question…
Are you sure of the buffer size? The border leads me to believe there is a difference between the sizes of copyBuffer = BufferUtils.createByteBuffer(width * height * 4); & copyArray = new byte[width * height * 4];

Second guess (and I don't particularly have a reason for it) is the min and mag filters on the offscreen texture. To be honest… I would try commenting those two lines out, because I don't think they are used unless the texture is being rendered to a geometry.

[java]
public void reshape(ViewPort vp) {
this.viewPort = vp;
height = vp.getCamera().getHeight();
width = vp.getCamera().getWidth();

    copyBuffer = BufferUtils.createByteBuffer(width * height * 4); &lt;&lt;&lt;&lt;
    copyArray = new byte[width * height * 4];  &lt;&lt;&lt;&lt;

    tempBuffer = new FrameBuffer(width, height, 0);

    tempBuffer.setDepthBuffer(Image.Format.Depth);
    tempBuffer.setColorBuffer(Image.Format.RGBA8);

    offTex = new Texture2D(width, height, Image.Format.RGBA8);
    offTex.setMinFilter(Texture.MinFilter.Trilinear);
    offTex.setMagFilter(Texture.MagFilter.Bilinear);

    tempBuffer.setDepthBuffer(Image.Format.Depth);
    tempBuffer.setColorTexture(offTex);

}

[/java]

@pspeed said: Curious... is there a reason you are using this particular approach to picking?

There was a thread recently where it was dissected pretty thoroughly, I think… if I read your original post correctly.

Oh yes i missed that. Seems really he had the same idea.

The reason i want to use that approach is that i want to implement a editor and i think it is ideal for that.
I want to be able to select Line and Point Geometries which i think is rather difficult with the ray collision approach.
Also the picking itself should be very fast so that it is no problem to pick all the time on analog mouse input (highlight objects on mouseover).
In an editor it is accectable to only allow picking if A: The Camera doesen’t move and B:The scene doesen’t change.
As it is only necessary to re-render the objects for selection if picking is enabled and either condition A or B has been false since the last pick, the re-rendering rarely occours in that editor situation.

Should work if it really doesn’t change often.

Possible down sides:
-not really that much faster overall as picking is pretty fast and rendering is relatively slow. Lines and points can be handled separately using pretty straight forward 2D math. Granted, looking up a pixel will be faster than picking but it cost a lot up front to do it… so you have to amortize those costs.
-you have to keep a whole frame-buffer around in addition to your objects which already had enough data to do picking.
-you can never select objects behind another object

If you have 10,000 or 20,000 objects or so and never move the camera or rarely change the scene, it could be a win. A 2D quad tree or fixed grid might still be better over all… after all, a rendered frame is just fixed grid with cell width 1.

[java]Forgot to mention… I think this is a cool idea =)

My first guess is here where the <<<< are.

Couple questions:
Is the black border single pixel? Or?
Which leads me to the next question…
Are you sure of the buffer size? The border leads me to believe there is a difference between the sizes of copyBuffer = BufferUtils.createByteBuffer(width * height * 4); & copyArray = new byte[width * height * 4];

Second guess (and I don't particularly have a reason for it) is the min and mag filters on the offscreen texture. To be honest… I would try commenting those two lines out, because I don't think they are used unless the texture is being rendered to a geometry.[/java]

Thanks :slight_smile: well it has it’s drawbacks as @pspeed mentions

I don't understand which border you mean ^^ there should not be any border at all. At least i don't know of any?

Yes, right the min mag filter settings seem absolutely useless :smiley: don't know why they are there probably i tested something there (i worked on that code for a long time).

@pspeed:

@pspeed said: Possible down sides: -not really that much faster overall as picking is pretty fast and rendering is relatively slow. Lines and points can be handled separately using pretty straight forward 2D math. Granted, looking up a pixel will be faster than picking but it cost a lot up front to do it.... so you have to amortize those costs. -you have to keep a whole frame-buffer around in addition to your objects which already had enough data to do picking. -you can never select objects behind another object

Yes i agree the preparation is really slow and costly. Are you sure it is possible to pick lines and points as accurately by using 2D Math? I guess one would have to re-program the rasterization algorythm to be that accurate…
And i mean in my case the re-rendering really rarely occours. For example first i move the camera (2 seconds) then i want to select a object (3 seconds) then i move a selected object (4 seconds), now i want to select a different object (2 seconds). In that example (2+3+4+2=11 seconds total) there are only 2 times a re-rendering has to take place and that is 1. after camera has been moved and 2. after object has been moved. Also there are 5 seconds i can take advantage of the fast pick by pixel color lookup.
If i would use ray intersection i would have had 5 seconds where the ray intersection has to be calculated at every single frame at least (because i want to hilight objects on mouse over). I guess it can be quite annoying if you want to select a object while the framerate is varrying.

You could never select a object behind another object yep :confused: But i guess most of the time you select a single object you really want to get what you see on the screen. In special cases one would have to hide the first object. If one would like to select multiple objects via rectagle this approach would be useless too i think.

@naas said: Yes i agree the preparation is really slow and costly. Are you sure it is possible to pick lines and points as accurately by using 2D Math? I guess one would have to re-program the rasterization algorythm to be that accurate...

Well, you’ve already boiled it down to a 2D problem… which in the graphics world picking in a 2D environment is pretty well understood. You can potentially even reuse some swing classes if you don’t want to do the math… since they already have methods for getting the distance to a line segment and distance to a point… and that’s the nature of picking lines and points. distSquared < thresholdSquared

It’s definitely more code, though.