Kinect + OpenNI texture into jme

Troubleshooting General Help
1

I would like to use the output of Kinect + OpenNI image buffer as a heightmap for my terrain.



Can anyone give me pointers on how to do this. The OpenNI Samples use awt classes. I want to use JME instead



I have a

private byte[] imgbytes; that is size 640x480 and values 1 to 256 that come out of the Kinect.



First:

How can i turn this byte array into a JME Texture



Second:

Would it be possible to have this Texture be updated in real time and be used as a Terrain Height Map?



Thanks for any help, links and suggestions.

2

Got it half working.

Have the Output of the Kinect as a Texture on a cube

Edit: It updates the Texture in real time.



http://i.imgur.com/tUzdH.png



[java]package org.tenbitworks.kinect;



import java.awt.image.BufferedImage;

import java.awt.image.DataBufferByte;

import java.awt.image.Raster;

import java.io.ByteArrayInputStream;

import java.io.InputStream;

import java.nio.ShortBuffer;



import org.OpenNI.Context;

import org.OpenNI.DepthGenerator;

import org.OpenNI.DepthMetaData;

import org.OpenNI.GeneralException;

import org.OpenNI.OutArg;

import org.OpenNI.ScriptNode;



import com.jme3.app.SimpleApplication;

import com.jme3.asset.TextureKey;

import com.jme3.material.Material;

import com.jme3.math.Vector3f;

import com.jme3.scene.Geometry;

import com.jme3.scene.shape.Box;

import com.jme3.texture.Texture;

import com.jme3.texture.Texture2D;

import com.jme3.texture.plugins.AWTLoader;



public class Test1 extends SimpleApplication {

int width, height;

private static final long serialVersionUID = 1L;

private OutArg<ScriptNode> scriptNode;

private Context context;

private DepthGenerator depthGen;

private byte[] imgbytes;

private float histogram[];

private final String SAMPLE_XML_FILE = “/home/greg/dev/kinect-wksp/kinect_test/src/Data/SamplesConfig.xml”;



private BufferedImage bimg;



Material mat_tt;



@Override

public void simpleInitApp() {

Box b = new Box(Vector3f.ZERO, 1, 1, 1); // create cube shape at the origin

Geometry geom = new Geometry(“Box”, b); // create cube geometry from the shape

// set the cube’s material









mat_tt = new Material(assetManager, “Common/MatDefs/Misc/Unshaded.j3md”);

mat_tt.setTexture(“ColorMap”, assetManager.loadTexture(“Textures/ColoredTex/Monkey.png”));



geom.setMaterial(mat_tt);

rootNode.attachChild(geom); // make the cube appear in the scene



try {

scriptNode = new OutArg<ScriptNode>();

context = Context.createFromXmlFile(SAMPLE_XML_FILE, scriptNode);



depthGen = DepthGenerator.create(context);

DepthMetaData depthMD = depthGen.getMetaData();



histogram = new float[10000];

width = depthMD.getFullXRes();

height = depthMD.getFullYRes();

System.out.println(“Width” + width + " Height" + height);

imgbytes = new byte[widthheight];



DataBufferByte dataBuffer = new DataBufferByte(imgbytes, widthheight);

Raster raster = Raster.createPackedRaster(dataBuffer, width, height, 8, null);

bimg = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);

bimg.setData(raster);



} catch (GeneralException e) {

e.printStackTrace();

System.exit(1);

}





}







@Override

public void simpleUpdate(float tpf) {

updateDepth();



createTexture();

}



private class ByteArrayInfo extends com.jme3.asset.AssetInfo {



private byte[] data;



/**

  • @param manager
  • @param key

    /

    public ByteArrayInfo(byte[] data) {

    super(null, new com.jme3.asset.TextureKey("ByteArray", true));

    this.data = data;

    }



    /     (non-Javadoc)
  • @see com.jme3.asset.AssetInfo#openStream()

    /

    @Override

    public InputStream openStream() {

    return new ByteArrayInputStream(data);

    }

    }



    private void calcHist(DepthMetaData depthMD)

    {

    // reset

    for (int i = 0; i < histogram.length; ++i)

    histogram = 0;



    ShortBuffer depth = depthMD.getData().createShortBuffer();

    depth.rewind();



    int points = 0;

    while(depth.remaining() > 0)

    {

    short depthVal = depth.get();

    if (depthVal != 0)

    {

    histogram[depthVal]++;

    points++;

    }

    }



    for (int i = 1; i < histogram.length; i++)

    {

    histogram += histogram[i-1];

    }



    if (points > 0)

    {

    for (int i = 1; i < histogram.length; i++)

    {

    histogram = (int)(256 * (1.0f - (histogram / (float)points)));

    }

    }

    }







    void updateDepth()

    {

    try {

    DepthMetaData depthMD = depthGen.getMetaData();



    context.waitAnyUpdateAll();



    calcHist(depthMD);

    ShortBuffer depth = depthMD.getData().createShortBuffer();

    depth.rewind();



    while(depth.remaining() > 0)

    {

    int pos = depth.position();

    short pixel = depth.get();

    imgbytes[pos] = (byte)histogram[pixel];

    }

    } catch (GeneralException e) {

    e.printStackTrace();

    }

    }



    void createTexture(){

    DataBufferByte dataBuffer = new DataBufferByte(imgbytes, width    height);

    Raster raster = Raster.createPackedRaster(dataBuffer, width, height, 8, null);

    bimg.setData(raster);

    AWTLoader loader = new AWTLoader();

    Texture tex = new Texture2D(loader.load(bimg, false));



    mat_tt.setTexture("ColorMap", tex);



    }





    }

    [/java]
1 Like
3

I have nothing useful to add… but… cool!

4

I got the color and the depth out of Kinect



http://i.imgur.com/sw8Cg.png

Warning Code is Hackish :slight_smile:

[java]package org.tenbitworks.kinect;



import java.awt.Graphics2D;

import java.awt.image.BufferedImage;

import java.awt.image.DataBufferByte;

import java.awt.image.Raster;

import java.io.ByteArrayInputStream;

import java.io.InputStream;

import java.nio.ByteBuffer;

import java.nio.ShortBuffer;



import org.OpenNI.Context;

import org.OpenNI.DepthGenerator;

import org.OpenNI.DepthMetaData;

import org.OpenNI.GeneralException;

import org.OpenNI.ImageGenerator;

import org.OpenNI.ImageMap;

import org.OpenNI.OutArg;

import org.OpenNI.ScriptNode;



import com.jme3.app.SimpleApplication;

import com.jme3.material.Material;

import com.jme3.math.ColorRGBA;

import com.jme3.math.Quaternion;

import com.jme3.math.Vector3f;

import com.jme3.scene.Geometry;

import com.jme3.scene.Node;

import com.jme3.scene.Spatial;

import com.jme3.scene.shape.Box;

import com.jme3.terrain.geomipmap.TerrainLodControl;

import com.jme3.terrain.geomipmap.TerrainQuad;

import com.jme3.terrain.heightmap.AbstractHeightMap;

import com.jme3.terrain.heightmap.ImageBasedHeightMap;

import com.jme3.texture.Image;

import com.jme3.texture.Texture;

import com.jme3.texture.Texture.WrapMode;

import com.jme3.texture.Texture2D;

import com.jme3.texture.plugins.AWTLoader;

import com.jme3.water.SimpleWaterProcessor;



public class Test3Water extends SimpleApplication {

int width, height;

private static final long serialVersionUID = 1L;

private OutArg<ScriptNode> scriptNode;

private Context context;

private DepthGenerator depthGen;

private ImageGenerator imageGen;

private byte[] imgbytes;

private byte[] colorimgbytes;

private float histogram[];

private final String SAMPLE_XML_FILE = “/home/greg/dev/kinect-wksp/kinect_test/src/Data/SamplesConfig.xml”;



private BufferedImage bimg;

private BufferedImage colorbimg;



Material mat_tt;

Material mat_color;

private TerrainQuad terrain;

Material mat_terrain;



Material mat_solid;





Material mat;

Spatial waterPlane;

Geometry lightSphere;

SimpleWaterProcessor waterProcessor;



Node sceneNode;

boolean useWater = true;

private Vector3f lightPos = new Vector3f(33,12,-29);



private boolean SHOW_TERRAIN = true;



Geometry floor_geo;



@Override

public void simpleInitApp() {



flyCam.setMoveSpeed(100);



mat_tt = new Material(assetManager, “Common/MatDefs/Misc/Unshaded.j3md”);

mat_tt.setTexture(“ColorMap”, assetManager.loadTexture(“Textures/ColoredTex/Monkey.png”));



mat_color = new Material(assetManager, “Common/MatDefs/Misc/Unshaded.j3md”);

mat_color.setTexture(“ColorMap”, assetManager.loadTexture(“Textures/ColoredTex/Monkey.png”));



mat_solid = new Material(getAssetManager(), “Common/MatDefs/Misc/Unshaded.j3md”);

mat_solid.setColor(“Color”, ColorRGBA.Black);



initTextureBox();

if(SHOW_TERRAIN){

loadTerrain();

}



initOpenNI();

//initTransparent();

//initScene();

//initWater();

initFloor();



cam.setLocation(new Vector3f(0, 100, 0));

cam.lookAt(Vector3f.ZERO, Vector3f.UNIT_Y);



}



private void initTextureBox(){

Box b = new Box(Vector3f.ZERO, 1, 1, 1); // create cube shape at the origin

Geometry geom = new Geometry(“Box”, b); // create cube geometry from the shape

// set the cube’s material

geom.setMaterial(mat_tt);

rootNode.attachChild(geom); // make the cube appear in the scene



}



// private void initTransparent(){

// /** A translucent/transparent texture, similar to a window frame. /

// Box boxshape3 = new Box(new Vector3f(0f,0f,0f), 1f,1f,0.01f);

// Geometry window_frame = new Geometry(“window frame”, boxshape3);

// Material mat_tt = new Material(assetManager, “Common/MatDefs/Misc/Unshaded.j3md”);

// mat_tt.setTexture(“ColorMap”, assetManager.loadTexture(“Textures/ColoredTex/Monkey.png”));

// mat_tt.getAdditionalRenderState().setBlendMode(BlendMode.Alpha);

// window_frame.setMaterial(mat_tt);

//

// /* Objects with transparency need to be in the render bucket for transparent objects: /

// window_frame.setQueueBucket(Bucket.Transparent);

// rootNode.attachChild(window_frame);

// }



private void initOpenNI(){

try {

scriptNode = new OutArg<ScriptNode>();

context = Context.createFromXmlFile(SAMPLE_XML_FILE, scriptNode);



depthGen = DepthGenerator.create(context);

DepthMetaData depthMD = depthGen.getMetaData();







histogram = new float[10000];

width = depthMD.getFullXRes();

height = depthMD.getFullYRes();

System.out.println(“Width” + width + " Height" + height);

imgbytes = new byte[widthheight];



DataBufferByte dataBuffer = new DataBufferByte(imgbytes, widthheight);

Raster raster = Raster.createPackedRaster(dataBuffer, width, height, 8, null);

bimg = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);

bimg.setData(raster);





//Image

imageGen = ImageGenerator.create(context);

ImageMap imap = imageGen.getImageMap();



System.out.println(“CWidth” + imap.getXRes() + " CHeight" + imap.getYRes() );

System.out.println(“BytesPerPixel” + imap.getBytesPerPixel());



colorimgbytes = new byte[widthheight];

DataBufferByte colordataBuffer = new DataBufferByte(colorimgbytes, width*height);

Raster colorraster = Raster.createPackedRaster(colordataBuffer, width, height, 8, null);

colorbimg = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);

colorbimg.setData(colorraster);



} catch (GeneralException e) {

e.printStackTrace();

System.exit(1);

}

}











@Override

public void simpleUpdate(float tpf) {

if(SHOW_TERRAIN){

updateColor();

updateDepth();



createTexture();

}

}



private class ByteArrayInfo extends com.jme3.asset.AssetInfo {



private byte[] data;



/**

  • @param manager
  • @param key

    /

    public ByteArrayInfo(byte[] data) {

    super(null, new com.jme3.asset.TextureKey("ByteArray", true));

    this.data = data;

    }



    /     (non-Javadoc)
  • @see com.jme3.asset.AssetInfo#openStream()

    /

    @Override

    public InputStream openStream() {

    return new ByteArrayInputStream(data);

    }

    }



    private void calcHist(DepthMetaData depthMD)

    {

    // reset

    for (int i = 0; i < histogram.length; ++i)

    histogram = 0;



    ShortBuffer depth = depthMD.getData().createShortBuffer();

    depth.rewind();



    int points = 0;

    while(depth.remaining() > 0)

    {

    short depthVal = depth.get();

    if (depthVal != 0)

    {

    histogram[depthVal]++;

    points++;

    }

    }



    for (int i = 1; i < histogram.length; i++)

    {

    histogram += histogram[i-1];

    }



    if (points > 0)

    {

    for (int i = 1; i < histogram.length; i++)

    {

    histogram = (int)(256 * (1.0f - (histogram / (float)points)));

    }

    }

    }







    void updateDepth()

    {

    try {

    DepthMetaData depthMD = depthGen.getMetaData();



    context.waitAnyUpdateAll();



    calcHist(depthMD);

    ShortBuffer depth = depthMD.getData().createShortBuffer();

    depth.rewind();



    while(depth.remaining() > 0)

    {

    int pos = depth.position();

    short pixel = depth.get();

    imgbytes[pos] = (byte)histogram[pixel];

    // float MIN = 100;

    // float MAX = 200;

    //

    // if( histogram[pixel]>MIN && histogram[pixel]<MAX ){

    // imgbytes[pos] = (byte)histogram[pixel];

    // }else {

    // imgbytes[pos] = (byte)0;

    // }



    }

    } catch (GeneralException e) {

    e.printStackTrace();

    }

    }



    void updateColor(){

    try {

    ImageMap imageMap = imageGen.getImageMap();

    ByteBuffer byteBuffer = imageMap.createByteBuffer();



    context.waitAnyUpdateAll();

    //Image image = new Image(Image.Format.BGR8, 640, 480, byteBuffer) ;

    Image image = new Image(Image.Format.RGB8, 640, 480, byteBuffer) ;

    Texture texture = new Texture2D();

    texture.setImage(image);



    mat_color.setTexture("ColorMap", texture);



    } catch (GeneralException e) {

    e.printStackTrace();

    }

    }



    void createTexture(){

    DataBufferByte dataBuffer = new DataBufferByte(imgbytes, width    height);

    Raster raster = Raster.createPackedRaster(dataBuffer, width, height, 8, null);

    bimg.setData(raster);

    AWTLoader loader = new AWTLoader();

    int type = bimg.getType() == 0? BufferedImage.TYPE_INT_ARGB : bimg.getType();

    Texture tex = new Texture2D(loader.load(Test3Water.resizeImage(bimg, type), false));



    mat_tt.setTexture("ColorMap", tex);



    updateTerrain(tex);







    }

    void updateTerrain( Texture tex){

    //Terrain

    /** 2. Create the height map /

    AbstractHeightMap heightmap = null;

    heightmap = new ImageBasedHeightMap(tex.getImage());

    heightmap.load();

    int patchSize = 65;

    rootNode.detachChild(terrain);

    terrain = new TerrainQuad("my terrain", patchSize, 65, heightmap.getHeightMap());

    //terrain.setMaterial(mat_terrain);

    terrain.setMaterial(mat_color);

    terrain.setLocalTranslation(0, -100, 0);

    terrain.setLocalScale(2f, 1f, 2f);

    rootNode.attachChild(terrain);



    }

    void loadTerrain(){

    /    * 1. Create terrain material and load four textures into it. /

    mat_terrain = new Material(assetManager,

    "Common/MatDefs/Terrain/Terrain.j3md");



    // /    * 1.1) Add ALPHA map (for red-blue-green coded splat textures) /

    // mat_terrain.setTexture("Alpha", assetManager.loadTexture(

    // "Textures/Terrain/splat/alphamap.png"));



    /    * 1.2) Add GRASS texture into the red layer (Tex1). /

    Texture grass = assetManager.loadTexture(

    "Textures/Terrain/splat/grass.jpg");

    grass.setWrap(WrapMode.Repeat);

    mat_terrain.setTexture("Tex1", grass);

    mat_terrain.setFloat("Tex1Scale", 200f);



    /    * 1.3) Add DIRT texture into the green layer (Tex2) /

    Texture dirt = assetManager.loadTexture(

    "Textures/Terrain/splat/dirt.jpg");

    dirt.setWrap(WrapMode.Repeat);

    mat_terrain.setTexture("Tex2", dirt);

    mat_terrain.setFloat("Tex2Scale", 0f);



    /    * 1.4) Add ROAD texture into the blue layer (Tex3) /

    Texture rock = assetManager.loadTexture(

    "Textures/Terrain/splat/road.jpg");

    rock.setWrap(WrapMode.Repeat);

    mat_terrain.setTexture("Tex3", rock);

    mat_terrain.setFloat("Tex3Scale", 225f);



    /    * 2. Create the height map /

    AbstractHeightMap heightmap = null;

    Texture heightMapImage = assetManager.loadTexture(

    "Textures/Terrain/splat/mountains512.png");

    heightmap = new ImageBasedHeightMap(heightMapImage.getImage());

    heightmap.load();



    /    * 3. We have prepared material and heightmap.
  • Now we create the actual terrain:
  • 3.1) Create a TerrainQuad and name it "my terrain".
  • 3.2) A good value for terrain tiles is 64x64 – so we supply 64+1=65.
  • 3.3) We prepared a heightmap of size 512x512 – so we supply 512+1=513.
  • 3.4) As LOD step scale we supply Vector3f(1,1,1).
  • 3.5) We supply the prepared heightmap itself.

    /

    int patchSize = 65;

    terrain = new TerrainQuad("my terrain", patchSize, 65, heightmap.getHeightMap());



    /    * 4. We give the terrain its material, position & scale it, and attach it. /

    terrain.setMaterial(mat_terrain);

    terrain.setLocalTranslation(0, -100, 0);

    terrain.setLocalScale(2f, 1f, 2f);

    rootNode.attachChild(terrain);



    /    * 5. The LOD (level of detail) depends on were the camera is: */

    TerrainLodControl control = new TerrainLodControl(terrain, getCamera());

    terrain.addControl(control);

    }

    private static final int IMG_WIDTH = 64;

    private static final int IMG_HEIGHT = 64;

    private static BufferedImage resizeImage(BufferedImage originalImage, int type){

    BufferedImage resizedImage = new BufferedImage(IMG_WIDTH, IMG_HEIGHT, type);

    Graphics2D g = resizedImage.createGraphics();

    g.drawImage(originalImage, 0, 0, IMG_WIDTH, IMG_HEIGHT, null);

    g.dispose();



    return resizedImage;

    }



    private void initFloor() {







    Box floor = new Box(Vector3f.ZERO, 100f, 0.1f, 100f);

    // floor.scaleTextureCoordinates(new Vector2f(3, 6));



    floor_geo = new Geometry("Floor", floor);

    floor_geo.setMaterial(mat_solid);

    floor_geo.setLocalTranslation(0, -0.1f, 0);

    floor_geo.setLocalRotation(Quaternion.DIRECTION_Z);

    rootNode.attachChild(floor_geo);

    }



    }

    [/java]
5

I have body tracking working now.

Put a Model on his head and a sword in his hand.

I need to figure out rotation

Cool so far :slight_smile:



http://i.imgur.com/5jWdb.png

1 Like
6

Some Progress



http://youtu.be/6yDBERRDVxA

1 Like
7

Oh wow, looks like you’re getting somewhere :smiley:

8

Really coming along great. @glaucomardano (see Kinect game template) ought to be aware of this.

9

great work mate!

10

Thanks for the comments.



Anyone know of the best way to lock a background image to the full screen?



I want the kinect web cam image to be the background and stretch to fill the background.



Currently I have a Box that I scale to the image size but was wondering if there was a better way?



EDIT: I moved this question to a different Thread http://hub.jmonkeyengine.org/groups/general-2/forum/topic/how-to-set-a-background-texture/

11

Haha very nice ;D. I didn’t touch the kinect template project anymore, but some months ago I did a lot of changes on it, and I’m just re-uploading the files, and I’ll be posting here when it’s done. it might be useful for you, and be free to keep the project xD.

12

Re-uploaded : http://code.google.com/p/jmonkeyplatform-contributions/downloads/list

(another time I upload the source code of my modified jme3 version used on this project).

13

@glaucomardano I will have to check it out thanks.

14

A bit more progress.

Issues I’m having are with the figuring out the bone rotation on third level bones (elbow to hand) works for half of the rotation but after my arms are over the horizon the rotate completely in the wrong direction.





http://youtu.be/jclcWQl10lQ

3 Likes
15

Really nice job :>

16
@gbluntzer said:
A bit more progress.
Issues I'm having are with the figuring out the bone rotation on third level bones (elbow to hand) works for half of the rotation but after my arms are over the horizon the rotate completely in the wrong direction.


http://youtu.be/jclcWQl10lQ


Wow, thats really great already.
17

The Art Project went over really well tonight. byob-satx-open About 100 people showed up. Lots of people waving their arms and jumping up and down to move the skeletons. There was a good band playing http://www.facebook.com/SalesmanATX



Thanks to all the jMonkey Developers for making an awesome piece of software.

-Greg

1 Like
18

Oooh, didn’t realize this was part of an art project. Congrats man! Any pictures from the event?



Edit: Also didn’t realize that’s how you make an Oo smiley.

19

Hey,

idk if you saw it or if you are still working on this but at least some code for retargeting and other things popped up on the web :slight_smile:
https://code.google.com/p/dancedoll/ its GPL but for recording animations in the SDK thats fine :smiley:

20

And I guess you were searching for “Dance Doll” for academic purposes, right Normen?