Oculus Rift Support

So basically its working. Nice job rickard!

Since my application is a little bit more complex and uses the Netbeans Platform i’m using the AwtPanelsContext (see TestAwtPanels for an example) for rendering to multiple Panels. It works quite well but not with the rift ;).

I added the StereoCamAppState to the TestAwtPanels (see below for a test case) but i’m getting an NPE. Maybe someone sees something? The viewport seems to be null in the FilterPostProcessor.

Heres the log:
[java]
EDT: addNotify
EDT: addNotify
EDT: componentResized 400, 300
EDT: componentResized 400, 300
OGL: Throttling update loop.
HMDInfo [HResolution = 1280, VResolution = 800, HScreenSize = 0.14976, VScreenSize = 0.0936, VScreenCenter = 0.0468, EyeToScreenDistance = 0.041, LensSeparationDistance = 0.0635, InterpupillaryDistance = 0.0647, DistortionK = [1.0, 0.22, 0.24, 0.0] , DesktopX = 0, DesktopY = 0, DisplayDeviceName = , DisplayId = 0]
Aug 13, 2013 11:25:30 AM com.jme3.app.Application handleError
Schwerwiegend: Uncaught exception thrown in Thread[LWJGL Renderer Thread,5,main]
java.lang.NullPointerException
at com.jme3.post.FilterPostProcessor.reshape(FilterPostProcessor.java:412)
at com.jme3.system.awt.AwtPanel.reshapeInThread(AwtPanel.java:270)
at com.jme3.system.awt.AwtPanel.initialize(AwtPanel.java:233)
at com.jme3.renderer.RenderManager.renderViewPort(RenderManager.java:955)
at com.jme3.renderer.RenderManager.render(RenderManager.java:1029)
at com.jme3.app.SimpleApplication.update(SimpleApplication.java:252)
at com.jme3.system.awt.AwtPanelsContext.updateInThread(AwtPanelsContext.java:188)
at com.jme3.system.awt.AwtPanelsContext.access$100(AwtPanelsContext.java:44)
at com.jme3.system.awt.AwtPanelsContext$AwtPanelsListener.update(AwtPanelsContext.java:68)
at com.jme3.system.lwjgl.LwjglOffscreenBuffer.runLoop(LwjglOffscreenBuffer.java:125)
at com.jme3.system.lwjgl.LwjglOffscreenBuffer.run(LwjglOffscreenBuffer.java:151)
at java.lang.Thread.run(Thread.java:722)

EDT: removeNotify
EDT: removeNotify
AL lib: (EE) alc_cleanup: 1 device not closed
Initializing Rift…
pHMD created
Attaching sensor
[/java]

And here a Test Case:
[java]
package jrift;

import com.jme3.app.SimpleApplication;
import com.jme3.app.state.StereoCamAppState;
import com.jme3.asset.plugins.FileLocator;
import com.jme3.material.Material;
import com.jme3.math.Vector3f;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.system.AppSettings;
import com.jme3.system.awt.AwtPanel;
import com.jme3.system.awt.AwtPanelsContext;
import com.jme3.system.awt.PaintMode;
import java.awt.BorderLayout;
import java.awt.Dimension;
import java.awt.Toolkit;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import java.util.concurrent.Callable;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.JFrame;
import javax.swing.SwingUtilities;

public class TestAwtPanels extends SimpleApplication {

private static TestAwtPanels app;
private static AwtPanel panel, panel2;
private static int panelsClosed = 0;
private boolean stereo = true;

private static void createWindowForPanel(AwtPanel panel, int location){
    JFrame frame = new JFrame("Render Display " + location);
    frame.getContentPane().setLayout(new BorderLayout());
    frame.getContentPane().add(panel, BorderLayout.CENTER);
    frame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
    frame.addWindowListener(new WindowAdapter() {
        @Override
        public void windowClosed(WindowEvent e) {
            if (++panelsClosed == 2){
                app.stop();
            }
        }
    });
    frame.pack();
    frame.setLocation(location, Toolkit.getDefaultToolkit().getScreenSize().height - 400);
    frame.setVisible(true);
}

public static void main(String[] args){
    Logger.getLogger("com.jme3").setLevel(Level.WARNING);
    
    app = new TestAwtPanels();
    app.setShowSettings(false);
    AppSettings settings = new AppSettings(true);
    settings.setCustomRenderer(AwtPanelsContext.class);
    settings.setFrameRate(60);
    app.setSettings(settings);
    app.start();
    
    SwingUtilities.invokeLater(new Runnable(){
        public void run(){
            final AwtPanelsContext ctx = (AwtPanelsContext) app.getContext();
            panel = ctx.createPanel(PaintMode.Accelerated);
            panel.setPreferredSize(new Dimension(400, 300));
            ctx.setInputSource(panel);
            
            panel2 = ctx.createPanel(PaintMode.Accelerated);
            panel2.setPreferredSize(new Dimension(400, 300));
            
            createWindowForPanel(panel, 300);
            createWindowForPanel(panel2, 700);
        }
    });
}

@Override
public void simpleInitApp() {
    assetManager.registerLocator("./Assets/MatDefs", FileLocator.class);
    assetManager.registerLocator("./Assets/MatDefs/Post", FileLocator.class);
    assetManager.registerLocator("./Assets", FileLocator.class);
    assetManager.registerLocator("./Assets/Materials", FileLocator.class);
    
    flyCam.setDragToRotate(true);
    
    StereoCamAppState stereoCamAppState = new StereoCamAppState();
    stateManager.attach(stereoCamAppState);

    
    Box b = new Box(Vector3f.ZERO, 1, 1, 1);
    Geometry geom = new Geometry("Box", b);
    Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
    mat.setTexture("ColorMap", assetManager.loadTexture("Interface/Logo/Monkey.jpg"));
    geom.setMaterial(mat);
    rootNode.attachChild(geom);
    
    panel.attachTo(true, viewPort);
    guiViewPort.setClearFlags(true, true, true);
    panel2.attachTo(false, guiViewPort);
}

}
[/java]

On a tangent… I eat my words that the Oculus Rift is going to stay niche for a while.
See this link what’s going to sell that thing… yes, THAT particular industry, and the price tag is just right for that particular audience.

@toolforger said: On a tangent... I eat my words that the Oculus Rift is going to stay niche for a while. See this link what's going to sell that thing... yes, THAT particular industry, and the price tag is just right for that particular audience.

I don’t think so. Unless they add a camera so you find your dick and the paper towels…

Sony doesn’t believe much in 3D either anymore as their last investments in that direction didn’t work out well. Personally, I think the Oculus people have kind of an “Apple advantage”: They found the parameter thats really important to get to the next level, namely the accurate head tracking. I think this will make all the difference and convert quite some many people if they get enough people to try it :slight_smile:

I was very happy to find out that someone is working on integrating Oculus Rift with jME., good job! Following this thread I managed to almost get it up and running in an hour. Currently there is some problem related to

java.lang.RuntimeException: Uncompilable source code - Erroneous sym type: com.jme3.post.FilterPostProcessor.getFilterList

but i figure it might be just another missing lib or conflict. Will report back once its up and running.

I think you’re overly pessimistic about the practical problems, any required hardware can be put within easy reach before putting on the headpiece.

And heck I don’t believe in Sony’s predictions :stuck_out_tongue_winking_eye:
Agreed on the Apple advantage. They didn’t get just that one parameter into acceptable range, but all of them, including price tag, field-of-vision coverage, resolution, and low bulk. Identifying all parameters and pushing stuff out only after all of them are acceptable is actually Apple’s marketing style, too, so they rightfully have an Apple advantage.

They now need to keep on their toes to keep that advantage though. I bet all game hardware companies are currently frantically working on me-too products.
That’s going to be good for consumers since mass production will help make the sensors and displays cheaper.
Not so good for JME because it will take a while until vendors can agree on a standard API, so there’s going to be different JME plugins for different hardware for a while.

<cite>@Lockhead said:</cite> So basically its working. Nice job rickard!

Since my application is a little bit more complex and uses the Netbeans Platform i’m using the AwtPanelsContext (see TestAwtPanels for an example) for rendering to multiple Panels. It works quite well but not with the rift ;).

I added the StereoCamAppState to the TestAwtPanels (see below for a test case) but i’m getting an NPE. Maybe someone sees something? The viewport seems to be null in the FilterPostProcessor.

[/java]

So i deactivated the BarrelDistortionFilter in the StereoCamAppState and now at least i get no exception and the doubled view but not centered and at the bottom of the view. Is the filter responsible for some of that?


Lockhead:

To answer your first problem; You will ge that NPE if reshape is called before the filter has been initialized. This can happen if reshape is called directly after being added for example, since initFilter, (i assume) is called in a thread safe manner.

The answer to the second question is, yes. The barrelfilter changes the projection matrix of the cameras so the images converges. In hindsight, perhaps the appstate should do this instead. It was just more convenient at the time. I’ll see if i should change this.

Sorry, don’t have time to look closer at the code right now.

@phr00t

The Oculus SDK v0.2.4 has been released! Release Highlights:
Added magnetometer calibration tool to the Oculus Config Utility. Saved calibration data can be used by Oculus SDK to automatically correct for yaw drift.
Saved magnetometer calibration data will now automatically be used by applications built with the SDK, including C++ and Unity samples.
<strong>Added Linux support for the Oculus - Unity integration.</strong></blockquote>

Seems like there is a bigger chance of getting it to work on linux now.

Personally though, my current prio is to making something of a showcase of the lib this weekend.

If you’re curious Rickard, this is what the project currently looks like.

Really nice looking!
I don’t understand the “game” element in it, but it’s looking really impressive technically.
Is the destructible geometry based on an article or something in the forum (care to link in that case?). I’d like to learn the technique.

Hope to be able to get going on something myself soon. :slight_smile:
I think i’ll start of with something character-based. I’ve also bought a Razer Hydra, which is fun to play with, but not so practical.

Yup, no game elements yet, still working on the core engine, 1 week left, yikes.

Anyways the destruction is based on CSG (constructive solid geometry) code from http://hub.jmonkeyengine.org/forum/topic/constructive-solid-geometry-code-for-jmonkey-give-away/ I had to make many adjustments to get to my point now though

Hmm, as of yesterday, my oculus fails to initialize properly in jmonkeyengine, and i haven’t made any changes to the API/lib.
Possibly related to a windows update on thursday? And this message in the log?

WARNING: Found unknown Windows version: Windows 7
I don't know, but i don't recognize it.

that doesn’t sound good at all

Ok, solved my problem (only took all day!).
Noticed i wasn’t using latest jdk, so i updated. Noticed i was building the dll with jdk6 linked. So i switched.
I’ve built the libs using the new 0.2.4 OVR. Will upload after some more testing.

Edit: Nope. Now it’s broken again. When it worked i didn’t get the “Unkown windows” error.

I got my rift a couple of days ago! I have been waiting about 20 years for this, ever since I saw “Lawnmover Man” back in 1992… :slight_smile:

I was exited to see you working on a port for Jmonkey Rickard, and tried to fire it up.

Unfortunately I’m getting:

java.lang.Exception: Oculus Rift could not be initialized
at oculusvr.input.OculusRiftReader.<init>(OculusRiftReader.java:36)
at oculusvr.TestOculus.simpleInitApp(TestOculus.java:41)

…when trying to run one of the examples. Is that the same init problem you where experiencing? Any progress on that?

/Henrik

Yes, it’s the same i’m getting. I didn’t make any progress, but i’ve been waiting for a new rig and i’m in the middle of getting it up to speed. The laptop was just too cramped with its 3 usb ports to develop for the rift with (and its limited resolution was of no help either). I’m hoping i’ll be able to look into it this weekend.

@rickard: I could not get your code to work. Ended up with what looked like one image covering the whole screen. Not sure if it because of not getting the hacked classes first in the classpath.

Anyway, I wrote my own version that sets up stereo a bit differently. It creates two offscreenbuffers, one for each eye. Maybe not the best way but it works without needing to hack anything.

Because it uses two offscreen buffers the clamping in the fragment shader was modified.

Lastly I could not manage to calculate the projectionCenterOffset correctly. The images would not converge. So I found a value that worked by trial and error. It can be manually changed with the “T” and “G” keys. I turned off the barreldistortion when I found the value.

[java]
package mygame;

import com.jme3.app.SimpleApplication;
import com.jme3.asset.AssetManager;
import com.jme3.input.KeyInput;
import com.jme3.input.event.KeyInputEvent;
import com.jme3.light.AmbientLight;
import com.jme3.light.DirectionalLight;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.FastMath;
import com.jme3.math.Matrix4f;
import com.jme3.math.Quaternion;
import com.jme3.math.Vector2f;
import com.jme3.math.Vector3f;
import com.jme3.math.Vector4f;
import com.jme3.renderer.Camera;
import com.jme3.renderer.RenderManager;
import com.jme3.renderer.ViewPort;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.texture.FrameBuffer;
import com.jme3.texture.Image;
import com.jme3.texture.Texture2D;
import com.jme3.ui.Picture;
import de.fruitfly.ovr.HMDInfo;
import de.fruitfly.ovr.OculusRift;

public class OculusRiftTestApp extends SimpleApplication {

public static void main(String[] args) {
    new OculusRiftTestApp().start();
}

Geometry cube = new Geometry("white cube", new Box(Vector3f.ZERO, 1, 1, 1));
Head head;
float theProjectionCenterOffset = 0.22834f;
HMDInfo hmd;
private boolean[] keyState = new boolean[0xff];

public void simpleInitApp() {
    rootNode.attachChild(assetManager.loadModel("Models/bunker/bunker.scene"));
    
    AmbientLight ambient = new AmbientLight();
    ambient.setColor(new ColorRGBA(0.5f, 0.5f, 0.5f, 1.0f));
    rootNode.addLight(ambient);

    DirectionalLight diffuse = new DirectionalLight();
    diffuse.setColor(new ColorRGBA(0.7f, 0.7f, 0.7f, 1.0f));
    diffuse.setDirection(new Vector3f(-1.0f, -1.0f, -1.0f));
    rootNode.addLight(diffuse);
    
    cube.setMaterial(new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md"));
    rootNode.attachChild(cube);
    viewPort.detachScene(rootNode);
    viewPort.setClearFlags(false, false, false);
    
    cam.setLocation(new Vector3f(0, 1, 4));

    OculusRift or = new OculusRift();
    or.init();
    
    hmd = or.getHMDInfo();
    System.out.println(hmd);
    
    head = new Head(or);
    
    inputManager.addRawInputListener(new RawInputListenerAdapter() {

        @Override
        public void onKeyEvent(KeyInputEvent evt) {
            keyState[evt.getKeyCode() % keyState.length] = (evt.isPressed() || evt.isRepeating());
            if (evt.isRepeating()) {
                return;
            }
            
            if (evt.isPressed()) {
                if (evt.getKeyCode() == KeyInput.KEY_SPACE) {
                }
            }
        }
    });
}


@Override
public void simpleRender(RenderManager rm) {
    super.simpleRender(rm);
}

@Override
public void simpleUpdate(float tpf) {
    // make the player rotate:
    cube.rotate(0, 2 * tpf, 0);
    
    if (keyState[KeyInput.KEY_T]) {
        theProjectionCenterOffset -= tpf * 0.1f;
        System.out.println("theProjectionCenterOffset=" + theProjectionCenterOffset);
    }
    if (keyState[KeyInput.KEY_G]) {
        theProjectionCenterOffset += tpf * 0.1f;
        System.out.println("theProjectionCenterOffset=" + theProjectionCenterOffset);
    }
    
    head.update(cam);
}

class Head {
    public final OculusRift or;
    public final Eye left;
    public final Eye right;
    
    Head(OculusRift or) {
        this.or = or;
        left = new Eye(true);
        right = new Eye(false);            
        guiNode.attachChild(left.picture);
        guiNode.attachChild(right.picture);
    }
    
    public void update(Camera center) {
        if (or.isInitialized()) {
            or.poll();
        }
        
        Vector3f centerLocation = center.getLocation();
        Vector3f leftEyeVec = center.getLeft().mult(hmd.InterpupillaryDistance / 2);
        Vector3f leftEyeWorld = new Vector3f(centerLocation).add(leftEyeVec);
        Vector3f rightEyeWorld = new Vector3f(centerLocation).subtract(leftEyeVec);            
        
        Quaternion camRotation = new Quaternion(center.getRotation());
        Quaternion rotation = new Quaternion();
        rotation.fromAngles(-or.getPitch(), or.getYaw(), -or.getRoll());
        camRotation.mult(rotation, camRotation);
        
        left.camera.setLocation(leftEyeWorld);
        left.camera.setRotation(camRotation);
        right.camera.setLocation(rightEyeWorld);
        right.camera.setRotation(camRotation);
        
        left.updateBarrelDistortion();
        right.updateBarrelDistortion();
    }
}

class Eye {
    
    boolean isLeft;
    Camera centerCamera;
    Camera camera;
    Texture2D texture;
    Picture picture;
    
    private float scaleFactor = 0.88f;
    
    
    Eye(boolean isLeft) {
        this.isLeft = isLeft;
        this.centerCamera = cam;
        String name = isLeft ? "Left" : "Right";
        camera = new Camera(cam.getWidth()/2, cam.getHeight());
        ViewPort viewPort = renderManager.createMainView(name, camera);
        viewPort.setClearFlags(true, true, true);
        FrameBuffer frameBuffer = new FrameBuffer(camera.getWidth(), camera.getHeight(), 1);
        texture = new Texture2D(camera.getWidth(), camera.getHeight(), Image.Format.RGBA8);
        frameBuffer.setColorTexture(texture);
        frameBuffer.setDepthBuffer(Image.Format.Depth);
        viewPort.setOutputFrameBuffer(frameBuffer);
        viewPort.attachScene(rootNode);
        picture = new Picture(name+"Picture");
        picture.setWidth(camera.getWidth());
        picture.setHeight(camera.getHeight());
        picture.setPosition(isLeft ? 0 : camera.getWidth(), 0);
        picture.setMaterial(createBarrelDistortion(isLeft, assetManager));
    }
    
    Material createBarrelDistortion(boolean isLeft, AssetManager assetManager) {
        Material material = new Material(assetManager, "MatDefs/Post/BarrelDistortion.j3md");

        float aspectRatio = (float)(camera.getWidth() * 1f) / (float) camera.getHeight();

        float halfScreenDistance = (hmd.VScreenSize / 2f);
        float yfov = 2.0f * FastMath.atan(halfScreenDistance/hmd.EyeToScreenDistance);
        centerCamera.setFrustumPerspective(FastMath.RAD_TO_DEG * yfov, aspectRatio, 0.1f, 10000f);

        float viewCenter = hmd.HScreenSize * 0.25f;
        float eyeProjectionShift = viewCenter - hmd.LensSeperationDistance * 0.5f;
        float projectionCenterOffset = 4.0f * eyeProjectionShift / hmd.HScreenSize;
        
        projectionCenterOffset = theProjectionCenterOffset;
        if (!isLeft){
            projectionCenterOffset = -projectionCenterOffset;
        }
        
        Matrix4f mat = new Matrix4f();
        mat.setTranslation(projectionCenterOffset, 0 ,0);
        mat.multLocal(centerCamera.getProjectionMatrix());
        camera.setProjectionMatrix(mat);

        float x = 0f;
        float y = 0f;
        float w = 1f;
        float h = 1f;
        
        Vector2f lensCenter = new Vector2f((w + projectionCenterOffset * 0.5f)*0.5f, h*0.5f);
        Vector2f screenCenter = new Vector2f(x + w * 0.5f, y + h * 0.5f);
        Vector2f scale = new Vector2f((w / 2.0f) * scaleFactor, (h / 2.0f) * scaleFactor * aspectRatio);
        Vector2f scaleIn = new Vector2f(2f / w, 2f / h / aspectRatio );
        material.setTexture("Texture", texture);
        material.setVector2("LensCenter", lensCenter);
        material.setVector2("ScreenCenter", screenCenter);
        material.setVector2("Scale", scale);
        material.setVector2("ScaleIn", scaleIn);
        float[] K = hmd.DistortionK;
        material.setVector4("HmdWarpParam", new Vector4f(K[0], K[1], K[2], K[3]));
        return material;
    }
    
    void updateBarrelDistortion() {
        float projectionCenterOffset = theProjectionCenterOffset;
        if (!isLeft){
            projectionCenterOffset = -projectionCenterOffset;
        }
        
        Matrix4f mat = new Matrix4f();
        mat.setTranslation(projectionCenterOffset, 0 ,0);
        mat.multLocal(centerCamera.getProjectionMatrix());
        camera.setProjectionMatrix(mat);
        
    }
}

}
[/java]

Modified fragment shader:
[java]
uniform sampler2D m_Texture;
varying vec2 texCoord;

uniform vec2 m_LensCenter;
uniform vec2 m_ScreenCenter;
uniform vec2 m_Scale;
uniform vec2 m_ScaleIn;
uniform vec4 m_HmdWarpParam;

vec2 HmdWarp(vec2 texIn){
vec2 theta = (texIn - m_LensCenter) * m_ScaleIn;
float rSq= theta.x * theta.x + theta.y * theta.y;
vec2 theta1 = theta * (m_HmdWarpParam.x + m_HmdWarpParam.y * rSq +
m_HmdWarpParam.z * rSq * rSq + m_HmdWarpParam.w * rSq * rSq * rSq);
return m_LensCenter + m_Scale * theta1;
}

void main(){
vec2 tc = HmdWarp(texCoord);
//if (any(notEqual(clamp(tc, m_ScreenCenter-vec2(0.25,0.5), m_ScreenCenter+vec2(0.25, 0.5)) - tc, vec2(0.0, 0.0)))){
// discard;//gl_FragColor = vec4(0.5, 0.5, 0.5, 1.0);
// }
// else{
gl_FragColor = texture2D(m_Texture, tc);
// }
}
[/java]

Not sure if it because of not getting the hacked classes first in the classpath.

I don’t know what you mean, by this.

He hacked some classes and put them in the class path before the “official” ones.

Hey monkeys!
Sorry that I didn’t properly introduce myself. I’m from Argentina and I’m actually playing a little bit with the Oculus VR.

You are amazing! =D

I download the code and try and found that I had the same problem that someone here has: I couldn’t get the splitted display (both images were on top of each other).

I was debugging a little bit to find out that I think it has to do with the material definition. I guess is related with the shader version.

I changed “BarrelDistortion.j3md” a bit:
[java]
MaterialDef Barrel {

MaterialParameters {
    Int NumSamples
    Texture2D Texture
    Vector2 LensCenter
    Vector2 ScreenCenter
    Vector2 Scale
    Vector2 ScaleIn
    Vector4 HmdWarpParam
    Texture2D WarpTexture
}

 Technique {
    VertexShader GLSL150:   Common/MatDefs/Post/Post15.vert
    FragmentShader GLSL150: MatDefs/Post/BarrelDistortion15.frag

    WorldParameters {
        WorldViewProjectionMatrix
    }

    Defines {
        RESOLVE_MS : NumSamples
    }

}

Technique {
    VertexShader GLSL100:   Common/MatDefs/Post/Post.vert
    FragmentShader GLSL100: MatDefs/Post/BarrelDistortion.frag

    WorldParameters {
        WorldViewProjectionMatrix
    }
}

}[/java]

and now It’s working!

Hope this help someone,

Regards,

2 Likes