Google Cardboard support

Oak · June 28, 2016, 11:58am

Anyone have any success making JME run with the new Google VR SDK?

There is a new GvrActivity and an interface GvrView.StereoRenderer which seem quite easy to hook into.

rickard · June 28, 2016, 2:12pm

I’ve looked at it, and yes it seems pretty straight forward. I just haven’t had the time to do anything. Maybe in a couple of weeks time. Mainly I want to get distortion meshes into the cardboard lib as well, and I don’t want to do that when it’s pretty much deprecated.

Oak · July 1, 2016, 6:58am

I may have misunderstood what “distortion meshes” are - but it seems the new API from Google requires nothing special but applying the Eye matrices before rendering. And then afterwards distortion is added according to the configuration/cardboard model.

https://developers.google.com/vr/android/samples/treasure-hunt#extend_gvractivity

It says:

The Google VR SDK applies distortion automatically, to render the final scene.

rickard · July 1, 2016, 7:54am

You’re probably right. I haven’t checked the API for a long time.

Oak · July 4, 2016, 3:50pm

I will try to get this running in a clone of the Google VR repo, please join if you like:

Oak · July 5, 2016, 2:20pm

I got something up and running - but not having worked with 3D-math for 10 years, I am uncertain I have done it right. A few things frustrate me:

I am unsure I hooked GoogleVR and jMonkeyEngine up a good way - there is something about how the objects know about each other that doesn’t seem quite right
I have to do something very wrong with a quaternion - which would suggest I am doing something wrong elsewhere
Some of the boxes in the example are rendered inside out… again, suggesting that I didn’t get it quite right.
The non-render-update(…) work should happen in the onNewFrame(…) call, while the render work should happen (twice) in the onDrawEye(…) call. Not sure how to get this split up, since SimpleGame does it all in “update()”.

JacobNordfalk · September 26, 2016, 9:54pm

Hello @Oak ,

Now the GVR 1.0 has arrived and supposedly the cardboard API is stable.

Did you get nearer to get your problems solved ?

Ive taken your work and updated it to GVR 1.0 and made a project with exactly the minimal stuff included. It’s here: GitHub - nordfalk/gvr-android-jme: JMonkeyEngine integration with Google VR SDK

I hope to solve 1) and split the harness into a small library which projects can depend on, and make/add/steal a few sample projects.

Yours,
Jacob

JacobNordfalk · September 28, 2016, 8:08am

Ive managed to get JME+Cardboard up and running in the x86 emulator and fixed some more bugs.

Ive added TestJaime from @rickard - it runs now, but it looks funny:

When I use @Oak 's version it looks like

This error is in the logs:
W/com.jme3.animation.SkeletonControl: WARNING Could not enable HW skinning due to shader compile error:
com.jme3.renderer.RendererException: Shader failed to link, shader:Shader[numSources=2, numUniforms=17, shaderSources=[ShaderSource[name=Common/MatDefs/Light/Lighting.frag, defines, type=Fragment, language=GLSL100], ShaderSource[name=Common/MatDefs/Light/Lighting.vert, defines, type=Vertex, language=GLSL100]]]
Vertex shader active uniforms exceed GL_MAX_VERTEX_UNIFORM_VECTORS (253)

                                      at com.jme3.renderer.opengl.GLRenderer.updateShaderData(GLRenderer.java:1279)
                                      at com.jme3.renderer.opengl.GLRenderer.setShader(GLRenderer.java:1291)

Rickard and Oak, ive added you to my repo (: GitHub - nordfalk/gvr-android-jme: JMonkeyEngine integration with Google VR SDK) if you want to participate from time to time.

Jacob

Ali_RS · September 28, 2016, 8:43am

Thought might be relevant to mention this here
Google VR SDK also added support for Daydream (Google’s project to bring VR to the Android platform) It is pretty new yet.

rickard · September 28, 2016, 5:49pm

Cheers! Good to see some more people getting involved in VR.

I think Jaime had some issues in the old cardboard lib as well. Possibly due to HW-skinning and lights.

JacobNordfalk · September 29, 2016, 9:04am

Yes, this controller is a really brilliant move - as soon as I get one I’ll start working on support.

BTW any idea why WASD keys are not working?
Ive enabled the fly camera and I’d suppose that would be enough (didnt look into this yet, but you might have a clue)

@rickard could you send me an APK file I can installl to see how it looked in your original cardboard app (I cant compile your code as its using the old API)

Jacob

rickard · October 2, 2016, 8:01am

Sorry, I don’t think I have any apk. It was a long time ago I worked on it.

JacobNordfalk · October 4, 2016, 9:38pm

A student and I have got a lot of progress.

Right now we have 3 obstacles:

The scene is rendered upside down
Terrain in the side of the view is not rendered. I think the culling is too agressive.
Textures are missing on terrains (also valid for a standard Android build).

Any clues or thoughts on these things?

As for 1) and 2) … could someone with experience in JME and 3D math have a look at gvr-android-jme/GvrOGLESContext.java at master · nordfalk/gvr-android-jme · GitHub ?

As for 3) we are using jmeEngineVersion = ‘3.1.0-beta1’. Would you think it could be solved if we move to another version ?

Thanks in advance!
Jacob

rickard · October 11, 2016, 3:24pm

Sorry for the late reply, I have been away.
1 sounds like a projection matrix issue. Alternatively, check that the matrices are converted properly. 2 as well sounds like the projection matrix.
I’ve sometimes had problems with using the “supposed” matrix (note that the lines are commented out in jme-cardboard/CardboardState.java at master · neph1/jme-cardboard · GitHub)
Try using the default proj matrix and see what happens.

JacobNordfalk · November 8, 2016, 8:32pm

Hi there.
After a lot of research I found the root cause of 3) (the missing textures):
The scene edtor wronly prefixes the path with /, and the AndroidLocator wronly has a / as rootpath, and that combination makes the texture loading fail.

An easy fix is to correct the wrong path in the Android Assetmanager:
assetManager.unregisterLocator("/", AndroidLocator.class);
assetManager.registerLocator("",AndroidLocator.class);

A temporary fix for 2) is to disable culling:
rootNode.setCullHint(Spatial.CullHint.Never);

JacobNordfalk · November 9, 2016, 4:41pm

OK, with 2) and 3) solved I took the time to debug the drawing part.

On the FIRST frame the scene is drawn correctly

The current code in GvrOGLESContext acutally, for unknown reasons, ignores the first call to onDrawEye, so only one eye is drawn, and its NOT upside down.

In subsequent calls rendering is upside down:

Any ideas?

rickard · November 10, 2016, 7:34am

Check what the projection matrices coming from the API say. I have this issue in AsteroidBeltTravel too, but it took me quite a long time to realize. (More difficult to establish up and down in a space environment).

ScienceOfficer · December 21, 2016, 10:11am

Hello

At the moment, I’m trying to get jMonkeyEngine to work under Android (which works great) with VR-Support. Unfortunately, I haven’t got it to work already.

I tried to use the gvr plugin from nordfalk, but i have some trouble, as all axis seems to be mirrored(?) - and even, if I mirror them again, there seems to be a bug with the depth buffer. Also I didn’t get the joystick to work.

Unfortunately, I’m doing this for my bachelor thesis and don’t have much time, so I tried to use rickards old cardboard plugin (also with the cardboard.jar he submitted to git) - which works and renders things correctly. I used his plugin plus the example together with JME3.1 Alpha 2. But: Only the left eye seems to be rendered (Except for the first rendering cycle! Both eyes are rendered in this step.). Does anyone have an idea, why this happens?

Thanks for your great work so far!

rickard · December 21, 2016, 10:39am

@ScienceOfficer It’s been so long since I worked on this that I don’t have any clear recollection of the issue you describe. If you care to do some debugging, you should put some lines in the update method of

github.com

neph1/jme-cardboard/blob/master/src/com/jme3/state/CardboardState.java

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package com.jme3.state;

import com.google.vrtoolkit.cardboard.Viewport;
import com.jme3.app.Application;
import com.jme3.app.SimpleApplication;
import com.jme3.app.state.AbstractAppState;
import com.jme3.app.state.AppStateManager;
import com.jme3.math.Matrix4f;
import com.jme3.math.Quaternion;
import com.jme3.math.Vector3f;
import com.jme3.renderer.Camera;
import com.jme3.renderer.RenderManager;
import com.jme3.renderer.ViewPort;
import com.jme3.scene.Spatial;
import com.jme3.system.android.CardboardContext;
import com.jme3.util.TempVars;

This file has been truncated. show original

Check : context.getRightEye().getViewport() and context.getRightEye().getEyeView() that they look alright (Unless it’s the left eye that isn’t drawn)

ScienceOfficer · December 21, 2016, 11:36am

Thank you very much for your quick reply!

The viewport is at

x: 734,
y: 0,
width: 734,
height: 813

in every cycle, including the first one, which seems right to me.
The Eyeview of the right eye seems to be the same as the eyeport on the left eye, except EyeView[12], which differs (right: -0.030832868, left: 0.02916713). But also, the values are almost the same for every cyle, differ only a little.

Exact values: http://pastebin.com/n2iT8cRy