# Accessing VR controller button presses - expand openVr for more buttons

I’ve been trying to set up a VR project using an Oculus Quest 2* (Over virtual desktop so it pretends to be a PC VR system). Its all working remarkably well.

*yes, this was an Christmas present, I’m very happy with it

## Background

However, the LWJGLOpenVRInput (Which seems like the right one to be using) is very Vive focused, e.g. the isButtonPressed method uses Vive specific constants (all the constants are in VRInputType). Despite the names being Vive specific it all works well for the oculus as well (I love open standards). However the oculus has a lot more buttons than the Vive (which come though just fine in the bitfield ulButtonPressed so they almost work, but there’s no way to get at them).

## What I’m thinking of doing

I want to add these extra buttons into VRInputType and will soon be putting in a PR for that.

But the question is what to call them. ViveButtonOne, ViveButtonTwo etc would be consistent with the other OpenVR buttons, but the vive doesn’t have those buttons. Equally its a bit confusing how these seem vive specific but aren’t. What I’d probably like to call these are OpenVrButtonOne etc. If I was given the ok to be a bit more destructive I’d probably also deprecate the existing ViveGripButton etc and create new enums called OpenVrGripButton etc.

Similarly button touched** “almost works”, so I intend to make that work as well (it comes through in the bitfield ulButtonTouched). But I might only make that work for LWJGLOpenVRInput unless its obvious how to do it for other input mappings (Also harder for me to test those) and leave the others as UnsupportedOperationException.

** button touched being when the finger rests on the button but doesn’t press it in.

All that feels a little controversial so I wanted to gather opinions before I just go and do it.

1 Like

Hmm, although the X and Y buttons are at kind of random bit positions in the bitmask

01000000000000000000000000000000000 - trigger - 2^33
10000000000000000000000000000000100 - grip - 2^34 or 2^2
00000000000000000000000000010000000 - X - 2^7
00000000000000000000000000000000010 - Y - 2^1
00100000000000000000000000000000000 - controller hat - 2^32

I expect the trigger, grip and controller hat probably translate well to other controllers. But the X and Y aren’t where I expected them to be. Maybe it won’t be as standard as I had hoped

Last time I was poking around the jme3-vr module, I realized almost the entirety of it needs to be rewritten for modern VR, everything we can do at this point is patchwork. It’s very much structured around 2016-17 era VR where each of the manufacturers had their own API you had to support.

For input, jme-vr is using the old way of getting input from OpenVR (device based (hardware specific) instead of action based (hardware neutral)) and in my attempts to rectify that (even though it would change the entire API) I’ve hit a bug either in lwjgl’s OpenVR binding or in SteamVR itself, as the actions didn’t report their values. So I kinda gave up on it.

Either way, OpenVR as a platform is being sunsetted in favor of OpenXR, which has way better cross platform compatibility, but sadly lwjgl still doesn’t provide a binding for it. So for a major rewrite we’d have to either make our own bindings or wait indefinitely until lwjgl folks bother to support it.

1 Like

Looks like a PR is submitted but still not merged.

1 Like

Thanks @Ali_RS & @grizeldi, that’s very useful information.

In that case I may change my plans. I’ll keep an eye on that PR and hope (open for 10 months isn’t a great sign), if it does get merged I’ll look at getting openXR to work with jme.

In the meantime I will move onto what I was going to do next; try to get a jme3-android + jme3-vr application to work running locally on the oculus quest. If I can do that I’ll document it and add enum values for the oculus buttons (It only makes sense if I can because no one will want to target my weird virtual desktop set up but a locally installed via the oculus store is a known quantity). Then at least 2016 style vr will work on the quest, which is perhaps the only place where such an approach is viable as the hardware will be constant there.

3 Likes

OpenXR is on the docket for LWJGL 3.4.0. now that the hurdle of 3.3.0 is out, I imagine we will see a lot of progress on 3.4.0.

2 Likes

Looks like it is milestoned for 3.3.1

3 Likes

Wow, even better! I am very excited for it as I will be using it in several applications once we get LWJGL support for it.

Does anybody know if OpenXR is supported on Android?
I could not find much from google.

Edit:
Nevermind, seems it does:

2 Likes

To report on progress with Oculus/android/vr (partially for my own notes).

A “normal” android app (displayed on a floating virtual monitor in VR) worked almost immediately, full VR has been more troublesome

VREnvironment uses the system property sun.arch.data.model but that is sun/oracle specific. os.arch seems to be the non-sun version, but its value is not exactly the same (outputs Aarch64, not just 64)

Change

vrSupportedOS = System.getProperty("sun.arch.data.model").equalsIgnoreCase("64");


To

vrSupportedOS = System.getProperty("os.arch").contains("64");


VRAppState uses GraphicsDevice, but that doesn’t exist on android. It seemed like the way the code flow went though it didn’t actually need it, so pushing it inside the if resolved that.

Current blocker is that liblwjgl.so seems not to be being bundled and I’m getting errors like

W/System.err: [LWJGL] Failed to load a library. Possible solutions:
a) Add the directory that contains the shared library to -Djava.library.path or -Dorg.lwjgl.librarypath.
b) Add the JAR that contains the shared library to the classpath.
[LWJGL] Enable debug mode with -Dorg.lwjgl.util.Debug=true for better diagnostics.
D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.jmonkeyandroidvr2, PID: 27282
java.lang.UnsatisfiedLinkError: Failed to locate library: liblwjgl.so
at org.lwjgl.system.Library.<clinit>(Library.java:50)
at org.lwjgl.openvr.VR.<clinit>(VR.java:3058)
at org.lwjgl.openvr.VR.VR_InitInternal(VR.java:3103)
at com.jme3.input.vr.lwjgl_openvr.LWJGLOpenVR.initialize(LWJGLOpenVR.java:143)
at com.jme3.app.VREnvironment.initialize(VREnvironment.java:485)
at com.example.game.Game.bootVr(Game.java:32)
at com.example.game.Game.<init>(Game.java:24)


Which is odd, as I’ve added jme3-lwjgl3 into the build.gradle. Does JME still do the “unpacking dlls” thing, I wonder if that’s not allowed in android world and that’s messing things up.

I’ve started playing with some of those “Possible solutions”, but that is a problem for another day.

1 Like

The unsatisfied link error is probably the same as this one:

1 Like

Hmm, I started a new project with gradle 7.0.2, so must be something different. Now that I think about it, the fact that a normal jme android app (floating window) works fine means that the lwjgl stuff must be there. But something about how VR loads it must be different. (I’ve tried booting VR both just before the SimpleApplication boots up and within the simpleInitApp). Still the fact normal jmonkey android stuff works and VR doesn’t does open up new avenues for investigation

Edit: I’m getting some way to understanding why that is. For VR liblwjgl it loaded as:

private static final SharedLibrary OPENVR = Library.loadNative(VR.class, "org.lwjgl.openvr", Configuration.OPENVR_LIBRARY_NAME.get(Platform.mapLibraryNameBundled("openvr_api")), true);

So although it doesn’t sound like its VR specific from the error message it actually is

Edit 2:

Actually edit 1 was wrong. Its a method call to load one .so triggering a static call to load a different .so. So its not actually VR specific

1 Like

Further progress (of a sort)

The standard set of dependencies to get android working do not include (either directly or transitively) lwjgl or lwjgl3

implementation "org.jmonkeyengine:jme3-android:${jmonkeyengine_version}" implementation "org.jmonkeyengine:jme3-android-native:3.4.0-stable" implementation "org.jmonkeyengine:jme3-core:$jmonkeyengine_version"


(Does jme3-android-native contain something like lwjgl?)

implementation "org.jmonkeyengine:jme3-vr:\$jmonkeyengine_version"


I implicitly get

org.jmonkeyengine:jme3-lwjgl3
org.jmonkeyengine:jme3-desktop


I gradle excluded jme3-desktop but presumed I needed jme3-lwjgl3.

If I run like this I get

* What went wrong:
> 42 files found with path 'META-INF/INDEX.LIST' from inputs:

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org


Based on what the internet had to say I added this to gradle (I was suspicious of this to begin with if I’m honest)

android{
....
packagingOptions {
exclude 'META-INF/INDEX.LIST'
}

}


This I suspect was a mistake. Because a load of these META-INF/INDEX.LIST files talk about lwjgl and generally look important. So I think the reason normal jme-android works but jme-vr-android doesn’t is because I’ve mangled my project to get org.jmonkeyengine:jme3-vr in with it still building

1 Like

Figured I would give an update I just saw, LWJGL just merged in support for OpenXR:
feat(OpenXR): add OpenXR bindings. Close #569 · LWJGL/[email protected] (github.com)

3 Likes

I’ve continued my VR experiments and it’s been a tale of 2 halves. Android hasn’t gone well, but Actions based upgrade for JME’s OpenVR has gone well.

# Android

Oculus do include documentation on using OpenXR bindings. However its all very C++ focused and I got out of my depth very quickly. I believe that we don’t use LWJGL in our android apps because android has java support for OpenGL without it. That said LWJGL now does support android and that would probably make android VR much easier.

Regardless I am officially giving up on Android VR for now at least (but will someday revisit, especially in light of OpenXR on LWJGL)

# PC VR

Trying to upgrade to allow JMonkey to use Actions based input on OpenVR (rather than the legacy input it currently uses) has gone much much better though. (Perhaps the bug @grizeldi hit has been resolved?). I have got a hybrid application where I mostly use JME to boot up a VR context then make some direct LWJGL calls to get actions based input values.

I realise in the long term we may want to look at OpenXR, but this will hopefully take us up to 2020 VR rather than 2016 VR, which is a step up at least.

### Action Manifests

In action based VR the application declares the actions it wants as well as mappings from buttons to those actions for controllers the application is aware of (the nice thing about this is if a new controller comes along the user themselves can configure the mappings themselves using (for example) SteamVR. These files must exist at a physical file location (i.e. they can’t just be in resources within the jar)

My actions manifest files were as follows for this test

actionsManifest.json

{
"default_bindings": [
{
"controller_type": "oculus_touch",
"binding_url": "oculusTouchDefaults.json"
}
],
"actions": [
{
"name": "/actions/main/in/OpenInventory",
"requirement": "mandatory",
"type": "boolean"
},
{
"name": "/actions/main/in/test2",
"requirement": "mandatory",
"type": "boolean"
},
{
"name": "/actions/main/in/scroll",
"type": "vector2",
"requirement": "mandatory"
}
],
"action_sets": [
{
"name": "/actions/main",
"usage": "leftright"
}
],
"localization" : [
{
"language_tag": "en_us",
"/actions/main" : "My Game Actions",
"/actions/main/in/OpenInventory" : "Open Inventory"
}
]
}


OculusTouchDefaults.json

{
"action_manifest_version" : 0,
"bindings": {
"/actions/main": {
"sources" : [
{
"inputs" : {
"click" : {
"output" : "/actions/main/in/OpenInventory"
}
},
"mode" : "button",
"path" : "/user/hand/left/input/x"
},
{
"inputs" : {
"click" : {
"output" : "/actions/main/in/test2"
}
},
"mode" : "button",
"path" : "/user/hand/left/input/y"
},
{
"inputs" : {
"position" : {
"output" : "/actions/main/in/scroll"
}
},
"mode" : "joystick",
"path" : "/user/hand/left/input/joystick"
}
]
}
},
"category" : "steamvr_input",
"controller_type" : "oculus_touch",
"description" : "Bindings for the jmetest demo for a oculusTouch controller",
"name" : "jmetest bindings for a oculusTouch controller",
"options" : {},
"simulated_actions" : []
}


### Java code

Within Java I got handles to the actions during the application initialisation (just after the call to VREnvironment#initialize) as well as setting up my objects that talk to native buffers. (This is more efficient than doing them every time)

public static void main(String[] args) {
AppSettings settings = new AppSettings(true);
settings.put(VRConstants.SETTING_VRAPI, VRConstants.SETTING_VRAPI_OPENVR_LWJGL_VALUE);

VREnvironment env = new VREnvironment(settings);

env.initialize();

if (env.isInitialized()){
VRAppState vrAppState = new VRAppState(settings, env);

VRInput.VRInput_SetActionManifestPath("C:/Users/richa/Documents/Development/jmonkeyVrTest/src/main/resources/actionManifest.json"); //hard coded for experimental purposes

//LongBuffer longBuffer = ByteBuffer.allocate( (Long.SIZE / 8) * 100 ).order(java.nio.ByteOrder.nativeOrder()).asLongBuffer();
LongBuffer longBuffer = BufferUtils.createLongBuffer(1);
int error1 = VRInput.VRInput_GetActionHandle("/actions/main/in/OpenInventory", longBuffer);
openInventoryHandle = longBuffer.get(0);

int error2 = VRInput.VRInput_GetActionSetHandle("/actions/main", longBuffer);
actionSetHandle = longBuffer.get(0);

int error3 = VRInput.VRInput_GetActionHandle("/actions/main/in/test2", longBuffer);
test2Handle = longBuffer.get(0);

VRInput.VRInput_GetActionHandle("/actions/main/in/scroll", longBuffer);
scrollHandle = longBuffer.get(0);

System.out.println("Handle: " + openInventoryHandle);

activeActionSets = VRActiveActionSet.create(1);
activeActionSets.ulActionSet(actionSetHandle);
activeActionSets.ulRestrictedToDevice(VR.k_ulInvalidInputValueHandle); // both hands

clickTriggerActionData = InputDigitalActionData.create();

inputAnalogActionData = InputAnalogActionData.create();

Main app = new Main(vrAppState);
app.setLostFocusBehavior(LostFocusBehavior.Disabled);
app.setSettings(settings);
app.setShowSettings(false);
app.start();
}
}


The use of an action manifest disables legacy inputs (so existing jme calls to VRInputAPI#isButtonDown stop working) which is expected, however the pose (where the hand is, what its pointing at) continue to work, which is pleasant.

Then later within every simpleUpdate the action handles are used to get the current user input

List<Geometry> handGeometries = new ArrayList<>();

@Override
public void simpleUpdate(float tpf) {

VRAppState vrAppState = getStateManager().getState(VRAppState.class);
int numberOfControllers = vrAppState.getVRinput().getTrackedControllerCount(); //almost certainly 2, one for each hand

//build as many geometries as hands, as markers for the demo (Will only tigger on first loop or if number of controllers changes)
while(handGeometries.size()<numberOfControllers){
Box b = new Box(0.1f, 0.1f, 0.1f);
Geometry handMarker = new Geometry("hand", b);
Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
mat.setColor("Color", ColorRGBA.Red);
handMarker.setMaterial(mat);
rootNode.attachChild(handMarker);
}

VRInputAPI vrInput = vrAppState.getVRinput();

for(int i=0;i<numberOfControllers;i++){
if (vrInput.isInputDeviceTracking(i)){ //might not be active currently, avoid NPE if that's the case

Vector3f position = vrInput.getFinalObserverPosition(i);
Quaternion rotation = vrInput.getFinalObserverRotation(i);

Geometry geometry = handGeometries.get(i);
geometry.setLocalTranslation(position);
geometry.setLocalRotation(rotation);
}
}

int error3 = VRInput.VRInput_UpdateActionState(activeActionSets,  VRActiveActionSet.SIZEOF);
int error4 = VRInput.VRInput_GetDigitalActionData(openInventoryHandle, clickTriggerActionData, VR.k_ulInvalidInputValueHandle);
if (clickTriggerActionData.bState()){
System.out.println("openInventory");
}

int error5 = VRInput.VRInput_GetDigitalActionData(test2Handle, clickTriggerActionData, VR.k_ulInvalidInputValueHandle);
if (clickTriggerActionData.bState()){
System.out.println("test2");
}

VRInput.VRInput_GetAnalogActionData(scrollHandle, inputAnalogActionData, VR.k_ulInvalidInputValueHandle);

System.out.println("Joystick control" +inputAnalogActionData.x()+"," + inputAnalogActionData.y());

}


# What I plan to do next

I plan to try to better integrate this with jmonkey (I.e. avoid all the direct calls to lwjgl) with a new VRInputAPI (deprecating the old one) and putting a PR in for that (as long as no one thinks I’m on totally the wrong track). I’ll also aim to add to the wiki for whatever ends up getting created (what should I do regarding documenting something that won’t be in the current version of jmonkey but would be in a future one?)

4 Likes

I’ve put in a PR for actions based openVR #1734 actions based vr input by richardTingle · Pull Request #1735 · jMonkeyEngine/jmonkeyengine · GitHub

2 Likes

Some links that may be of interest from my past work with jme vr.

Perhaps this can be useful.

1 Like

That does look very useful. And may be an important step on fixing the “why does Steam think my app is called ‘java’ problem”

We need someone (other than @richtea) who understands VR to review this PR. Otherwise it’ll probably languish.

I might take a look at it if I find the time to do so.

3 Likes