Augmented Reality + jMonkey

Troubleshooting android
1

Hello,



I’m trying to implement an Augmented Reality App on Android using jMonkey as the game engine. I plan to only use this for Android and realize that jMonkey was not meant to be limited to one platform but I’m OK with this.



So far, I’m able to start a camera preview but am lost on how to overlay the jMonkey scene graph on top. Currently it does not display a 3D box on top of the camera preview nor does it appear to be underneath (I tried reducing the size of the camera preview). I feel that I should add a view on top of mPreview.



Here’s MainActivity which contains the code for the camera preview (hardware camera). Go easy as I am new to Java and Android and so I’m still learning the fundamentals while tackling advance topics.



[java]package com.mycompany.mygame;



import android.content.Context;

import android.content.pm.ActivityInfo;

import android.hardware.Camera;

import android.os.Bundle;

import android.util.Log;

import android.view.SurfaceHolder;

import android.view.SurfaceView;

import android.widget.FrameLayout;

import com.jme3.app.AndroidHarness;

import com.jme3.system.android.AndroidConfigChooser.ConfigType;

import com.mycompany.mygame.R.id;

import java.io.IOException;



public class MainActivity extends AndroidHarness {



/*

  • Note that you can ignore the errors displayed in this file,
  • the android project will build regardless.
  • Install the ‘Android’ plugin under Tools->Plugins->Available Plugins
  • to get error checks and code completion for the Android project files.

    */

    private static final String TAG = "CameraActivity";

    // These are member fields/variables that are directly accessible only

    // within this class because of the "private" modifier. Public methods could

    // be used to indirectly access the field values.

    private Camera mCamera;

    private ARCameraActivity mPreview;



    public MainActivity() {

    // Set the application class to run

    appClass = "mygame.Main";

    // Try ConfigType.FASTEST; or ConfigType.LEGACY if you have problems

    eglConfigType = ConfigType.BEST;

    // Exit Dialog title & message

    exitDialogTitle = "Exit?";

    exitDialogMessage = "Press Yes";

    // Enable verbose logging

    eglConfigVerboseLogging = false;

    // Choose screen orientation

    screenOrientation = ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE;

    // Invert the MouseEvents X (default = true)

    mouseEventsInvertX = true;

    // Invert the MouseEvents Y (default = true)

    mouseEventsInvertY = true;

    }



    @Override

    public void onCreate(Bundle savedInstanceState) {

    super.onCreate(savedInstanceState);

    setContentView(R.layout.main);



    // Create our Preview view and set it as the content of our activity.

    mPreview = new ARCameraActivity(this, mCamera);

    Log.d(TAG, "mPreview" + mPreview);

    // "this" is in reference to the current object mPreview.

    // mPreview is an object of the ARCameraActivity class which

    // uses a constructor in ARCameraActivity which contains SurfaceView

    // callback for passing

    // the surface to be displayed here.

    FrameLayout preview = (FrameLayout) findViewById(id.camerapreview);

    preview.addView(mPreview);



    }



    public class ARCameraActivity extends SurfaceView implements

    SurfaceHolder.Callback {

    // These are member variables / fields that are directly accessible only

    // within this class because of the "private" modifier. Public methods could

    // be used to indirectly access the field values.



    private SurfaceHolder mHolder;

    //

    private static final String TAG = "ARCameraActivity";

    /**
  • Indicates whether the camera is still started or not.

    /

    private boolean started;

    private Camera mCamera;



    // ARCameraActivity constructor for CameraActivity. Everything done in this

    // constructor is passed back to mPreview in CameraActivity.

    public ARCameraActivity(Context context, Camera camera) {

    super(context);

    // TODO Auto-generated constructor stub

    // “started, mCamera and mHolder” are objects

    started = false;

    mCamera = camera; // mCamera was passed by CameraActivity



    // Install a SurfaceHolder.Callback so we get notified when the

    // underlying surface is created and destroyed.

    mHolder = getHolder();

    mHolder.addCallback(this);// This line references an object’s (mHolder

    // which is actually a SurfaceHolder object)

    // method (addCallback) outside of the

    // object’s class. “this” is passed as an

    // argument to addCallback method with

    // refers to mHolder. The “this” is in

    // reference to the current object.

    // deprecated setting, but required on Android versions prior to 3.0

    mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);

    }



    // “public” modifier, “void” return data type, “surfaceCreated” method name,

    // “(parameter list)”.

    public void surfaceCreated(SurfaceHolder holder) {

    // TODO Auto-generated method stub

    // The Surface has been created, now tell the camera where to draw the

    // preview.

    // Create an instance of Camera

    // Local variable mCamera that contains the returned result (Camera)

    // from getCameraInstance().

    mCamera = getCameraInstance();

    /    * A safe way to get an instance of the Camera object. */

    try {

    if (mCamera != null) {

    mCamera.setPreviewDisplay(holder);

    }

    } catch (IOException e) {

    Log.d(TAG,

    "Error setting camera preview in surfaceCreated: "
  • e.getMessage());



    }

    }



    public void surfaceDestroyed(SurfaceHolder holder) {

    // TODO Auto-generated method stub

    // empty. Take care of releasing the Camera preview in your activity.



    try {

    stop();

    } catch (Exception e) {

    Log.e("ARCameraActivity surfaceDestroyed", e.getMessage());

    }

    }



    public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {

    // TODO Auto-generated method stub

    // New - test the camera parameters section

    // Camera.Parameters param = mCamera.getParameters();//sets param to be

    // equal to camera parametors

    // param.setPreviewSize(w, h);//sets width and height to that of what is

    // passed back to it when callback calls it

    // param.setFlashMode(Parameters.FLASH_MODE_TORCH);

    // mCamera.setParameters(param);//sets the camera parameters to param



    Log.d(TAG, "*** surfaceChanged >>>>> ***");

    Log.d(TAG, "format=" + format + ", width=" + w + ", height=" + h
  • ", holder=" + holder);

    /*
  • 08/14/2012 - Removed for testing Camera.Parameters params =
  • mCamera.getParameters(); Camera.Size size =
  • getBestPreviewSize(params,w,h);

    *
  • if (size != null) params.setPreviewSize(size.width, size.height);
  • mCamera.startPreview();

    /



    // If your preview can change or rotate, take care of those events here.

    // Make sure to stop the preview before resizing or reformatting it.

    // mCamera.stopPreview();

    if (mHolder.getSurface() == null) {

    // preview surface does not exist

    Log.d(TAG, "mHolder.getSurface()=" + mHolder);

    return;// used to branch out of a control flow block and exit the

    // method

    }



    // stop preview before making changes

    try {

    mCamera.stopPreview();

    } catch (Exception e) {

    // ignore: tried to stop a non-existent preview

    }



    // set preview size and make any resize, rotate or

    // reformatting changes here

    // Now that the size is known, set up the camera parameters and begin

    // the preview.

    /
  • Camera.Parameters parameters = mCamera.getParameters();
  • parameters.setPreviewSize(mPreviewSize.width, mPreviewSize.height);
  • requestLayout(); mCamera.setParameters(parameters);

    */

    Log.d(TAG, "Before setPreviewDisplay(mHolder)");

    // start preview with new settings

    try {

    mCamera.setPreviewDisplay(mHolder);// mCamera=null

    Log.d(TAG, "mCamera=" + mCamera);

    Log.d(TAG, "mHolder=" + mHolder);

    // start();

    } catch (Exception e) {



    Log.d(TAG,

    "Error starting camera preview in surfaceChanged: "
  • e.getMessage());

    }

    start();

    }



    /*
  • 08/14/2012 - Removed for testing; part of above section /* Get supported
  • preview size and select best one. (non-Javadoc)

    *
  • @see android.view.SurfaceHolder.Callback#surfaceDestroyed(android.view.
  • SurfaceHolder)

    *
  • public Camera.Size getBestPreviewSize(Camera.Parameters parameters, int
  • w, int h) { Camera.Size result = null;

    *
  • for (Camera.Size size : parameters.getSupportedPreviewSizes()) { if
  • (size.width <= w && size.height <= h) { if (null == result) result =
  • size; else { int resultDelta = w - result.width + h - result.height; int
  • newDelta = w - size.width + h - size.height;

    *
  • if (newDelta < resultDelta) result = size; } } } return result; }

    /

    private Camera getCameraInstance() {

    // TODO Auto-generated method stub



    Camera c = null;

    try {

    c = Camera.open(); // attempt to get a Camera instance

    Log.d(TAG, "    ** Camera instance >>>>>>> *");

    } catch (Exception e) {

    // Camera is not available (in use or does not exist)

    Log.e(TAG, "Error starting camera preview_1: " + e.getMessage());

    }

    return c; // returns null if camera is unavailable

    }



    /
  • Starts the camera.

    /

    public void start() {

    if (!started) {

    Log.d(TAG, "    ** ARCameraActivity start >>>>> *");

    mCamera.startPreview();

    started = true;

    }

    }



    /
  • Stops the camera.

    /

    public void stop() {

    if (mCamera != null) {

    Log.d(TAG, "    ** ARCameraActivity stop >>>>> ");

    // mCamera.stopPreview(); // Method being called after release()

    // even though it is before



    mCamera.release();

    started = false;

    mCamera = null;



    }

    }

    }



    @Override

    public void onResume() {

    super.onResume(); // Always call the superclass method first

    Log.d(TAG, "     CameraActivity onResume >>>>> ");

    // mCamera.setPreviewCallback(null);

    // Get the Camera instance as the activity achieves full user focus



    }



    @Override

    public void onPause() {

    super.onPause(); // Always call the superclass method first



    // Release the Camera because we don’t need it when paused

    // and other activities might need to use it.

    if (mCamera != null) {

    Log.d(TAG, "     CameraActivity onPause >>>>> *");

    mCamera.release();

    mCamera = null;



    }

    }

    }

    [/java]



    Here’s Main.java.

    [java]package mygame;



    import com.jme3.app.SimpleApplication;

    import com.jme3.material.Material;

    import com.jme3.math.ColorRGBA;

    import com.jme3.math.Vector3f;

    import com.jme3.renderer.RenderManager;

    import com.jme3.scene.Geometry;

    import com.jme3.scene.shape.Box;



    /
  • test
  • @author normenhansen

    */

    public class Main extends SimpleApplication {



    public static void main(String[] args) {

    Main app = new Main();

    app.start();

    }



    @Override

    public void simpleInitApp() {

    Box b = new Box(Vector3f.ZERO, 1, 1, 1);

    Geometry geom = new Geometry(“Box”, b);



    Material mat = new Material(assetManager, “Common/MatDefs/Misc/Unshaded.j3md”);

    mat.setColor(“Color”, ColorRGBA.Blue);

    geom.setMaterial(mat);



    //geom.setLocalTranslation(0,0,1);// New addition to be tested; to move text to the foreground.



    rootNode.attachChild(geom);

    }



    @Override

    public void simpleUpdate(float tpf) {

    //TODO: add update code

    }



    @Override

    public void simpleRender(RenderManager rm) {

    //TODO: add render code

    }

    }

    [/java]



    Here’s Main.xml.

    [java]<?xml version=“1.0” encoding=“utf-8”?>

    <LinearLayout xmlns:android=“http://schemas.android.com/apk/res/android”

    android:orientation=“horizontal”

    android:layout_width=“fill_parent”

    android:layout_height=“fill_parent”

    >

    <FrameLayout

    android:id="@+id/camerapreview"

    android:layout_width=“0dp”

    android:layout_height=“fill_parent”

    android:layout_weight=“1”

    android:keepScreenOn=“true”

    />

    </LinearLayout>[/java]

    I’ve played with Skyebooks’ code https://github.com/skyebook/JME3-Android-Camera which takes the camera image as a stream and places it as a texture on a 3D box but I believe converting the preview to a texture is consuming on the processor.



    Your help is appreciated.
1 Like
2

Nice!

About the sensors, @iwgeric recently did a change that allows you to map the Gyroscope sensor into a JME joystic.

see http://hub.jmonkeyengine.org/groups/android/forum/topic/creating-engine-support-for-android-sensor-input/?topic_page=3&num=15#post-189378

3

You should not do your layout in the on create, you have to extends the layoutDisplay mehtod.

At this point you can do whatever layout you want, and the JME view is initialized and in an attribute called “view”.

Also if you want that your JME view overlays and another view you have to se the configuration to pick the translucent setting.

just put this in your activity

[java]

eglConfigType = ConfigType.BEST_TRANSLUCENT;

[/java]

4

Thanks nehon.



For future reference for others regarding:

[java]eglConfigType = ConfigType.BEST_TRANSLUCENT;[/java]

Refer to http://hub.jmonkeyengine.org/javadoc/com/jme3/system/android/AndroidConfigChooser.ConfigType.html


You should not do your layout in the on create, you have to extends the layoutDisplay mehtod.


What class would I extend to access layoutDisplay method?

At this point you can do whatever layout you want, and the JME view is initialized and in an attribute called “view”.

Are you refering to http://hub.jmonkeyengine.org/javadoc/com/jme3/renderer/ViewPort.html? Or NiftyGui?

Thanks
5
@platz said:
What class would I extend to access layoutDisplay method?

The AndroidHarness. Like you did.

@platz said:
Are you refering to http://hub.jmonkeyengine.org/javadoc/com/jme3/renderer/ViewPort.html? Or NiftyGui?

No, I mean you can do any android layout, Framelayout, RelativeLayout, and so on. You can build a complete android UI and just integrate the JME view into it.
What you have to do here in your case is just take the code you have in the onCreate, put it in the layoutDisplay method and just do in the end
[java]
preview.addView(view);
[/java]
It should work.
6

Thanks for clarifying that nehon, it works!



Here’s the updated changes to MainActivity as per your advise for future users:

[java]package com.mycompany.mygame;



import android.content.Context;

import android.content.pm.ActivityInfo;

import android.hardware.Camera;

import android.os.Bundle;

import android.util.Log;

import android.view.SurfaceHolder;

import android.view.SurfaceView;

import android.widget.FrameLayout;

import com.jme3.app.AndroidHarness;

import com.jme3.system.android.AndroidConfigChooser.ConfigType;

import com.mycompany.mygame.R.id;

import java.io.IOException;



public class MainActivity extends AndroidHarness {



/*

  • Note that you can ignore the errors displayed in this file,
  • the android project will build regardless.
  • Install the ‘Android’ plugin under Tools->Plugins->Available Plugins
  • to get error checks and code completion for the Android project files.

    */

    private static final String TAG = "CameraActivity";

    // These are member fields/variables that are directly accessible only

    // within this class because of the "private" modifier. Public methods could

    // be used to indirectly access the field values.

    private Camera mCamera;

    private ARCameraActivity mPreview;



    public MainActivity() {

    // Set the application class to run

    appClass = "mygame.Main";

    // Try ConfigType.FASTEST; or ConfigType.LEGACY if you have problems

    eglConfigType = ConfigType.BEST_TRANSLUCENT;

    // Exit Dialog title & message

    exitDialogTitle = "Exit?";

    exitDialogMessage = "Press Yes";

    // Enable verbose logging

    eglConfigVerboseLogging = false;

    // Choose screen orientation

    screenOrientation = ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE;

    // Invert the MouseEvents X (default = true)

    mouseEventsInvertX = true;

    // Invert the MouseEvents Y (default = true)

    mouseEventsInvertY = true;

    }



    @Override

    public void onCreate(Bundle savedInstanceState) {

    super.onCreate(savedInstanceState);



    }



    @Override

    public void layoutDisplay() {

    setContentView(R.layout.main);



    // Create our Preview view and set it as the content of our activity.

    mPreview = new ARCameraActivity(this, mCamera);

    Log.d(TAG, "mPreview" + mPreview);

    // "this" is in reference to the current object mPreview.

    // mPreview is an object of the ARCameraActivity class which

    // uses a constructor in ARCameraActivity which contains SurfaceView

    // callback for passing

    // the surface to be displayed here.

    FrameLayout preview = (FrameLayout) findViewById(id.camerapreview);

    preview.addView(mPreview);

    preview.addView(view); //Add JME view which is called in Main.java

    }



    public class ARCameraActivity extends SurfaceView implements

    SurfaceHolder.Callback {

    // These are member variables / fields that are directly accessible only

    // within this class because of the "private" modifier. Public methods could

    // be used to indirectly access the field values.



    private SurfaceHolder mHolder;

    //

    private static final String TAG = "ARCameraActivity";

    /**
  • Indicates whether the camera is still started or not.

    /

    private boolean started;

    private Camera mCamera;



    // ARCameraActivity constructor for CameraActivity. Everything done in this

    // constructor is passed back to mPreview in CameraActivity.

    public ARCameraActivity(Context context, Camera camera) {

    super(context);

    // TODO Auto-generated constructor stub

    // “started, mCamera and mHolder” are objects

    started = false;

    mCamera = camera; // mCamera was passed by CameraActivity



    // Install a SurfaceHolder.Callback so we get notified when the

    // underlying surface is created and destroyed.

    mHolder = getHolder();

    mHolder.addCallback(this);// This line references an object’s (mHolder

    // which is actually a SurfaceHolder object)

    // method (addCallback) outside of the

    // object’s class. “this” is passed as an

    // argument to addCallback method with

    // refers to mHolder. The “this” is in

    // reference to the current object.

    // deprecated setting, but required on Android versions prior to 3.0

    mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);

    }



    // “public” modifier, “void” return data type, “surfaceCreated” method name,

    // “(parameter list)”.

    public void surfaceCreated(SurfaceHolder holder) {

    // TODO Auto-generated method stub

    // The Surface has been created, now tell the camera where to draw the

    // preview.

    // Create an instance of Camera

    // Local variable mCamera that contains the returned result (Camera)

    // from getCameraInstance().

    mCamera = getCameraInstance();

    /    * A safe way to get an instance of the Camera object. */

    try {

    if (mCamera != null) {

    mCamera.setPreviewDisplay(holder);

    }

    } catch (IOException e) {

    Log.d(TAG,

    "Error setting camera preview in surfaceCreated: "
  • e.getMessage());



    }

    }



    public void surfaceDestroyed(SurfaceHolder holder) {

    // TODO Auto-generated method stub

    // empty. Take care of releasing the Camera preview in your activity.



    try {

    stop();

    } catch (Exception e) {

    Log.e("ARCameraActivity surfaceDestroyed", e.getMessage());

    }

    }



    public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {

    // TODO Auto-generated method stub

    // New - test the camera parameters section

    // Camera.Parameters param = mCamera.getParameters();//sets param to be

    // equal to camera parametors

    // param.setPreviewSize(w, h);//sets width and height to that of what is

    // passed back to it when callback calls it

    // param.setFlashMode(Parameters.FLASH_MODE_TORCH);

    // mCamera.setParameters(param);//sets the camera parameters to param



    Log.d(TAG, "*** surfaceChanged >>>>> ***");

    Log.d(TAG, "format=" + format + ", width=" + w + ", height=" + h
  • ", holder=" + holder);

    /*
  • 08/14/2012 - Removed for testing Camera.Parameters params =
  • mCamera.getParameters(); Camera.Size size =
  • getBestPreviewSize(params,w,h);

    *
  • if (size != null) params.setPreviewSize(size.width, size.height);
  • mCamera.startPreview();

    /



    // If your preview can change or rotate, take care of those events here.

    // Make sure to stop the preview before resizing or reformatting it.

    // mCamera.stopPreview();

    if (mHolder.getSurface() == null) {

    // preview surface does not exist

    Log.d(TAG, "mHolder.getSurface()=" + mHolder);

    return;// used to branch out of a control flow block and exit the

    // method

    }



    // stop preview before making changes

    try {

    mCamera.stopPreview();

    } catch (Exception e) {

    // ignore: tried to stop a non-existent preview

    }



    // set preview size and make any resize, rotate or

    // reformatting changes here

    // Now that the size is known, set up the camera parameters and begin

    // the preview.

    /
  • Camera.Parameters parameters = mCamera.getParameters();
  • parameters.setPreviewSize(mPreviewSize.width, mPreviewSize.height);
  • requestLayout(); mCamera.setParameters(parameters);

    */

    Log.d(TAG, "Before setPreviewDisplay(mHolder)");

    // start preview with new settings

    try {

    mCamera.setPreviewDisplay(mHolder);// mCamera=null

    Log.d(TAG, "mCamera=" + mCamera);

    Log.d(TAG, "mHolder=" + mHolder);

    // start();

    } catch (Exception e) {



    Log.d(TAG,

    "Error starting camera preview in surfaceChanged: "
  • e.getMessage());

    }

    start();

    }



    /*
  • 08/14/2012 - Removed for testing; part of above section /* Get supported
  • preview size and select best one. (non-Javadoc)

    *
  • @see android.view.SurfaceHolder.Callback#surfaceDestroyed(android.view.
  • SurfaceHolder)

    *
  • public Camera.Size getBestPreviewSize(Camera.Parameters parameters, int
  • w, int h) { Camera.Size result = null;

    *
  • for (Camera.Size size : parameters.getSupportedPreviewSizes()) { if
  • (size.width <= w && size.height <= h) { if (null == result) result =
  • size; else { int resultDelta = w - result.width + h - result.height; int
  • newDelta = w - size.width + h - size.height;

    *
  • if (newDelta < resultDelta) result = size; } } } return result; }

    /

    private Camera getCameraInstance() {

    // TODO Auto-generated method stub



    Camera c = null;

    try {

    c = Camera.open(); // attempt to get a Camera instance

    Log.d(TAG, "    ** Camera instance >>>>>>> *");

    } catch (Exception e) {

    // Camera is not available (in use or does not exist)

    Log.e(TAG, "Error starting camera preview_1: " + e.getMessage());

    }

    return c; // returns null if camera is unavailable

    }



    /
  • Starts the camera.

    /

    public void start() {

    if (!started) {

    Log.d(TAG, "    ** ARCameraActivity start >>>>> *");

    mCamera.startPreview();

    started = true;

    }

    }



    /
  • Stops the camera.

    /

    public void stop() {

    if (mCamera != null) {

    Log.d(TAG, "    ** ARCameraActivity stop >>>>> ");

    // mCamera.stopPreview(); // Method being called after release()

    // even though it is before



    mCamera.release();

    started = false;

    mCamera = null;



    }

    }

    }



    @Override

    public void onResume() {

    super.onResume(); // Always call the superclass method first

    Log.d(TAG, "     CameraActivity onResume >>>>> ");

    // mCamera.setPreviewCallback(null);

    // Get the Camera instance as the activity achieves full user focus



    }



    @Override

    public void onPause() {

    super.onPause(); // Always call the superclass method first



    // Release the Camera because we don’t need it when paused

    // and other activities might need to use it.

    if (mCamera != null) {

    Log.d(TAG, "     CameraActivity onPause >>>>> ***");

    mCamera.release();

    mCamera = null;



    }

    [/java]



    Note that to have the following option available:

    [java]eglConfigType = ConfigType.BEST_TRANSLUCENT;[/java]

    I had to update the .jar files to the latest nightly build thru:

    Tools->Plugins->Settings Tab->Select “jMonkeyEngine SDK Nightly”

    Now change Automatically Check for Updates “Check Interval:” to Every Startup, then restart jME3 and accept to download and install updates. Afterwards, you can change the “Check Interval:” back to your previous setting if desired.



    Now on to Android orientation sensors and mapping it to the scene camera. I believe this will require studying the following topics:

    jME Maths Concepts

    jME Camera

    Android Sensors Overview → http://developer.android.com/guide/topics/sensors/sensors_overview.html



    But that’s all for another thread when I can’t take it any further on my own.



    Thanks again.
1 Like
7

@platz thanks for your code.is it possible to put it for download?

8

Good day and happy new year community.

Today I am very interested in AR and want to know if anyone out there got this to work with jME3.0?

Are there anyone who got some of the exciting frameworks like, http://catchoom.com/ or http://www.wikitude.com/ or http://artoolkit.org/ to work with jME?

Also, I want to be able to do image recognition with an android devices camera.

Thanks in advance and please, any tips or direction I may take would be much appreciated.

9

For image recognition, you can take a look at opencv (native + java binding) or BoofCV (a pure java implementation), available for android, iirc there are a demo app on playstore.

2 Likes
10

Thanks, I will have a look.