Augmented Reality jme on your Pc (or whateva, webcam background anyway)


Hi, quite pleased with coming up with this way of getting your web cam as a background in jme on the Pc. Just had to brag 'bout it as it opens the door to some realy exciting apps. Please feel free to suggest improvements. (Probably need some being a noobie)


 * In jMonkeyEngine choose File -> New Project and choose JME BasicGame.
 * In the in simpleInitApp insert these lines; 
 * PcWebCamAppState pcWebCamAppState = new PcWebCamAppState(settings);
 * stateManager.attach(pcWebCamAppState);
 * Left click Libraries and Add JAR/Folder: javcpp.jar, javacv.jar, opencv-2.4.8-windows-x86_64.jar
 * from javacv-cppjars and javacv-bin  (opencv javawrappers) that you will find on the internett (64-bit pc)
 * Add this file as ( to your package
 * Voila you have a live video background from your PcWebCam in your project
package mygame;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
import com.googlecode.javacv.cpp.opencv_highgui;
import com.jme3.material.Material;
import com.jme3.renderer.ViewPort;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Quad;
import com.jme3.texture.Image;
import com.jme3.texture.Texture2D;
import com.jme3.texture.plugins.AWTLoader;
import java.awt.image.BufferedImage;
import com.jme3.renderer.Camera;
import com.jme3.asset.AssetManager;
import com.jme3.renderer.RenderManager;
import com.jme3.renderer.queue.RenderQueue;
import com.jme3.system.AppSettings;

 * @author Harald Heide Gundersen
public class PcWebCamAppState extends AbstractAppState {
    private SimpleApplication app;
    private Camera cam;
    private RenderManager renderManager;
    private AssetManager assetManager;
    private AppSettings appSettings;
//Webcam Start
    private Geometry mVideoBGGeom;// The geometry which will represent the video background
    private Material mvideoBGMat;// The material which will be applied to the video background geometry.
    private Texture2D mCameraTexture;// The texture displaying the Android camera preview frames.
//Webcam End
    public Image webcamJmeImage;
    public boolean mNewCameraFrameAvailable; 
    private opencv_highgui.CvCapture capture;
    public PcWebCamAppState(AppSettings appSettings){
        this.appSettings = appSettings;
    public void initialize(AppStateManager appStateManager, Application app) {
        super.initialize(appStateManager, app); = (SimpleApplication) app; =;
        this.renderManager =;
        this.assetManager =;
        Thread myImageLoop = new Thread(new ImageLoop("ImageLoop"));
        initVideoBackground(appSettings.getWidth(),appSettings.getHeight());  //'settings.getWidth(),settings.getHeight()'

    public void update(float tpf) {
    	if(mNewCameraFrameAvailable) {
            mvideoBGMat.setTexture("ColorMap", mCameraTexture);
            mNewCameraFrameAvailable = false;
        // we have to update the video background node before the root node gets updated by the super class
    public void cleanup() {
    public class ImageLoop implements Runnable{
        public ImageLoop(String Name){} 
        public void run(){
            BufferedImage webcamBufferedImage;
            // choose your camera here
            capture = opencv_highgui.cvCreateCameraCapture(0); //0=Front, 1=Back camera
            opencv_highgui.cvSetCaptureProperty(capture, opencv_highgui.CV_CAP_PROP_FRAME_HEIGHT,720);
            opencv_highgui.cvSetCaptureProperty(capture, opencv_highgui.CV_CAP_PROP_FRAME_WIDTH,1280);
            IplImage webcamIplImage = opencv_highgui.cvQueryFrame(capture);
            while(opencv_highgui.cvQueryFrame(capture) != null)
                webcamBufferedImage = webcamIplImage.getBufferedImage();
                webcamJmeImage = new AWTLoader().load(webcamBufferedImage, false);
                mNewCameraFrameAvailable = true;
    public void initVideoBackground(int screenWidth, int screenHeight) {
        Quad videoBGQuad = new Quad(1, 1, true);// Create a Quad shape.
        mVideoBGGeom = new Geometry("quad", videoBGQuad);// Create a Geometry with the Quad shape
        float newWidth = 1.f * screenWidth / screenHeight;
        mVideoBGGeom.setLocalTranslation(-0.5f * newWidth, -0.5f, 0.f);// Center the Geometry in the middle of the screen.
        mVideoBGGeom.setLocalScale(1.f * newWidth, 1.f, 1);// Scale (stretch) the width of the Geometry to cover the whole screen width.
        mvideoBGMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");// Apply a unshaded material which we will use for texturing.
        mCameraTexture = new Texture2D();// Create a new texture which will hold the Android camera preview frame pixels.
        Camera videoBGCam = cam.clone();
        ViewPort videoBGVP = renderManager.createMainView("VideoBGView", videoBGCam);


Fixed that codeblock for ya, check the edit :wink:

Thanks for sharing! Would love to see a video demonstration.

[quote=“HaraldHeide, post:1, topic:31232”]
Please feel free to suggest improvements.
[/quote]It’s hard to make feedback based on that. Unless they’re using it themselves, no one’s gonna be reading through your code line-by-line. If there’s a particular part of your code that you can point to and say “see what I did here? is that a good way of doing it?” then that’s much easier to address in a short comment.