Help with GUI mouse picking

I’m creating a simple title screen where you select options with the mouse using the GUI. However I can’t find any examples on GUI mouse picking that aren’t based on Nifty. My interface is super simple so I don’t need to use Nifty.

I basically need to select the image or text the GUI puts on screen.

So where is your problem then?
Use the Input-Manager to get the (x, y) coordinates of the click and then you simply need to know where your buttons are.

Like if you have three buttons you could use Rectangle.contains() or how that special method is called, you could also use some own calculations (like x > button1.getX() && x <= button1.getWidth()) but I really suggest you to use nifty/t0neg0d/lemur/javaFX.

It’s so easy to use for simple GUIs and it’s easy to include other things you might need

I’m not using the java gui I’m using the 2d functions of jme. I want to detect collision of my mouse with different images/text on the guinode.

I have a bitmaptext and I want to know when the mouse is over it. The collidable function in the bitmaptext still refers to 3d space.

Another solution (maybe less efficient but more scene generic) to detect collisions between the mouse and any scene spatial you have all documentation here.

I tried this

Vector2f click2d = inputManager.getCursorPosition();
        Vector3f click3d = cam.getWorldCoordinates(new Vector2f(click2d.x, click2d.y), 0f).clone();
        Vector3f dir = cam.getWorldCoordinates(new Vector2f(click2d.x, click2d.y), 1f).subtractLocal(click3d).normalizeLocal();
        // Aim the ray from the clicked spot forwards.
        Ray ray = new Ray(click3d, dir);
        
    CollisionResults results = new CollisionResults();
    
    guiNode.collideWith(ray, results);

But the collision gets results when the mouse isn’t even close to the image.

This is the full test code

setDisplayStatView(false);
    inputManager.setCursorVisible(true);

    guiFont = assetManager.loadFont("Interface/Fonts/Default.fnt");
    BitmapText tt = new BitmapText(guiFont, false);
    tt.setSize(guiFont.getCharSet().getRenderedSize() * 2);
    tt.setText("test");
    tt.setLocalTranslation( 50,50,0);
    guiNode.attachChild(tt);
    
    Vector2f click2d = inputManager.getCursorPosition();
    Vector3f click3d = cam.getWorldCoordinates(new Vector2f(click2d.x, click2d.y), 0f).clone();
    Vector3f dir = cam.getWorldCoordinates(new Vector2f(click2d.x, click2d.y), 1f).subtractLocal(click3d).normalizeLocal();
    // Aim the ray from the clicked spot forwards.
    Ray ray = new Ray(click3d, dir);
        
    CollisionResults results = new CollisionResults();
    
    guiNode.collideWith(ray, results);
    
    if (results.size() > 0) {
          // The closest collision point is what was truly hit:
          CollisionResult closest = results.getClosestCollision();
          // Let's interact - we mark the hit with a red dot.
      tt.setText("hit"); // crosshairs
        } else {
          // No hits? Then remove the red mark.
      tt.setText("miss"); // crosshairs
        }

For the guiNode you don’t need to make a ray in world coordinates… because then they’d be in the 3D scene and not in the 2D scene. You just need to create a ray with the screen coordinates.

Or you can use Lemur’s built in picking support that works with any JME spatial. Search for the Lemur Gems on picking. I haven’t force restarted firefox in the last three hours or so, so it’s about to take a dump and I can’t look it up for you. I’ll follow up later if I can.

The ray only accepts 3d coordinates unless there is something else I’m missing.

What paul means is your ray would be from (x, y, 0) in direction (0, 0, 1) (which means from the current x,y into the room (with a collision at z = 0).

It would be better to use GUI toolsets though, because they even support like onMouseEnter() or animate the text, which is also a nice effect of visual feedback.

Other than that my solution still applies (Spatials have .getLocalTranslation().getX()/.y and possibly a size.

With Nifty you could simply create a small xml file, no real java code needed, just the onClick method. (JMP even has a visual builder for those xml files included iirc)

Using Lemur to easily do picking on any JME Spatial, whether 2D or 3D:

See the full Lemur site/repo here:

Inculding the “Getting Started” page if you want to dive in deeper:

You don’t have to use the UI portions of Lemur to get the scene picking… just call one initialize method and then add listeners to the spatials you want. Though you can also easily create buttons with images if that’s easier for you… and then place them wherever you want.

2 Likes