Using swing - awt.mouse to camera / world coords

>> Modified since first post

Have just found post

Seems that you will make a AWTMouse in .10, so i hope this will satisfy the issues below. In particular will the implemetation be usable on AWTCanvases and the JMECanvasImplementor.

Was pondering on toggelling between mouse implementations on Panel enter/exit events, maintaining a referernce to each the JMEMouse and AWTMouse


Ive extended the JMECanvasImplementor and put the terrain code in.

Im looking to let the user add a tree by clicking on the TerainPage.

Im using the awt mouse to obtain the x and y of the screen coords ( uses a top left origin axis ), i need to use the display to get the world coordinates, but i am so new to this stuff i would appreciate any pointers.

as the view is releative to the camera - should i be translating the mouse coordinates to the cam node first, then getting world coordinates ??.

public void mouseClicked(java.awt.event.MouseEvent e) {

System.out.println("Clck " + e.getX() + " " + e.getY());

Vector3f wcc = display.getWorldCoordinates(new Vector2f(e.getX(),0), e.getY());

System.out.println("world " + wcc);


Have a look at RenControlEditor (in CVS or nightly build) - does that already satisfy your needs?

Thanks irrisor…

Got the nightly build and ran the demo - its a great third person handler and will be usefull for some other scenarios, however the particular scenrio of clicking on the terrain page to place a tree isnt resolved. Awt mouse coordiantes translated to object space

May need to get my maths books out, im not fully competent in 3d translations as yet. Camera space to world space …

the trouble with firing a ray is that a pixel on the terrain block doenst have its own boundingbox, the block itself has the boundingbox, so i need to translate the specific coordinate - eg the x and z coordiantes of the block ( y is height ).

I dont want to reinvent the wheel and i noticed there is an awtmouse in the nightly build.

Maybe one ultimate question is - will joshua's AWTMouse implemetation have a absolute / relative version ??? - maybe that would suffice

I don't think that a relative mouse would really help you (if I understood what you want to do). The absolute awt mouse is used in RenControlEditor (that's why I pointed you to that one).

To transform the mouse coordinates into a ray you can do something like this:

            mousePos.set( mouse.getLocalTranslation().x, mouse.getLocalTranslation().y );

            DisplaySystem.getDisplaySystem().getWorldCoordinates( mousePos, 0, pickRay.origin );

            DisplaySystem.getDisplaySystem().getWorldCoordinates( mousePos, 0.3f, pickRay.direction ).subtractLocal( pickRay.origin ).normalizeLocal();

(I use this piece of code with lwjgl mouse, but it should work with awt as well. Btw. don't do this too often - e.g. only 10 times per second - getWorldCoordinates is quite slow at this time)

AWTMouseInput is not a Mouse, but rather an input class.  The Absolute and Relative mice USE the input classes. 

That said, the AWTMouseInput class is fairly customizeable, among other things allowing you to use mouseover and/or drag to send mouse events to the input system.  So you should be able to approximate most control setups in swing/awt.

um, true - thanks for clarifying this - for me the "mouse" is the input class/device. I would consider the other thing a mouse cursor -> maybe we should rename Mouse to MouseCursor as it's a Quad?

shrug  Since one is called Mouse and the other MouseInput, I think the distinction is made already.  :slight_smile:

Okay - i have removed all the clutter from the prior post.

The actual question is

I have a TriMesh pick result from clicking on the TerrainPage/block, i have stored Triangular indices in TriMesh.getTriangle

How do i relate the indices of the picked triangle to the Terrain page so i can call addHeightAtPoint(x,z,newHeight) on the TerrainPage

Am i right in thinking that each triangle's indices are not unique as there may be many terrain blocks

Anyone ??? - how can i convert my mouse click point to the terrain page point ??

There is no such thing as a direct conversion between the two.  You would need to do a Pick cast from the point you click on with your mouse and use that. 

how can i convert my mouse click point to the terrain page point ??

I used a derived class of AbsoluteMouse and TrianglePickResults:

results = new TrianglePickResults();

// Get the position that the mouse is pointing to
Vector2f screenPos = new Vector2f(mouse.getHotSpotPosition().x, mouse.getHotSpotPosition().y);

// Get the world location of that X,Y value - near
Vector3f startPoint = DisplaySystem.getDisplaySystem().getWorldCoordinates(screenPos, 0);

// Get the world location of that X,Y value - far
Vector3f endPoint = DisplaySystem.getDisplaySystem().getWorldCoordinates(screenPos, 1);

Ray ray = new Ray(startPoint, endPoint.subtract(startPoint));

rootNode.findPick(ray, results);

Vector3f loc = new Vector3f();
Vector3f[] vertex = new Vector3f[3];

if (results.getNumber() > 0) {

   TriMesh mesh = (TriMesh) results.getPickData(0).getTargetMesh();

   boolean hit = false;

   for (int j = 0; j < mesh.getTriangleQuantity(); j++) {
      mesh.getTriangle(j, vertex);
      if (hit = ray.intersectWhere(vertex[0].addLocal(mesh.getWorldTranslation()),


If got so far by extending pick results


1> im using TerrainPage not terrain block - so the method hasTriangleCollision(mesh) isnt available - guess i could extend page and add one.

2>I get three mesh indices - eg 220 550 54 - how do i trabnslate these to the height map points so i can call methods like addHeightMapValue on the TerrainPage.

This has been one hell of a learning curve - very useful one too, but i feel like i have hit a wall on this one

PickResults results = new TrianglePickResults() {
      public void processPick() {
         int found;
         if ((found = getNumber()) > 0) {
            for (int j = 0; j < getNumber(); j++) {
               PickData pData = getPickData(j);
               ArrayList tris = pData.getTargetTris();
               TriMesh mesh = (TriMesh) pData.getTargetMesh();
               // Does the pick effect the terrain block
               if(tb.hasTriangleCollision(mesh)) {
                  int[] indices = new int[3];               
                  if(tris.size() > 0) {                    
                     int triIndex = ((Integer) tris.get(0)).intValue();
                     mesh.getTriangle(triIndex, indices);                    
                     if(indices[0] + indices[1] + indices[2] != 0) {  
                        System.out.println("Of " + found + " Clicked on " + triIndex);
                        System.out.println("Mesh = " + indices[0] + " " + indices[1] + " " +indices[2]);
                        float scalex = rootNode.getLocalScale().x;
                        float scalez = rootNode.getLocalScale().y;
                        float transx = rootNode.getLocalTranslation().x;
                        float transz = rootNode.getLocalScale().y;
                        System.out.println("Scale " + scalex + " " + scalez);
                        System.out.println("Trans " + transz + " " + transz);
                        int x = (int)(indices[0] / scale);
                        int z = (int)(indices[2] / scale);
                         if(isSelectionPointOnHeightMap(x,z)) {
                        x = (int)(transx - indices[0] / scale);
                        z = (int)(transz - indices[2] / scale);
                        if(isSelectionPointOnHeightMap(x,z)) {
                     } else {
                        System.out.println("No mesh found");