Translate Screen X,Y coordinate into local enviroment coordinate

I am trying to figure out how to convert the local screen x,y coordinate (i.e mouse events) into the coordinates used by the geometric objects. I want to set a specific plane within the Camera view to translate the local x,y into the enviroment x,y. Can someone explain to me how the camera fulstrum is displayed onto the java panel ? Is the far plane the whole view that can be seen by the user (i.e. what is the relationship between the fulstrum and actual java panel size).


There are many frustum tutorials out there (here's one: that explain how the frustum works nicely.

Converting mouse coordinates to world coordinates requires a translation from 2d to 3d.  So you have the x and y coordinates from the mouse, but you also have to say how deep in your view frustum the pick is.  One method of doing this is using a Ray, casting it from your eye ball, through the point that was clicked in the near plane and seeing what it intersects.

And of course jME has some classes and demos for this. Look in the jmetest.intersection package.

Thanks for the response

I don't have a problem with the math behind the distance transalation but between the screen x and y and the conversion into the evniroment coordinates. Is it safe to assume that the screen x and y is an direct mapping of the near plane on the frustrum ?

My code is below and its behavior is very strange, if i move the camera in the x and y direction, the whole coordinate system would mess up.

 public Vector3f translate(float localx, float localy){
       float near = impl.getRenderer().getCamera().getFrustumNear();
       float ResolutionRatio = (float) glCanvas.getWidth()/ (float) glCanvas.getHeight();

       float nearHeight = FastMath.tan(FastMath.DEG_TO_RAD*22.5f)*near*2f;
       float nearWidth = nearHeight*ResolutionRatio;
       float ratioX = nearWidth/(float) glCanvas.getWidth();
       float ratioY = nearHeight/(float) glCanvas.getHeight();
       Vector3f location = impl.getRenderer().getCamera().getLocation();
        Vector3f direction = impl.getRenderer().getCamera().getDirection();
        Vector3f nc = location.add(direction.mult(near));
        Vector3f zc = location.add(direction.mult(location.z));
        Vector3f up = new Vector3f(0,1,0);
        Vector3f right = up.cross(direction);
       //Get Top Left corner of near plane so we can map it to the screen coordinates
        Vector3f origin = nc.subtract(up.mult(nearHeight/2f)).add(right.mult(nearWidth/2));
       // Translate the screen coordinate into the near plane coordinate
       Vector3f first_translation = new Vector3f(localx*ratioX + origin.x,localy*ratioY + origin.y,0);
       first_translation = first_translation.subtract(location);
       // translate the near plane coordinate into the plane z=0 coordinate (This is only for the 2d case where all object are at z=0)
       float heightRatio = FastMath.abs(location.z)/near;
       Vector3f second_translation = new Vector3f(first_translation.x * heightRatio, first_translation.y * heightRatio, 0);
       second_translation = second_translation.add(location.x,location.y,0);
       return second_translation;

Any help would be greatly appreciated.

display.getWorldCoordinates method might be helpful to you.

Something like.

Vector2f screenPos = new Vector2f();
screenPos.set(absolutemouse.getHotSpotPosition().x, absolutemouse.getHotSpotPosition().y); //mouse location on the screen
Vector3f worldCoords = display.getWorldCoordinates(screenPos, 1f); //world location of that x, y value

Then if you wanted to cast a ray

Ray mouseRay = new Ray(cam.getLocation(), worldCoords.subtractLocal(cam.getLocation()));

You also can use display.getPickRay to skip all the messy steps.

renanse said:

You also can use display.getPickRay to skip all the messy steps.

Leave it to Renanse to do it the easy way. :P

Heh, I'm lazy… that's why I write easy ways. :wink:

lazyness is the only true path to clean, optimized code…i live by that motto

I tried that motto for a while, but then a few weeks went by and nothing got done so I decided to re-evaluate. :wink:

Perhaps I was just an overachiever though.  8)

well, if that aint clean and optimized, i dunno what is :wink: