[SOLVED] ACCURATE spatial-on-screen detection

Troubleshooting General Help
1

Hi all,

I’m detecting whether a spatial is on-screen with the following code:



[java]int planeState = cam.getPlaneState();

cam.setPlaneState(0);

FrustrumIntersect result = cam.contains(spatial);

cam.setPlaneState(planeState);

boolean onScreen = result.equals(Camera.FrustumIntersect.Inside) || result.equals(Camera.FrustumIntersect.Intersects);[/java]





The trouble is that the BoundingBox (surrounding the irregularly shaped spatial) isn’t tight, so I get false positives.



For example, here the spatial is correctly reported as being on screen (BoundingBox shown in red):







And here the same spatial is incorrectly still reported as being on screen:







I really need it to be accurate. I know I can make a tight CollisionShape using the physics system, but how would I convert that to a BoundingVolume to use with cam.contains() ? And I didn’t feel I could use a ray, because I won’t know which point on the spatial to fire it at…



Many thanks,

Peter.

2

It’s not a trivial operation, doing collisions. You could use a physics mesh collision shape for the model and set up collision planes along the view frustum.

3
@phate666 said:
This could be easily done with occlusion queries, but the devs have no interest in adding them to the engine...

Anyone is welcome to add and submit features. All devs are unpaid and have to set priorities for these items.
4

The patch wasn’t complete enough to include into core. It just needs some work. If you open up a new thread (or re-open the previous one) you could work with @Momoko_Fan and @nehon to get it solid enough to put into core.

5

Thanks @Sploreg. SO, leaving the discussion on occlusion queries to one side for the moment… Here’s the solution I’m trying to implement:


  1. Work out the direction from camera along the right of its frustum


  2. Shine a ray along that direction and see if it intersects with a tight mesh created using CollisionShapeFactory.createDynamicMeshShape. Shining this ray half way up the screen should be an ok simplification.



    Unfortunately I’m having difficulty with the maths of step 1. Here’s what I’ve got:



    [java]// Point in the middle of the frustum far - this works fine

    Vector3f frustumFarCenter = cam.getLocation().add(cam.getDirection().normalize().mult(cam.getFrustumFar()));



    // Point on the right of the frustum far - this is not the correct result

    Vector3f frustumFarRight = new Vector3f();

    frustumFarRight.setX(frustumFarCenter.getX());

    frustumFarRight.setZ(frustumFarCenter.getZ() + (FastMath.tan(fov/2) * cam.getFrustumFar()));



    // Now shine a ray from cam.getLocation() to frustumFarRight[/java]



    I think the reason it’s not working is that I’m hard coding which is X and which is Z? Getting slightly out of my depth here :S
6

Tried a quaternion approach to step 1, equally miserable failure:



[java]// Take the vector from the camera to the frustum far’s center, rotate it by FOV/2 degrees to get the right side of the frustum

Quaternion q = new Quaternion().fromAngleAxis((fov/2)*FastMath.DEG_TO_RAD, Vector3f.UNIT_Y.clone());

frustumFarRight = q.mult( frustumFarCenter.subtract(cam.getLocation()) );[/java]

7

Sorry for all the spam, I’ve got there now.



Here’s the code for future reference:



[java]// Work out camera’s right-hand frustum edge in world space:

Vector3f frustumFarCenter = cam.getLocation().add(cam.getDirection().normalize().mult(cam.getFrustumFar()));

Quaternion frustumAngle = new Quaternion().fromAngleAxis((-fov/2)*FastMath.DEG_TO_RAD, Vector3f.UNIT_Y.clone());

Vector3f frustumRightDirection = frustumAngle.mult(frustumFarCenter.subtract(cam.getLocation()).normalize());

Vector3f frustrumFarRight = cam.getLocation().clone().add(frustumRightDirection.mult(cam.getFrustumFar()));[/java]

8

Can I ask why it’s so important you know this? There may be a simpler way to achieve the end result.

9

Thanks @zarch , yes this isn’t a very elegant solution.



The JMonkey app creates an animation, exported a frame by frame to form a video. It’s for use in a scientific experiment - we’ll be scanning the brains of volunteers while they watch the videos! I need to know exactly what’s in each frame as it’ll be correlated against their brain activity. The motion is complex and irregular, so is custom implemented with rules in the simpleupdate() method rather than using cinematics.



So… Any simpler way of accurately knowing what’s in each frame?



Cheers!

Pete.

10

@peterz



One idea that occurred to me:



Since frame rate isn’t a problem you could run your rendering twice.



The first time set each object to unshaded with a different colour.



Then its just a matter of looking at what colours are present that frame.



Then put your normal materials in and record the video for your test subjects to watch.



(So essentially I’m suggesting a 3 stage process:



Stage 1: render in false colour

Stage 2: analyse renderings to see what objects were present each frame

Stage 3: render for real)



You would need to be careful to synch the two runs to make sure everything happened at the same speed but that should be reasonably straightforward.



Alternatively you could even theoretically use two viewpoints and two “virtual” root nodes and render both at once…

11

Thanks, I think that’s a clever suggestion. But I do wonder if there’s a simpler approach. The rendering engine must know what it’s rendering in a frame, so presumably there’s scope for me to add a listener hook somewhere…?

12

The problem is that the fine detail determination of that is done inside the graphics card. The bounding box is just used to do the initial rough culling.



There is a way in Open GL of getting flags to be rendered into buffers (like z buffer but different) - http://en.wikibooks.org/wiki/OpenGL_Programming/Stencil_buffer. That would be ideal if possible as you could still do it all in one render step and then just check the stencil buffer afterwards to see what objects were rendered.



I don’t know if/how/etc JME3 lets you interact with that though, you’d need someone with more experience than me in this area to comment as to be honest we’re right at the edge of what I know about :slight_smile:

13

Generally this would be done with opengl occlusion queries which give you a “visible pixel count” for each mesh rendered. Unfortunately jme3 doesnt support it out of the box since its only used for very specific circumstances such as yours. There was however a patch posted on the forum that enabled this feature but I am not sure about how easy it is to use

14

Thanks all.



@phate666 and @Momoko_Fan @zarch - I think this will be very helpful when eventually implemented.



EDIT: Finally resolved this below



May I just ask you all a final (possibly obvious) question to get this resolved…



I now have a ray pointing where I want, along the right of the camera frustum:



[java]Ray ray = new Ray(cam.getLocation(), frustumRightDirection);[/java]



And I have a physics mesh for the spatial:



[java]CollisionShape shape = CollisionShapeFactory.createDynamicMeshShape(l.getSpatial());[/java]



As CollisionShape doesn’t implement Collidable, how can I manually collide them? Looking at the JME code, it seems that even if I set up the Spatial up with a RigidbodyControl initialized with the CollisionShape, the ray.collidewith() method can only use a BoundingVolume…



Cheers,

Peter.

15

OK it’s done. The above was achieved by adding each spatial to the physics space (which I was trying to avoid to save memory, but never mind) using a GhostControl for each.



So here’s my pseudocode for the whole thing:



function checkForCollisions(){

  1. Check which objects are in view with cam.contains() on each spatial


  2. Any spatial with Camera.FrustumIntersect.Inside is definately on screen, so add them to the list of visible spatials


  3. Any spatial with result.equals(Camera.FrustumIntersect.Intersects) might be on screen, or it might be that the edges of its bounding box are tricking us. So…

    3a. Use bulletAppState.getPhysicsSpace().rayTest() to cast rays from the camera along its left and right frustum. If they fail to intersect with the tight CollisionShape, we will assume this was a false positive and the spatial was not visible.

    }



    Phew!
16

That will work so long as objects never enter/leave by the top or bottom of the screen (or near the top bottom of the screen edges) and the objects are not terribly irregular shapes.



You may be able to intersect with a plane to improve on that side of things… Could even define 4 planes, one for each side of the screen.

17

Oh excellent idea @zarch, that’ll make it a lot more stable. Thanks.

18
@zarch said:
You may be able to intersect with a plane to improve on that side of things... Could even define 4 planes, one for each side of the screen.

uh...that's pretty much what the cam does, except it has 2 more planes for near and far
19

I simplified the solution by casting a grid of rays across the whole frustum.



Cheers all,

Peter.

20

Hi peterz,
Seems like I’m looking at the exact same problem. Can you share your solution, the code for accurately determining on screen spatials? Cheers