Multithreading jME-Physics

Well, I've come up with an unfortunate problem that I guess I should have realized would be a potential problem.  ConcurrentModificationExceptions occurring because jME-Physics makes updates to jME within its thread make it impossible at the current time to do multithreading with jME-Physics.  I wrote this to try to support multithreading in a powerful way in jME-Physics and got slapped:


package test;

import java.util.concurrent.locks.*;
import java.util.logging.*;

import com.jme.util.*;
import com.jmex.game.state.*;
import com.jmex.physics.*;

/**
 * <code>PhysicsGameState</code> provides physics encapsulation into a GameState.
 *
 * @author Matthew D. Hicks
 */
public class PhysicsGameState extends BasicGameState {
   private PhysicsSpace physics;
   private PhysicsThread thread;
   
   public PhysicsGameState(String name) {
      super(name);
      physics = PhysicsSpace.create();
      thread = new PhysicsThread(physics, 60);
      new Thread(thread).start();
   }
   
   public PhysicsSpace getPhysicsSpace() {
      return physics;
   }
   
   public void setActive(boolean active) {
      super.setActive(active);
      thread.setEnabled(active);
   }
   
   public void shutdown() {
      thread.shutdown();
   }
   
   public void lock() {
      thread.lock();
   }
   
   public void unlock() {
      thread.unlock();
   }
}

class PhysicsThread implements Runnable {
   private PhysicsSpace physics;
   private long preferredTicksPerFrame;
   private boolean enabled;
   private boolean keepAlive;
   private Timer timer;
   private boolean limitUpdates;
   private Lock updateLock;
   
   public PhysicsThread(PhysicsSpace physics, int desiredUpdatesPerSecond) {
      this.physics = physics;
      enabled = false;
      keepAlive = true;
      updateLock = new ReentrantLock(true); // Make our lock be fair (first come, first serve)
      
      timer = new NanoTimer();
      if (desiredUpdatesPerSecond == -1) {
         limitUpdates = false;
      } else {
         preferredTicksPerFrame = Math.round((float)timer.getResolution() / (float)desiredUpdatesPerSecond);
         limitUpdates = true;
      }
   }
   
   public void setEnabled(boolean enabled) {
      this.enabled = enabled;
   }
   
   public void run() {
      // We have to lock it up while we're starting
      lock();
      
      long frameStartTick = 0;
      long frameDurationTicks = 0;
      float tpf;
      while (keepAlive) {
         if (limitUpdates) {
            frameStartTick = timer.getTime();
         }
         timer.update();
         tpf = timer.getTimePerFrame();
         update(tpf);
         if ((limitUpdates) && (preferredTicksPerFrame >= 0)) {
            frameDurationTicks = timer.getTime() - frameStartTick;
            while (frameDurationTicks < preferredTicksPerFrame) {
               long sleepTime = ((preferredTicksPerFrame - frameDurationTicks) * 1000) / timer.getResolution();
               try {
                  Thread.sleep(sleepTime);
               } catch (InterruptedException exc) {
                  LoggingSystem.getLogger().log(Level.SEVERE, "Interrupted while sleeping in fixed-framerate",
                              exc);
               }
               frameDurationTicks = timer.getTime() - frameStartTick;
            }
         }
      }
   }
   
   public void update(float tpf) {
      if (!enabled) return;
      // Open the lock up for any work that needs to be done
      unlock();
      // Now lock up again so nothing can happen while we're updating
      lock();
      
      physics.update(tpf);
   }
   
   public void lock() {
      updateLock.lock();
   }
   
   public void unlock() {
      updateLock.unlock();
   }
   
   public void shutdown() {
      keepAlive = false;
   }
}



Essentially what has to happen is either maybe have jME-Physics provide an interface that can be called before and after a change is going to be made to the jME scenegraph, or it needs to be able to somehow pass the changes through GameTaskQueue (which SimpleGame does support directly now BTW).

I'd love to be able to use this code I've written as it would make the game operate SO much smoother on hyperthreaded/multicore/multiprocessor machines.

Okay, I've gotten it to work, but ODE still crashes at a certain number of boxes at one time…I can't even get 500 boxes in the scene without it doing a native crash.  But that is just issues with ODE, nothing to do with my multithreading I don't think.



I need someone with a multicore/multiprocessor/ and/or hyperthreading machine to do some testing for me. :slight_smile:


package test;

import jmetest.*;
import jmetest.input.controls.*;

import com.jme.bounding.*;
import com.jme.image.*;
import com.jme.math.*;
import com.jme.scene.shape.*;
import com.jme.scene.state.*;
import com.jme.system.*;
import com.jme.util.*;
import com.jmex.game.*;
import com.jmex.game.state.*;
import com.jmex.physics.*;

public class StressPhysics {
   public static void main(String[] args) throws Exception {
      StandardGame game = new StandardGame("Stress Physics");
      game.getSettings().setVerticalSync(false);      // We want to see what FPS we're running at
      game.start();
      
      // Move the camera
      game.getCamera().setLocation(new Vector3f(0.0f, 50.0f, -100.0f));
      game.getCamera().lookAt(new Vector3f(0.0f, 0.0f, 0.0f), new Vector3f(0.0f, 1.0f, 0.0f));
      
      // Create a DebugGameState to give us the toys we want
      DebugGameState debug = new DebugGameState();
      debug.setActive(true);
      GameStateManager.getInstance().attachChild(debug);
      
      // Create our GameState
//      PhysicsMultithreadedGameState physics = new PhysicsMultithreadedGameState("PhysicsState");
      PhysicsGameState physics = new PhysicsGameState("PhysicsState");
      GameStateManager.getInstance().attachChild(physics);
      
//       Wait to activate the scene until we're all done
      
//        physics.lock();
       
      // Create the floor
      Box visualFloor = new Box("Floor", new Vector3f(), 500.0f, 0.5f, 500.0f);
       visualFloor.getLocalTranslation().set(0.0f, -25.0f, 0.0f);
       TextureState wallTextureState = DisplaySystem.getDisplaySystem().getRenderer().createTextureState();
        wallTextureState.setTexture(TextureManager.loadTexture(TestChooser.class.getResource("data/texture/wall.jpg"), Texture.MM_LINEAR, Texture.FM_LINEAR));
        visualFloor.setRenderState(wallTextureState);
       // Create the physics for the floor
        StaticPhysicsNode floor = physics.getPhysicsSpace().createStaticNode();
        floor.attachChild(visualFloor);
        floor.generatePhysicsGeometry();
        physics.getRootNode().attachChild(floor);
       
        // Create falling boxes
        TextureState ts = game.getDisplay().getRenderer().createTextureState();
       Texture t = TextureManager.loadTexture(TestSwingControlEditor.class.getClassLoader().getResource("jmetest/data/images/Monkey.jpg"), Texture.MM_LINEAR_LINEAR, Texture.FM_LINEAR);
       t.setWrap(Texture.WM_WRAP_S_WRAP_T);
       ts.setTexture(t);
        int boxes = 300;
        for (int i = 0; i < boxes; i++) {
           Box box = new Box("Box" + i, new Vector3f(), 5.0f, 5.0f, 5.0f);
           box.setModelBound(new BoundingBox());
           box.updateModelBound();
           box.setRenderState(ts);
          DynamicPhysicsNode node = physics.getPhysicsSpace().createDynamicNode();
          node.attachChild(box);
          node.generatePhysicsGeometry();
          node.setLocalTranslation((float)(Math.random() * 500.0f) - 250.0f, (float)(5.0f + (Math.random() * 100.0f)), (float)(Math.random() * 500.0f) - 250.0f);
          physics.getRootNode().attachChild(node);
        }
       
        // Update the rootNode
        physics.getRootNode().updateRenderState();
       
//        physics.unlock();
       
        physics.setActive(true);
   }
}



package test;

import java.util.concurrent.locks.*;
import java.util.logging.*;

import com.jme.util.*;
import com.jme.util.lwjgl.*;
import com.jmex.game.state.*;
import com.jmex.physics.*;

/**
 * <code>PhysicsGameState</code> provides physics encapsulation into a GameState.
 *
 * @author Matthew D. Hicks
 */
public class PhysicsMultithreadedGameState extends BasicGameState {
   private PhysicsSpace physics;
   private PhysicsThread thread;
   
   public PhysicsMultithreadedGameState(String name) {
      super(name);
      thread = new PhysicsThread(30);
      physics = thread.getPhysics();
      new Thread(thread).start();
   }
   
   public PhysicsSpace getPhysicsSpace() {
      return physics;
   }
   
   public void update(float tpf) {
      super.update(tpf);
      if (!thread.isEnabled()) {
         thread.setEnabled(true);
      }
   }
   
   public void shutdown() {
      thread.shutdown();
   }
   
   public void lock() {
      thread.lock();
   }
   
   public void unlock() {
      thread.unlock();
   }
}

class PhysicsThread implements Runnable {
   private PhysicsSpace physics;
   private long preferredTicksPerFrame;
   private boolean enabled;
   private boolean keepAlive;
   private Timer timer;
   private boolean limitUpdates;
   private Lock updateLock;
   
   public PhysicsThread(int desiredUpdatesPerSecond) {
      physics = PhysicsSpace.create();
      physics.update(0.01f);
      enabled = false;
      keepAlive = true;
      updateLock = new ReentrantLock(true); // Make our lock be fair (first come, first serve)
      
      timer = new LWJGLTimer();
      if (desiredUpdatesPerSecond == -1) {
         limitUpdates = false;
      } else {
         preferredTicksPerFrame = Math.round((float)timer.getResolution() / (float)desiredUpdatesPerSecond);
         limitUpdates = true;
      }
   }
   
   public PhysicsSpace getPhysics() {
      return physics;
   }
   
   public void setEnabled(boolean enabled) {
      this.enabled = enabled;
   }
   
   public boolean isEnabled() {
      return enabled;
   }
   
   public void run() {
      // We have to lock it up while we're starting
      lock();
      
      long frameStartTick = 0;
      long frameDurationTicks = 0;
      float tpf;
      while (keepAlive) {
         if (limitUpdates) {
            frameStartTick = timer.getTime();
         }
         timer.update();
         tpf = timer.getTimePerFrame();
         update(tpf);
         if ((limitUpdates) && (preferredTicksPerFrame >= 0)) {
            frameDurationTicks = timer.getTime() - frameStartTick;
            while (frameDurationTicks < preferredTicksPerFrame) {
               long sleepTime = ((preferredTicksPerFrame - frameDurationTicks) * 1000) / timer.getResolution();
               try {
                  Thread.sleep(sleepTime);
               } catch (InterruptedException exc) {
                  LoggingSystem.getLogger().log(Level.SEVERE, "Interrupted while sleeping in fixed-framerate",
                              exc);
               }
               frameDurationTicks = timer.getTime() - frameStartTick;
            }
         }
         Thread.yield();
      }
   }
   
   public void update(float tpf) {
      if (!enabled) return;
      // Open the lock up for any work that needs to be done
      unlock();
      // Now lock up again so nothing can happen while we're updating
      lock();
      
      physics.update(tpf);
   }
   
   public void lock() {
      updateLock.lock();
   }
   
   public void unlock() {
      updateLock.unlock();
   }
   
   public void shutdown() {
      keepAlive = false;
   }
}



package test;

import com.jmex.game.state.*;
import com.jmex.physics.*;

/**
 * <code>PhysicsGameState</code> provides physics encapsulation into a GameState.
 *
 * @author Matthew D. Hicks
 */
public class PhysicsGameState extends BasicGameState {
   private PhysicsSpace physics;

   public PhysicsGameState(String name) {
      super(name);
      physics = PhysicsSpace.create();
   }

   public void update(float tpf) {
      super.update(tpf);
      physics.update(tpf);
   }

   public PhysicsSpace getPhysicsSpace() {
      return physics;
   }
}



Test it first with the current code, and see how well it performs, and then comment out line 34, and uncomment lines 33, 39, and 74 and see if there's much if any of a performance gain.  I'm running on an AMD 64 but running in Windows XP in 32-bit so I'm not really seeing any performance boost.  I'm tempted to run this on my machine at work that reports 8 processors in Windows XP.  :D

I noticed the same problem, and haven't tackled it yet… not exactly sure how I want todo it.





Edit:



Odd… my FPS drops slowly until the boxes STOP moving? 



227 -> 76 



Shouldn't that be the other way around?





/Amd 2000, with a AGP GeForce 7800 GS, 100 boxes.

I noticed this as well.  I think it has to do with collisions stacking up?  It seems that when a box is just sitting on the platform it is actually using more system power (I'm assuming because of constant collisions) than when it's falling.



@Irrisor, can you shed some light on this?

Yes, each contact introduces a new constraint for the solver. Disabling boxes which do not move helps here. There is an auto disable feature in ODE. We should check that it is used by jME Physics 2 and adjust the thresholds triggering the auto disable.



Adding thousands of boxes (or sphere, or whatever) is not a problem. But you need to take away that floor (or disable gravity). As I already wrote ODE can’t handle too much collisions.



Using the mutithreaded version (with those lines commented in) gives me this error immediately:



(on a Core 2 Duo)

I just checked disabling and DynamicPhysicsNodeImpl.sceneToOde calls setEnabled( true ) each frame. It seems some lazy guy wrote this

It would need to know if the location was changed in the scenegraph. While jME does not support this we could read the old position from the body and only enable it if the location does not match.

Maybe I can look into this on the weekend, if nobody beats me to it :wink:

That ODE INTERNAL ERROR happens for me on both multithreading and non-multithreading if I tell it to create too many boxes ODE just flips onto its back and dies.



Try reducing the number of boxes it's creating and see if that fixes it.


irrisor said:

Maybe I can look into this on the weekend, if nobody beats me to it ;)


Do I still have write access to CVS?  It's been since jME-Physics 1 since I've done much with this so I'm still trying to catch up.

BTW, once these states are more stable would you think them beneficial in the project or would you rather them not?

It's not the same error with more boxes. Multithreading gives "invalid operation for locked space". Which seem to happen when updating ode stuff while collision detection or solver is running. Seems there still is some work to do.



Yes, you still have write access. And yes, such states would be beneficial, I think. You could put them in com.jmex.physics.util.states…

I don't see how that can be the case…there is a single thread that everything physics runs in and a single thread that everything jME runs in. Also, I'm disabling physics updates when I'm adding things to the scenegraph…



Yes, I've still got quite a bit of work to do before this is ready to go. :slight_smile:

Yay! I have already repaired auto disable. 300 boxes don't drop below 100fps :slight_smile:



With a little optimization in the natives I even get above 200fps with 300 boxes and still 60fps with 1000 boxes: if the collisions between two disabled bodies are no longer checked, collision detection has a lot less to do. The only drawback of this optimization (and that's why I did not commit it) are the missing collision events in this case. A box lying on the floor would no longer report any contacts. So jME Physics would need to keep a list of contacts from the last update step for all pairs of disabled bodies…



Regarding that error, darkfrog: It definitely only happens with the multi-threaded version. And it happens only with more than one core! So how could this be something different than a threading issue???

I committed the optimized native lib for windows. Add

physics.getPhysicsSpace().setAutoDisableThreshold( 0.05f );


To StressPhysics to see the difference :)

Well, given my own code I can't imagine how this could be the case.  I'm not doubting the fact that it is occurring in the multithreaded version and not the single-threaded, but the test doesn't touch ODE after startup except to call physics.update(tpf)…once I've created the scene all I'm doing is calling update in StandardGame and update in PhysicsMultithreadedGameState in two separate threads…

updateGeometricState touches ode!

hmmm…that could cause a problem. :o



I'm going to have to take some time and try to figure out a way to handle this…you have any ideas?

It would, of course, be possible to do that in the physics update instead. But the concept of the jME Physics ODE binding currently is to directly change ode stuff when the scenegraph is changed. So updating position in the physics update would not solve the entire problem. Deferring all ode calls into the physics update would result it quite an amount of work…

What about something like GameTaskQueue in jME-Physics that could allow you to defer those changes that have occurred because of the scenegraph change into the jME-Physics update?  That would then minimize the amount of changes, still keep the structure good, but allow for multithreading in jME-Physics.

Yes, if these requirements can be met:

  1. I'd want to have it transparent (the API interface should abstract from the internal deferring), which would cause quite some workload for implementation
  2. The queuing should be optional to avoid degraded performance for single threaded apps



    with "amount of changes" you meant implementation changes or ode updates?

I'll take a look and see if I can come up with any ideas.



I meant implementation changes.

Is the only place it calls off to ODE during the jME update in updateWorldVectors() in the PhysicsCollisionGeometry.



For example in OdeBox:


public void updateWorldVectors() {
        super.updateWorldVectors();
        //TODO: only if necessary!
        ( (OdePhysicsNode) getPhysicsNode() ).updateTransforms( geom );
        final Vector3f worldScale = this.worldScale;
        if ( worldScale.x <= 0 || worldScale.y <= 0 || worldScale.z <= 0 ) {
            // this makes ODE crash - so prefer to throw an exception
            throw new IllegalArgumentException( "scale must not have 0 as a component!" );
        }
        geom.setSize( worldScale );
    }



Then in there the only call off to ODE itself would be:

( (OdePhysicsNode) getPhysicsNode() ).updateTransforms( geom );



Is that right or is there more to it than that?

Sorry for posting to a dead thread, but its still the most relevant one I can see on the topic. I don't know if anyone is going to check this, so I'll repost it in its own topic in a few days if no one answers.



I noticed that PhysicsMultithreadedGameState has been added to the cvs, but using it I'm still getting the same ODE locked space error that Irrisor posted here.



So I have two questions:

  1) does anyone know if the issues discussed on this page were ever resolved (it could be that I'm just doing something wrong)?

  2) if I understand the source of the concurrency error properly, it is caused solely by calling updateGeometricState from outside the physics thread while the physics thread is working - for example, while updating the root node. Could this problem be resolved simply by putting all physicsNodes in their own root node, and then from within the multithreaded game state updating that node from the physics thread update call, while rendering that node in the game state render() method? Or does updateGeometricState need to be called from the GL thread?