Physics-library performance comparison

I compared the performance of the jme3-bullet, jme3-jbullet, and Minie physics libraries.

My metric was how many rigid bodies they could support at 15 frames per second.

I ran a small test program (source code below) for 3 different collision shapes: sphere, box, and hull.

I ran it 3 times for each (library, shape) pair.

I tried to keep the test conditions as similar as possible: 640x480 window, Vsynch on, assertions disabled, no JVM tuning, anti-aliasing disabled.

Here are the raw results (number of bodies @ 15 fps) for my Windows7 desktop:

                   Minie (v0.6.3)      jme3-jbullet (3.3-6703)    jme3-bullet (3.3-6703)
1. Sphere           528, 459, 414         1087, 1053, 1092            434, 549, 521
2. Box              298, 287, 327           528, 515, 527             327, 290, 309
3. Hull               10, 8, 9                26, 29, 23                13, 11, 10

Taking the median result for each (library, shape) pair and normalizing so that jme3-jbullet = 1.00 :

                   Minie (v0.6.3)      jme3-jbullet (3.3-6703)    jme3-bullet (3.3-6703)
1. Sphere                0.42                     1.0                     0.48
2. Box                   0.57                     1.0                     0.59
3. Hull                  0.35                     1.0                     0.42

In other words, jme3-jbullet handled about twice as many bodies as the native-based libraries. Not what I expected/hoped to find.

Is there something dubious about my methodology? Does JNI have some massive overhead that I’m unaware of?

Here’s the source code:

import com.jme3.app.SimpleApplication;
import com.jme3.bullet.BulletAppState;
import com.jme3.bullet.PhysicsSpace;
import com.jme3.bullet.collision.shapes.BoxCollisionShape;
import com.jme3.bullet.collision.shapes.CollisionShape;
import com.jme3.bullet.collision.shapes.HullCollisionShape;
import com.jme3.bullet.collision.shapes.SphereCollisionShape;
import com.jme3.bullet.control.RigidBodyControl;
import com.jme3.bullet.objects.PhysicsRigidBody;
import com.jme3.font.BitmapFont;
import com.jme3.font.BitmapText;
import com.jme3.math.Quaternion;
import com.jme3.math.Vector3f;
import com.jme3.scene.Geometry;
import com.jme3.scene.Node;
import java.util.Random;
import java.util.logging.Logger;

/**
 * Determine how many rigid bodies Bullet Physics can support at 15 fps.
 *
 * Run with Vsync enabled!
 */
public class RigidBodyStressTest extends SimpleApplication {
    // *************************************************************************
    // constants and loggers

    /**
     * message logger for this class
     */
    final public static Logger logger
            = Logger.getLogger(RigidBodyStressTest.class.getName());
    // *************************************************************************
    // fields

    /**
     * true for one frame
     */
    private boolean isFirstFrame = true;
    /**
     * scene-graph node for displaying user-interface text
     */
    private BitmapText uiText;
    /**
     * shape for falling gems
     */
    private CollisionShape gemShape;
    /**
     * accumulate tpf up to 1 second
     */
    private float secondCounter = 0f;
    /**
     * count the frames in one second
     */
    private int frameCounter = 0;
    /**
     * number of falling bodies in the scene
     */
    private int numGems = 0;
    /**
     * physics space
     */
    private PhysicsSpace physicsSpace;
    /**
     * pseudo-random generator
     */
    final private Random random = new Random(1L);
    // *************************************************************************
    // new methods exposed

    /**
     * Main entry point for the application.
     *
     * @param arguments array of command-line arguments (not null)
     */
    public static void main(String[] arguments) {
        RigidBodyStressTest application = new RigidBodyStressTest();
        application.start();
    }
    // *************************************************************************
    // SimpleApplication methods

    /**
     * Initialize this application.
     */
    @Override
    public void simpleInitApp() {
        configureCamera();
        configureUi();
        configurePhysics();
        addBox();

        int shapeId = 3;
        switch (shapeId) {
            case 1:
                gemShape = new SphereCollisionShape(0.1f);
                break;
            case 2:
                gemShape
                        = new BoxCollisionShape(new Vector3f(0.1f, 0.1f, 0.1f));
                break;
            case 3:
                Geometry teapot = (Geometry) assetManager.loadModel(
                        "Models/Teapot/Teapot.obj");
                gemShape = new HullCollisionShape(teapot.getMesh());
                gemShape.setScale(new Vector3f(0.5f, 0.5f, 0.5f));
        }

        gemShape.setMargin(0.005f);
    }

    /**
     * Callback invoked once per frame.
     *
     * @param tpf time interval between frames (in seconds, ≥0)
     */
    @Override
    public void simpleUpdate(float tpf) {
        super.simpleUpdate(tpf);

        if (isFirstFrame) {
            // The first frame includes startup time, so ignore it.
            isFirstFrame = false;
        } else {
            secondCounter += getTimer().getTimePerFrame();
        }
        /*
         * Calculate the frame rate and abort the test if it's too low.
         */
        frameCounter++;
        if (secondCounter >= 1f) {
            float fps = frameCounter / secondCounter;
            if (fps < 15f) {
                System.out.printf("final numGems = %d%n", numGems);
                stop();
            }
            secondCounter = 0f;
            frameCounter = 0;
        }
        /*
         * Add complex shapes once per second, simple shapes once per frame.
         */
        if (!(gemShape instanceof HullCollisionShape) || frameCounter == 0) {
            addAGem();
        }
    }
    // *************************************************************************
    // private methods

    /**
     * Add a falling PhysicsRigidBody to the scene.
     */
    private void addAGem() {
        float x = 2f * random.nextFloat() - 1f;
        float y = 2f * random.nextFloat() - 1f;
        float z = 2f * random.nextFloat() - 1f;
        Vector3f startLocation = new Vector3f(x, y, z);
        startLocation.multLocal(0.5f, 1f, 0.5f);
        startLocation.y += 4f;

        float mass = 1f;
        PhysicsRigidBody body = new PhysicsRigidBody(gemShape, mass);
        body.setDamping(0.6f, 0.6f);
        body.setFriction(1f);
        body.setKinematic(false);
        body.setPhysicsLocation(startLocation);

        physicsSpace.add(body);
        body.setGravity(new Vector3f(0f, -9f, 0f));

        ++numGems;
        /*
         * Update the user interface.
         */
        String msg = String.format("numGems=%d", numGems);
        uiText.setText(msg);
    }

    /**
     * Add a large static box to serve as a platform.
     */
    private void addBox() {
        Node boxNode = new Node("box");
        rootNode.attachChild(boxNode);

        float halfExtent = 50f;
        boxNode.move(0f, -halfExtent, 0f);

        Vector3f hes = new Vector3f(halfExtent, halfExtent, halfExtent);
        BoxCollisionShape bcs = new BoxCollisionShape(hes);
        float mass = 0f;
        RigidBodyControl boxBody = new RigidBodyControl(bcs, mass);
        boxBody.setPhysicsSpace(physicsSpace);
        boxNode.addControl(boxBody);
    }

    /**
     * Configure the camera during startup.
     */
    private void configureCamera() {
        flyCam.setEnabled(false);
        cam.setLocation(new Vector3f(0f, 1.5f, 7f));
        cam.setRotation(new Quaternion(0f, 0.9935938f, -0.113f, 0f));
    }

    /**
     * Configure physics during startup.
     */
    private void configurePhysics() {
        BulletAppState bulletAppState = new BulletAppState();
        bulletAppState.setDebugEnabled(true);
        stateManager.attach(bulletAppState);

        physicsSpace = bulletAppState.getPhysicsSpace();
        physicsSpace.setAccuracy(1f / 60); // 16.67 msec timestep
        physicsSpace.setSolverNumIterations(10);
    }

    /*
     * Add a BitmapText in the upper-left corner of the display.
     */
    private void configureUi() {
        BitmapFont font = assetManager.loadFont("Interface/Fonts/Default.fnt");
        uiText = new BitmapText(font);
        guiNode.attachChild(uiText);
        float displayHeight = cam.getHeight();
        uiText.move(0f, displayHeight, 0f);
    }
}

4 Likes

hmm. i didnt expect it too.

maybe its just wrote better way and thats all?

anyway you said there are no jbullet open source version as i remember, right?

in JNI need to care a lot about “clear memory” because garbage collector might not collect garbage from it if in end of method its not cleared.

but i think it might be case that even small methods in JNI require memory actions, while non-JNI java version can handle it better way.

anyway im curious too.

1 Like

Well in theory your approach isn’t good for a number of reasons:

  1. You measure FPS not frame time (minor problem)
  2. Mixing Visuals with Physics calculations (you shouldn’t show the spatials), because then: Have you used the Parallel Threading Mode? Otherwise you’d also have quadratic growth (one more spatial slows down rendering and physics).
  3. In regards of 2, the best would probably be to call update() without jme’s framework at all and measure the time. Even better: You could there run 1234892146234 update calls and take the sum of them.
  4. Maybe there are some discrepancies like the time it takes before an object is treated as inactive and all?
  5. Usually when doing jvm benchmarks you have to disable hotspot or “heating it up” by calling hundreds of iterations before measuring so that hotspot optimization takes place (though I guess java code profits from that)
  6. Know what you are measuring: The number of inactive Physics objects in a scene? Things may look different if you’d have a rotated ground where spheres are rolling down or when you do massive mesh accurate collisions (i.e. computational expensive situations vs. measuring the overhead for many bodies)

Those are my suggestions

2 Likes

I got following results in average running same test (15 fps) :

Sphere Shape :

  • jbullet: final numGems = 700
  • bullet : final numGems = 1060

Box Shape :

  • jbullet: final numGems = 210
  • bullet : final numGems = 630

For me native bullet was faster !!

1 Like

as i know JNI memory is not limited as standard Java code (via JVM setting).(At least not JNI related functions)

so if for example @Ali_RS got Java memory limit set in JVM or somewhere else, then jBullet will work slower ofc.

@Ali_RS could you try set more java(jvm) memory for app?

correct me if im wrong.

1 Like

Yes, I increased heap memory from -Xmx512m to -Xmx1024m but see no difference in result.

2 Likes

hmm, you sure its in proper place? (please note, if you run via IDE it might override it too)

could you try set more like 4G “-Xmx4G” (i hope you got 4 giga memory or more) and -Xms same as -Xmx

if this will change nothing, then i dont know why it work slower for you, while faster for @sgold

1 Like

Yep, I am setting it in gradle build

applicationDefaultJvmArgs = ["-Xmx4G","-XX:MaxDirectMemorySize=512m"]

I tried with both -Xmx4G and -Xms4G, yet no difference.

2 Likes

Yeah this doesn’t seem right. Although one thing I can say for sure is that jbullet really dies when compared to bullet in compound colliders.

Colliding two shapes of like a few hundred merged box colliders in a compound shape in both would give you something like 5 fps in jbullet and more than 60 in native.

Native also has the advantage of containing relatively recent updates and bugfixes since it’s not from 2008 like jbullet is.

1 Like

Thanks for your input, @Darkchaos .

I agree that FPS is a bad metric.

I think of frame times as a distribution that I’m sampling over 1-second intervals. The app stops adding bodies the first time the harmonic mean of sampled frame times exceeds 66.7 msec.

I could try a different stop criterion, such as:

  • the first time any frame’s time exceeds some threshold, or
  • when some % of the sampled frame times exceed some threshold.

Suggestions?

I had similar thoughts when I woke up this morning. I’ll try setDebugEnabled(false) and ThreadingType.PARALLEL and report results.

What would be the benefit?

I’m unclear what this would entail. Hundreds of build iterations? Or hundreds of simulation timesteps?

I had setDebugEnabled(true) so I could visualize what’s going on. It’s a pile of bodies on a flat surface. They’re all in a small area with plenty of collisions occuring.

Thanks again!

I thought of writing some kind of sampling method to determing how many objects and vertices could be handled by the graphics card in question. My thought was to add large amounts of objects/verts (whichever one I was testing) until the framerate started nearing the minimum fps and lessen the input as it drew closer. A bit like the opposite of an exponential function.

To allow for hiccups and anomalies my thought was to wait 5 or 10 seconds after the minimum FPS was reached. That way it would be a solid result that wouldnt be affected by anything for a (relatively) large duration of time.

1 Like

I’ve set up a project at GitHub:
GitHub - stephengold/Banana: Physics tests and performance benchmarks for the jMonkeyEngine3 game engine.

3 Likes

I thought name would be HowManyBananas :smiley:

3 Likes

Sorry for reading this so late:

That you remove any background things that could happen out of your measurement. It probably doesn’t change a thing because it affects all variants the same but things like inputManager querying the keyboard or classes re-calculating the World Bounds of a Sphere after falling would happen somehow in the background (Actually I guess both aren’t happening really but there could be something one does not think about).
Another thing in this could be GC: It could happen that when visualizing everything you run more quickly into garbage collection which kicks in at random times. That’s just things you don’t really want.

That’s a good question, since you are probably only measuring the simulation duration, simulation timesteps.

1 Like

Been spending way too much time on the Banana Project.

Made a couple improvements to RigidBodyStressTest:

  1. setDebugEnabled(false) and ThreadingType.PARALLEL.
  2. Catch the dynamic bodies on a cubic corner instead of a flat surface, so spheres don’t roll around so much.

Also added a new test app, called TestRunner, to the Project. This is similar to RigidBodyStressTest and tests the same collision shapes, but it incorporates some further refinements:

  1. Cleaner code design, with an AppState dedicated to low-level test management.
  2. The default measurement interval is much longer (300 seconds instead of 1 second), and each measurement is preceded by a warmup/stabilization interval (60 seconds). This makes the results more reproducible, though it does mean tests take longer to run.
  3. Bodies are added and removed at a steady rate during the stabilization and measurement intervals. This keeps all the bodies active and makes the workload more interesting.
  4. The main criterion for a sustainable load is now “average physics time-step < 16.7 msec”, instead of “average time-per-frame < 66.7 msec”.
  5. Bisection is used to find the maximum sustainable load, instead of increasing the load monotonically.
  6. For the math geeks, more statistics are printed, including a histogram.
2 Likes

The measurements still aren’t consistent enough for what I want to do, even with the 5-minute measurement interval. Could use a longer interval, of course, but then the bisection search will take even longer.

Rather than measuring capacity, it might be better to just measure latency at a single, fixed load.

Or perhaps what’s needed is a steadier workload. With box shapes in particular, randomness in the test sometimes creates voids that collapse suddenly.