[SOLVED] EXCEPTION_ACCESS_VIOLATION after 64 KinematicRagdollControls

Yes, basically the class was written for two reasons that plagued JME all over the place:

  1. repeatedly iterating over mostly static lists inefficiently.
  2. iterating over lists of listeners where a child might want to remove itself during iteration.

Probably this was just never converted.

1 Like

ah well @pspeed hit the nail on the head i guess, bunch of words wasted from my side :smiley:

To be thorough, you would also have to setGravity on the baseRigidBody of the KinematicRagdoll.

If you think the list of collision listeners is worth upgrading, open an issue, and I’ll handle it.

1 Like

I just had a go with the DLL containing the implementation of finalizeNative. When the 64th KinematicRagdollControl gets added to a PhysicsSpace (using parallel threading) I still get an EXCEPTION_ACCESS_VIOLATION, but it’s no longer complaining about UnsatisfiedLinkErrors. It’s now complaining about PrivilegedActionExceptions and something called “implicit null exceptions”:

Event: 3.219 Thread 0x0000000059d7b000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1774410) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.220 Thread 0x0000000059d7b000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1775038) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 19.476 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000031f4e36 to 0x00000000031f5819
Event: 25.686 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000036b48b1 to 0x00000000036b4b9d
Event: 25.686 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000035ca612 to 0x00000000035caa12
Event: 25.686 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000036b0afc to 0x00000000036b0d39
Event: 25.687 Thread 0x0000000059d7b000 Implicit null exception at 0x000000000361f030 to 0x000000000361f6cd
Event: 25.702 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000036e9ab4 to 0x00000000036ea4d9
Event: 26.006 Thread 0x0000000059d7b000 Implicit null exception at 0x000000000370e5c5 to 0x0000000003710e25
Event: 28.600 Thread 0x0000000059d7b000 Implicit null exception at 0x00000000035ec0cf to 0x00000000035ed525

I don’t know whether that’s due to an error in the implementation of finalizeNative, or something completely different.

Before I created this topic, and tried to write a simple app to reproduce the EXCEPTION_ACCESS_VIOLATION I was getting. I was intending it to add a KinematicRagdollControl to a PhysicsSpace 64 times. However, instead, I found that I got an EXCEPTION_ACCESS_VIOLATION just trying to do one ragdoll. The body of the app is given below.

    private Spatial model;
    private BulletAppState bulletAppState;
    
    @Override
    public void simpleInitApp() {
        cam.setLocation(new Vector3f(0, 4, 18));
        
        /** A white ambient light source. */ 
        AmbientLight ambient = new AmbientLight();
        ambient.setColor(ColorRGBA.White);
        rootNode.addLight(ambient);
        
        /** A white, directional light source */ 
        DirectionalLight sun = new DirectionalLight();
        sun.setDirection((new Vector3f(-0.5f, -0.5f, -0.5f)).normalizeLocal());
        sun.setColor(ColorRGBA.White);
        rootNode.addLight(sun); 
        
        loadModel();
        createBulletAppState();
        addFloor();
        startRagdoll();
    }
    
    private void loadModel() {
        model = assetManager.loadModel("Models/Sinbad/Sinbad.mesh.xml");
        
        model.setLocalTranslation(0, 10, 0);
        
        rootNode.attachChild(model);
    }
    
    private void createBulletAppState() {
        bulletAppState = new BulletAppState();
        
        stateManager.attach(bulletAppState);
    }
    
    private void addFloor() {
        Plane plane = new Plane(Vector3f.UNIT_Y, 0);
        CollisionShape shape = new PlaneCollisionShape(plane);
        PhysicsRigidBody floor = new PhysicsRigidBody(shape, 0);
        
        bulletAppState.getPhysicsSpace().add(floor);
    }
    
    private void startRagdoll() {
        KinematicRagdollControl control = new KinematicRagdollControl();
        
        model.addControl(control);
        
        bulletAppState.getPhysicsSpace().add(control);
        
        control.setRagdollMode();
    }

Running this, if I use native Bullet, I get an EXCEPTION_ACCESS_VIOLATION; if I use jBullet, it works absolutely fine. Does that fall over with native Bullet for anyone else? The funny thing is, my game has ragdolls, so it’s not as if my computer is too old to do ragdolls or something (although it is old). The internal exceptions I get in this case are:

Event: 0.438 Thread 0x00000000024b8000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000e04e7798) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\methodHandles.cpp, line 1167]
Event: 0.441 Thread 0x00000000024b8000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000e04f4ae8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\methodHandles.cpp, line 1167]
Event: 0.546 Thread 0x0000000059ca5000 Exception <a 'java/io/FileNotFoundException'> (0x00000000e0679a58) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 0.688 Thread 0x0000000059ca5000 Exception <a 'java/lang/NoClassDefFoundError'> (0x00000000e07195c8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1394]
Event: 3.104 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e17018a8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.104 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1703418) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.104 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1704538) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.105 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1709658) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.105 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e170a2b0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.105 Thread 0x0000000059fdc000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e170aed8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]

Again, they include PrivilegedActionExceptions. So, when I run my game with the latest bulletjme.dll, and get an EXCEPTION_ACCESS_VIOLATION caused by PrivilegedActionExceptions, unfortunately I’m not in a position to say whether that’s because there’s something wrong with the implementation of finalizeNative, or whether it’s due to whatever causes that test app to not work for me either.

1 Like

I think you hit issue 740:
https://github.com/jMonkeyEngine/jmonkeyengine/issues/740

Could you try again with a recent build of JME from the master branch? If not, you can probably work around the issue by copying the setupSinbad() routine from jme3test.bullet.TestBoneRagdoll and invoking it right after constructing the control.

First, as per issue 740, using 3.2.1, TestBoneRagdoll works fine for me, but TestRagdollCharacter results in an EXCEPTION_ACCESS_VIOLATION. However, after upgrading to the 3.3.0-SNAPSHOT version of native Bullet, TestRagdollCharacter works fine. So, it certainly seems that 740 will be fixed in 3.3.0. :+1:

Second, upgrading to 3.3.0-SNAPSHOT doesn’t fix my Sinbad test app. I still get an EXCEPTION_ACCESS_VIOLATION with exactly the same internal exceptions. However, incorporating the setupSinbad method does fix it, even if I’m still using 3.2.1. So, that’s presumably caused by a different problem.

Third, I’ve managed to reproduce an EXCEPTION_ACCESS_VIOLATION in a similar way to how it happens in my game, but with far less code, as shown below.

    private int count;
    
    @Override
    public void simpleInitApp() {}
    
    @Override
    public void simpleUpdate(float tpf) {
        System.out.println(++count);
        
        BulletAppState bulletAppState = new BulletAppState();
        
        bulletAppState.setThreadingType(ThreadingType.PARALLEL);
        
        stateManager.attach(bulletAppState);
    }

So, it simply adds another BulletAppState with parallel threading every frame. For me, I get an EXCEPTION_ACCESS_VIOLATION after precisely 66 BulletAppStates have been added. Removing the BulletAppStates makes no difference. The internal exceptions (below) are exactly the same as I was getting with my Sinbad test app before. Whether that means it’s caused by the same problem, or whether it’s just coincidence, I wouldn’t know.

Event: 0.411 Thread 0x0000000002708000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000e04ef408) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\methodHandles.cpp, line 1167]
Event: 0.414 Thread 0x0000000002708000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000e04fc758) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\methodHandles.cpp, line 1167]
Event: 0.533 Thread 0x0000000059cd9000 Exception <a 'java/io/FileNotFoundException'> (0x00000000e0676fa8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 0.677 Thread 0x0000000059cd9000 Exception <a 'java/lang/NoClassDefFoundError'> (0x00000000e0716d18) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1394]
Event: 3.114 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e178d1c0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.114 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e178ed30) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.114 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e178fe50) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.115 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1794f70) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.115 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e1795bc8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]
Event: 3.115 Thread 0x0000000059fc4000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000e17967f0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u161\10277\hotspot\src\share\vm\prims\jvm.cpp, line 1390]

Although my game gets different internal exceptions, if whatever is causing the simple test app above to crash could be fixed, it might also fix the issue with my game, because the behaviour is similar. It’s certainly the nearest I’ve been able to get to reproducing the problem with a standalone app.

I’d be interested to know if anyone else gets the same behaviour running the above app. In case it makes a difference, I’m using 64-bit Windows.

1 Like

James:

With the code shown, I was able to reproduce the crash on my 64-bit Windows machine.

OK, that’s good. Obviously, it’s not as easy to see what the issue is with this one. NoSuchFieldError, FileNotFoundException, and NoClassDefFoundError sound as if something’s missing, although I can’t easily tell what that is from looking at the error report file.

Should I raise it as an issue on GitHub?

1 Like

Sure, open an issue.

I did find this in btQuickprof.h:

BT_QUICKPROF_MAX_THREAD_COUNT = 64;

As I suspected, disabling Quickprof resolves the issue.

Interesting! And well spotted! But, how does a max thread count being exceeded result in a NoClassDefFoundError?

Anyway, a max thread count of 64 may well be a perfectly reasonable feature of Bullet, as long as it’s talking about concurrent threads. But, if I repeatedly attach and detach the same BulletAppState (see below), I get the same EXCEPTION_ACCESS_VIOLATION, even though in that case there will never be more than one concurrent BulletAppState.

    private int count;
    private int frame;
    private BulletAppState bulletAppState;
    
    @Override
    public void simpleInitApp() {}
    
    @Override
    public void simpleUpdate(float tpf) {
        if (frame % 4 == 0) {
            System.out.println(++count);
            
            bulletAppState = new BulletAppState();
            
            bulletAppState.setThreadingType(ThreadingType.PARALLEL);
            
            stateManager.attach(bulletAppState);
        } else if (frame % 4 == 2) {
            stateManager.detach(bulletAppState);
        }
        
        frame++;
    }

So, if my understanding of the issue is correct, when I create the issue, I’ll include the above code, and state that the issue is that if the total number of BulletAppStates ever attached exceeds 64, then the JVM crashes, as opposed to the total number of BulletAppStates that are currently attached.

If the total number of BulletAppStates currently attached exceeds 64, then it probably still shouldn’t cause the JVM to crash, but, that’s a separate issue.

I see btQuickprof.h is in Bullet. Does this mean that the issue is in Bullet? Or is it something you reckon we can fix in jMonkeyEngine?

Of course, key to this problem is that the threading type is set to parallel, and in the error report file, I see 66 Java threads: pool-1-thread-1 to pool-66-thread-1. Perhaps the problem is that jMonkeyEngine isn’t terminating the thread when the BulletAppState is detached from the AppStateManager?

1 Like

Issue opened: https://github.com/jMonkeyEngine/jmonkeyengine/issues/928

1 Like

Given its function, I suspect QuickProf keeps data on every thread it sees, even after that thread is detached.

The planned fix involves recompiling Bullet with the #define that disables QuickProf. Since there’s currently no JNI interface to access QuickProf data, it’s no big loss.

1 Like

You should also add a link to this forum thread in your issue post.

Disregard, already in post, sorry about that.

I’ve just tried out the fix for 928, and I can confirm that my game no longer crashes after 64 ragdolls! :grinning:

Thanks for everything, Stephen. You’ve been a tremendous help. :+1:

1 Like

As you probably know, JME’s KinematicRagdollControl is an unfinished work in progress. It’s been that way for many years. I try to fix any issues I find in it. Even though it’ll be deprecated in JME 3.3, I think such fixes will add value to JME 3.2.2 .

My main project right now is to develop an improved kinematic ragdoll control that works with models other than Sinbad. For the moment I’m targeting JME 3.2.* . I’ll post a forum announcement when it’s ready for testing.

4 Likes

Yes, I noticed that KinematicRagdollControl is now deprecated in the master branch. Is there something somewhere explaining what it’s going to be replaced with?

I didn’t realise there was going to be a 3.2.2. I take it 3.2.2 will also have the fixes for 913 and 928 in it? And presumably those fixes will still be necessary in 3.3.0? PhysicsJoint will still need a native implementation of finalizeNative, and QuickProf will still need to be disabled in case you have more than 64 BulletAppStates.

Yes, when I first came to add ragdolls to my game, KinematicRagdollControl didn’t work with my model. It was falling over when it tried to create the CollisionShapes. I managed to work around it by supplying a separate j3o containing the model’s collision shapes. Took a bit of work to integrate that into KinematicRagdollControl, but I managed it eventually. The rest of KinematicRagdollControl's functionality works perfectly fine with my model though.

1 Like

If you haven’t already, read this topic:
Next release(s) of the Engine

I expect that 3.3.x releases (when they occur) will incorporate all changes currently in master, and that simple bug fixes currently in master (such as 913 and 928) will get back-ported to the v3.2 branch for inclusion in future 3.2.x releases (when they occur). However, I don’t expect to make those decisions and don’t know when those releases will occur.

I’m impressed that you got KRC to work with a model other than Sinbad.

I suspect the main reason KRC is deprecated in master is that it relies heavily on Bone, Skeleton, and SkeletonControl, which are also deprecated. If someone were to rewrite KRC to use Joint, Armature, and SkinningControl instead, there’d be less reason to deprecate it. But as I said, I’m focused on making it work, for me, with the old animation system; I’ve barely looked at the new animation system.