Inverse Kinematics control + skeleton debugger improvements

[nicholas_leehone]

Here is the current progress that I have made on getting an inverse kinematics control to work nicely with animations. The IK solver is based on a world space exact calculation, and cyclic coordinate descent which is repeated a given number of times.



It is still a work in progress but once it is done (and I have cleaned the code up a bit), I will release the source.

I have also implemented some changes to the skeleton debugger so that I could see what was going on easier (which will also be uploaded).



In the video, there are two models, the one in the centre has my inverse kinematics control, as well as a simple animation (done in blender) running at the same time. On the right model, there is only the animation for comparison. Towards the middle of the video, I show what happens when you ask a bit too much of the IK control, and it starts popping back and forth. This is normal since it is trying to match the animation which flips from side to side, and at the same time, satisfy the IK, which it can only do if it jumps like that.



The bottom bone sometimes bends in the opposite direction of the motion, which is also normal, since it tries to match the animation which is only on bone2, so bone1 tries to rearrange itself to compensate for that.



Sorry for the poor quality of the video, this is the first time that I am using recording software and my laptop seems to struggle to record at a decent frame rate (although if you look at the frame rate of the program, it is doing quite nicely even with the IK control).

http://www.youtube.com/watch?v=utxRMF0Chbk

The next thing I am planning on implementing are constraints for the bones. The end goal is to have an IK control which can easily be used to correctly position a model's foot on the ground during animation, or move a hand to an object, or any other use you can think of.

I would welcome any feature request (that is reasonable), as I am running out of ideas as to what useful things need to be implemented in this sort of control.

Todo:
- Constraints
- Render constraint boxes
- Automatic blender constraint loading
- Procedural animation

[Setekh]

Wow, something like that would be quite the asset for jme3, I’m excited.

You should get in contact with Kaelthas, he did the blender importer; working together to implement inverse kinematics and constraints for blender importer would be the next best thing imho.



This is exciting can’t wait to hear more about it!



PS: I’m gonna freak if i see procedural animations XD

[nehon]

Awesome!!

[nicholas_leehone]

@Setekh I’ve been using the ogrexml exporter for blender as for some reason I couldn’t get the blender importer to work with blender 2.6 models. So you’re suggesting exporting the IK constraints from blender so that they are loaded automatically instead of having to create the IK chain by hand?



I’ll also start looking into procedural animation. Quick google search revealed this : http://runevision.com/thesis/rune_skovbo_johansen_thesis.pdf

Summer project

[remorhaz]

@nicholas-leehone said:
I'll also start looking into procedural animation. Quick google search revealed this : http://runevision.com/thesis/rune_skovbo_johansen_thesis.pdf
Summer project :D

It is basically Unity's locomotion system:
http://unity3d.com/support/resources/example-projects/locomotion-ik

I mean the guy who wrote the thesis made a reference implementation for Unity. It is not completely procedural but a mixture of example based and procedural techniques. It is a free plugin but I don't know if it is open source or not.

I am reading the thesis right now and wanted to implement it to JME. But I have a lot catching up as I am new to JME and know basically nothing about how animations work (but I learn fast, hehe) so you probably have a better chance to make something that actually works in less time. If you need an apprentice let me know, don't feel obliged though, I have other project ideas.

[nicholas_leehone]

Yeah, I noticed that it was implemented in Unity. As for the open source or not question, I have no plans to copy the code from Unity so it’s not a problem. I am just going to use the information in the thesis to come up with my own code. Since I already have the IK solver in place, I don’t think it would be too hard to implement (wishful thinking I know). But before I do that, I need to make sure I can set constraints so that someone’s leg doesn’t bend in the opposite direction that it is supposed to.



Unfortunately exam time is coming for me now, so I’m gonna have to slow down the development for a bit. But I’ll keep everyone updated in this thread as I progress.

[Setekh]

@nicholas.leehone creating the IK chain by hand will give more mobility and that’s cool; hope it will have both methods. But what i meant to say is that @Kaelthas is interested in the idea as far as i know since there are some empty classes in the trunk… thought maybe you guys could help each other.



I would really like to see this happen since i really believe this project has way more potential then some block game minecraft copy and showing off shaders/filters. I bet this will inspire people to do more creative stuff.

[glaucomardano]

Yay, this is what I needed :).

[aevun]

Procedural animation? Like in Overgrowth?



If that is the case; Hell Yeah

[nicholas_leehone]

@Setekh Good point (about keeping the two methods). I will get in contact with Kaelthas once exams are over.

[nicholas_leehone]

All right, small update now that exams are finally over. I have looked into implementing constraints and because of the method I am using to solve the ik chain, I don’t think it is possible.



The way that my system works is by calculating the exact analytical solution for the first two bones in the chain, then locking the first bone and second bone together, and using their parent as the second bone and solving the exact solution again. Repeat this till the end of the chain and then iterate a couple of times. It’s not exactly CCD since it doesn’t move the bones incrementally.



This gives extremely stable solutions (and repeatable) that are as close to the animation as possible, while being relatively quick (it converges usually within a couple of iterations). However, since I do not move the bones incrementally, I can’t figure out a way to implement constraints. If anyone has any ideas about how to do this, I would be glad to hear them.



So having given up on the constraints (for now), I am moving on to implementing this ik chain into a full leg controller. The first step is finding a decent model with a walk animation implemented (front, back and sideways). I am not bad with blender but I’d rather spend my time designing the Ik rather than modeling.



I’m still cleaning up my code for the ik chain, and once it is done and a bit more flexible i’ll release it (maybe a couple more weeks).

[Setekh]

Well about the constraints that’s beyond me, and you don’t really need to model just use some capsules and a blender armature rig (Rigify addon in blender).

I’d love to help you tho im going on a little trip so i won’t be back till Tuesday, and on that trip i’ll still have to work on shader injection(with no internet connection). So as much as i like and admire what are you doing i can’t help till Tuesday. >.<



Anyway glad to see ya back xD

[nicholas_leehone]

All right, so I managed to hack together a test model in blender 2.6 using some boxes and the default armature + root bone and two target bones (that aren’t used yet). Looks pretty basic but it’s enough for now.



This is just a quick and dirty test to see what the capabilities of the current system are (hacked together in about half an hour). There are two models once again, one that serves as reference, and the other that is being controlled by the IK. The one being controlled by the IK has the target set by the blue box (only on one leg). It still has plenty of issues, but considering it was only half an hour of work, i’m happy :D.



http://youtu.be/ikPHVttn7WE

Methodology:
- Cast a ray from the hip bone to the foot bone (where the blue box is)
- If the ray intersects the ground above the level of the foot bone, move the foot bone, otherwise don't (to prevent the foot sticking to the ground)
- Apply a slight offset in the y direction to help avoid the foot penetrating the ground too much

Ok, back to the boring part of cleaning up the code for release ;)

[nicholas_leehone]

All right, so here it is, my code for the Ik control. I’ve cleaned it up a bit and made it easier to use. If anyone finds any bugs or finds any way to make the code better, please let me know.



[java]

public class IkControl extends AbstractControl{

private Bone targetBone = new Bone();

private Bone firstBone = new Bone();

private int maxChain;

private Skeleton skeleton;

private Node target;



// How close we have to be before giving up

private float threshold;



// Number of times to attempt IK solution, more is slower but more accurate

private int iterations;



public IkControl(){

iterations = 10;

threshold = 0.001f;

maxChain = 1;

}



public IkControl(Skeleton s){

skeleton = s;

iterations = 10;

threshold = 0.001f;

maxChain = 1;

}



public void setMaxChain(int max){

maxChain = max;

}



public int getMaxChain(){

return maxChain;

}



public void setSkeleton(Skeleton s){

skeleton = s;

}



public Skeleton getSkeleton(){

return skeleton;

}



public void setIterations(int i){

iterations = i;

}



public int getIterations(){

return iterations;

}



public void setThreshold(float newThreshold){

threshold = newThreshold;

}



public float getThreshold(){

return threshold;

}



public void setTargetBone(Bone target){

targetBone = target;

}



public Bone getTargetBone(){

return targetBone;

}



public void setFirstBone(Bone first){

firstBone = first;

}



public Bone getFirstBone(){

return firstBone;

}



private void updateBonePositions(Bone bone){

Transform t = new Transform();

for(Bone b : bone.getChildren()){

t = b.getCombinedTransform(bone.getModelSpacePosition(), bone.getModelSpaceRotation());

b.setUserTransformsWorld(t.getTranslation(), b.getModelSpaceRotation());

updateBonePositions(b);

}

}



public void updateBone(Vector3f target, Bone tBone, Bone bone, float tpf, int count){

if(count < 1)

return;



// The position of the bone that we are going to rotate

Vector3f vBone = bone.getModelSpacePosition();



// Stop the calculation if we are closer than threshold

if(target.distance(vBone) < threshold)

return;



// The position of the bone that we want to match to the target

Vector3f vEnd = tBone.getModelSpacePosition();



Vector3f vTarget = target.subtract(vBone);

Vector3f vEndBone = vEnd.subtract(vBone);



Vector3f axis = vEndBone.cross(vTarget);

axis.normalizeLocal();



if(axis.equals(Vector3f.ZERO))

return;



float angle = (float)Math.acos(vEndBone.dot(vTarget)/(vEndBone.length()*vTarget.length()));



// Make sure we don’t rotate by a bad value

if(Float.isNaN(angle))

return;



Quaternion q1 = new Quaternion();

q1.fromAngleAxis(angle, axis);

Quaternion q2 = new Quaternion();

q2.fromAngleAxis(-angle, axis);



Quaternion modelSpaceRotation = bone.getModelSpaceRotation().clone();



bone.setUserTransformsWorld(vBone, modelSpaceRotation.mult(q1));

updateBonePositions(bone);

float dist1 = tBone.getModelSpacePosition().distance(target);

bone.setUserTransformsWorld(vBone, modelSpaceRotation.mult(q2));

updateBonePositions(bone);

float dist2 = tBone.getModelSpacePosition().distance(target);



// We choose the orientation that makes distance smallest (also helps reduce popping for some reason)

if(dist1 < dist2)

bone.setUserTransformsWorld(vBone, modelSpaceRotation.mult(q1));

else

bone.setUserTransformsWorld(vBone, modelSpaceRotation.mult(q2));



updateBonePositions(bone);



// Go up the chain till we reach the root

if(bone.getParent() != null && count < maxChain){

updateBone(target, tBone, bone.getParent(), 1, count + 1);

}

}



public void setTarget(Node node){

target = node;

}



public Node getTarget(){

return target;

}



public void solveIk(){

for(int i = 0; i < skeleton.getBoneCount(); i++)

skeleton.getBone(i).setUserControl(true);



Vector3f targetLocal = spatial.worldToLocal(target.getWorldTranslation(), null);

for(int i = 0; i < iterations; i++){

updateBone(targetLocal, targetBone, firstBone, 1, 1);

}



for(int i = 0; i < skeleton.getBoneCount(); i++)

skeleton.getBone(i).setUserControl(false);

}



@Override

protected void controlUpdate(float tpf) {

solveIk();

}



@Override

protected void controlRender(RenderManager rm, ViewPort vp) {

}



public Control cloneForSpatial(Spatial spatial) {

IkControl control = new IkControl();

control.setSkeleton(skeleton);

control.setSpatial(spatial);

return control;

}

}

[/java]



Here is the code I used for the setup in the previous video.

[java]

ikControl = new IkControl(skeleton);

model.addControl(ikControl);

//ikControl.setEnabled(false);

ikControl.setTarget(targetNode);

ikControl.setFirstBone(skeleton.getBone(“shin.L”));

ikControl.setMaxChain(2);

skeleton.reset();

skeleton.updateWorldVectors();

ikControl.setTargetBone(skeleton.getBone(“heelTarget.L”));

[/java]



Here is how the IK control works:

First you need to set a target for it to attempt to follow. That is the targetNode. The targetBone will attempt to get as close as possible to the targetNode. This means that you might need to add an extra bone to the model, depending on how you want the control to work.



The firstBone is the first actual bone that will be moved in the chain. What this allows us to do is specify a different bone than the targetBone, that will make all the movements to bring the targetBone to the targetNode. The maxChain is the number of bones that can be moved. Once the firstBone has been moved, the Ik control will go to it’s parent bone and move it as long as we haven’t reached the maxChain value.



Also, remember that if you are using this along with animations, you must set the animation controller first, and then the IK control.



If you are having problems using the control, please let me know.





On another note, I’ve started working on a leg control based on the thesis that I posted previously. I can now extract the animation information and find the next footstep position. Next step is to create a dynamic terrain to test the models in, and then to add the IK control on top of the animation info extraction code so that it can try to move the foot to the correct footstep position.

[nehon]

Great.

I’ll test it and report back

[Setekh]

I can play with this thing till the end of time XD

I’m gonna try making him run on terrain.

(hmm image button doesn’t work…)

[staugaard]

@Setekh

Can u make a vid, and throw it on youtube?

[Setekh]

@staugaard said:
@Setekh
Can u make a vid, and throw it on youtube? :)

Unfortunately i do not have the time, so ima post my test case.
To mention, you'll see a lot of snapping, that's cus of 3 things:
- My method of moving the boxes
- No bone rotation constraints
- That skeleton is hating us.

To get better results increse the Iterations.
@nicholas-leehone is just awesome XD

[java]
package ravenclaw;

import com.jme3.animation.AnimChannel;
import com.jme3.animation.AnimControl;
import com.jme3.animation.Bone;
import com.jme3.animation.LoopMode;
import com.jme3.animation.Skeleton;
import com.jme3.animation.SkeletonControl;
import com.jme3.app.FlyCamAppState;
import com.jme3.app.SimpleApplication;
import com.jme3.collision.CollisionResult;
import com.jme3.collision.CollisionResults;
import com.jme3.input.MouseInput;
import com.jme3.input.RawInputListener;
import com.jme3.input.controls.MouseButtonTrigger;
import com.jme3.input.event.JoyAxisEvent;
import com.jme3.input.event.JoyButtonEvent;
import com.jme3.input.event.KeyInputEvent;
import com.jme3.input.event.MouseButtonEvent;
import com.jme3.input.event.MouseMotionEvent;
import com.jme3.input.event.TouchEvent;
import com.jme3.light.AmbientLight;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.Ray;
import com.jme3.math.Vector3f;
import com.jme3.scene.Geometry;
import com.jme3.scene.Node;
import com.jme3.scene.debug.SkeletonDebugger;
import com.raven.character.control.IkControl;
import com.raven.tools.Tools;

/**
* @author Seth
*/
public class IKControlTest extends SimpleApplication {

private final Node prime = new Node("Prime");

/* (non-Javadoc)
* @see com.jme3.app.SimpleApplication#simpleInitApp()
*/
@Override
public void simpleInitApp()
{
flyCam.setMoveSpeed(10);
Node model = (Node) assetManager.loadModel("Models/Sinbad/Sinbad.mesh.xml");
//model.setLocalScale(0.5f);
model.setLocalTranslation(0, 5, 0);
AmbientLight al = new AmbientLight();
al.setColor(ColorRGBA.White);
rootNode.addLight(al);

Skeleton skeleton = model.getControl(SkeletonControl.class).getSkeleton();
IkControl ikControl = new IkControl(skeleton);

rootNode.attachChild(model);

AnimControl control = model.getControl(AnimControl.class);

AnimChannel animChannel = control.createChannel();
// animChannel.setAnim("RunTop");
// animChannel.setLoopMode(LoopMode.Loop);
//
// animChannel = control.createChannel();
// animChannel.setAnim("RunBase");
// animChannel.setLoopMode(LoopMode.Loop);

Node node = new Node("Target L");
Geometry g1 = Tools.createShape(assetManager, ColorRGBA.Red, true);
g1.scale(1.25f);
node.attachChild(g1);
node.setLocalTranslation(1f, 0f, 0);
prime.attachChild(node);

Node node2 = new Node("Target R");
Geometry g2 = Tools.createShape(assetManager, ColorRGBA.Blue, true);
g2.scale(1.25f);
node2.attachChild(g2);
node2.setLocalTranslation(-1f, 0f, 0);
prime.attachChild(node2);

Node nodea = new Node("Target AL");
Geometry ga1 = Tools.createShape(assetManager, ColorRGBA.Red, true);
ga1.scale(1.25f);
nodea.attachChild(ga1);
nodea.setLocalTranslation(1f, 5f, 0);
prime.attachChild(nodea);

Node nodea2 = new Node("Target AR");
Geometry ga2 = Tools.createShape(assetManager, ColorRGBA.Blue, true);
ga2.scale(1.25f);
nodea2.attachChild(ga2);
nodea2.setLocalTranslation(-1f, 5f, 0);
prime.attachChild(nodea2);

model.addControl(ikControl);
ikControl.setTarget(node);
ikControl.setFirstBone(skeleton.getBone("Calf.L"));
ikControl.setMaxChain(2);
ikControl.setIterations(50);
ikControl.setTargetBone(skeleton.getBone("Foot.L"));

ikControl = new IkControl(skeleton);
model.addControl(ikControl);

ikControl.setTarget(node2);
ikControl.setFirstBone(skeleton.getBone("Calf.R"));
ikControl.setMaxChain(2);
ikControl.setIterations(50);

ikControl.setTargetBone(skeleton.getBone("Foot.R"));

// =======================================
ikControl = new IkControl(skeleton);
model.addControl(ikControl);
ikControl.setTarget(nodea);
ikControl.setFirstBone(skeleton.getBone("Humerus.L"));
ikControl.setMaxChain(2);
ikControl.setIterations(50);
ikControl.setTargetBone(skeleton.getBone("Hand.L"));

ikControl = new IkControl(skeleton);
model.addControl(ikControl);

ikControl.setTarget(nodea2);
ikControl.setFirstBone(skeleton.getBone("Humerus.R"));
ikControl.setMaxChain(2);
ikControl.setIterations(50);
ikControl.setTargetBone(skeleton.getBone("Hand.R"));

skeleton.reset();
skeleton.updateWorldVectors();

rootNode.attachChild(prime);

for (Bone bone : skeleton.getRoots())
{
System.out.println(bone);
}

SkeletonDebugger skeletonDebug = new SkeletonDebugger("skeleton", skeleton);
Material mat2 = new Material(getAssetManager(), "Common/MatDefs/Misc/Unshaded.j3md");
mat2.getAdditionalRenderState().setWireframe(true);
mat2.setColor("Color", ColorRGBA.Green);
mat2.getAdditionalRenderState().setDepthTest(false);
skeletonDebug.setMaterial(mat2);
skeletonDebug.setLocalTranslation(model.getLocalTranslation());
rootNode.attachChild(skeletonDebug);

flyCam.setDragToRotate(true);
FlyCamAppState state = stateManager.getState(FlyCamAppState.class);
stateManager.detach(state);
flyCam.registerWithInput(inputManager);

inputManager.deleteMapping("FLYCAM_ZoomIn");
inputManager.deleteMapping("FLYCAM_ZoomOut");
inputManager.deleteMapping("FLYCAM_RotateDrag");
inputManager.addMapping("FLYCAM_RotateDrag", new MouseButtonTrigger(MouseInput.BUTTON_RIGHT));
inputManager.addListener(flyCam, "FLYCAM_RotateDrag");

inputManager.addRawInputListener(new RawInputListener() {

Geometry target;
float distance;

@Override
public void onTouchEvent(TouchEvent evt)
{
}

@Override
public void onMouseMotionEvent(MouseMotionEvent evt)
{
if(target != null) {
Vector3f dir = cam.getWorldCoordinates(inputManager.getCursorPosition(), .9f);

target.getParent().setLocalTranslation(dir);
}
}

@Override
public void onMouseButtonEvent(MouseButtonEvent evt)
{
System.out.println("ress");
if(evt.isPressed() && evt.getButtonIndex() == MouseInput.BUTTON_LEFT) {

Vector3f pos = cam.getWorldCoordinates(inputManager.getCursorPosition(), .0f);
Vector3f dir = cam.getWorldCoordinates(inputManager.getCursorPosition(), .3f);
dir.subtractLocal(pos).normalizeLocal();

CollisionResults rs = new CollisionResults();
prime.collideWith(new Ray(pos, dir), rs);

System.out.println("press");
if(rs.size() > 0) {
CollisionResult result = rs.getClosestCollision();
System.out.println("result");
distance = result.getDistance();
//if(result.getGeometry().getName().startsWith("Target")) {
target = result.getGeometry();
System.out.println("result2");
//}
}
}
else
target = null;
}

@Override
public void onKeyEvent(KeyInputEvent evt)
{
}

@Override
public void onJoyButtonEvent(JoyButtonEvent evt)
{
}

@Override
public void onJoyAxisEvent(JoyAxisEvent evt)
{
}

@Override
public void endInput()
{
}

@Override
public void beginInput()
{
}
});
}

public static void main(String[] args)
{
new IKControlTest().start();
}

}
[/java]

[nicholas_leehone]

Glad to see that someone is making good use of it



I’m still working on getting the unity motion working like in the link I posted in one of the first posts. I’ll show some updates when I resolve some popping issues.

[Setekh]

@nicholas-leehone said:
I'm still working on getting the unity motion working like in the link I posted in one of the first posts. I'll show some updates when I resolve some popping issues.

Glad to hear from you!
Looking forward to see what other goodies your IK control can cook up :)