Hi,
I have been trying for days to manipulate a character using information from a Kinect device. I keep running into problems though, especially with child bones. My method is to find the bone vectors in Kinect, then set the rotation using the setUserTransforms function.
For example, if Bone A is a parent of Bone B, which is a parent of Bone C, I can manipulate Bone B successfully. Firstly I find the quaternion between the initial bone vector of B and the Kinect bone vector of B (called q1). Then I find the appropriate rotation by multiplying the inverse of the initial rotation of B by q1 and then multiplying this by the inverse of initial rotation of A (B’s parent).
This works for bones which are attached to the root joint. However, I cannot figure out how I can extend this to work for Bone C. I have tried everything. Can anyone point me in the right direction for this? Clarification of the meaning of localRotation, worldBindInverseRotation and modelSpaceRotation will also be useful as I am struggling to work out what they represent.
Thanks in advance.
Hey man, actually I’m working on kinect implementation for jme :). I’m using OpenNI. What’s the libraly are you using?
I’m just using the Kinect SDK. Basically I think that I’m missing something to do with the rotations, but I don’t know what. I figured out that localRotation includes the initial rotation of the bone, so I should inverse it to get the “true rotation” from the parent? But then I have to use the absolute (I think) parent rotation somehow. Quaternions make me dizzy…
@deev, its not just about rotation. Think of this: when you attach a child object to a parent and make it have an offset in location then you will not just rotate but also move the child object (think of the moon rotating around the earth). jME3 provides a combined class for rotation and location called the “Transform”. This also has a method .combineWithParent() which might help you here. Also make sure you check out these, they explain the jme3 scene graph and math in a simple way that should help you get along with quaternions:
https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:scenegraph_for_dummies
https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:math_for_dummies
I’ve created a motion capture control based on KinematicRagdollControll and AnimControl, and I just store the initial translation/rotation from each bone. If you look at the KinematicRagdollControll you’ll see a class called PhysicsBoneLink. It stores the informations of each bone.
normen:
Thanks for the advice. Is there an example of an application using the Transform code? I think this is what I am looking for. I know that a rotation of a parent will translate the child and that this must affect the rotation of the child as the coordinates will change (I think). Putting this information together is still a little tricky. So far I’ve been experimenting heavily with only modifying the local rotation of the bone, but maybe using Transform is more suitable.
glaucomardano
Yeah, I tried implementing a similar class which stored the initial values of each bone. I need this information to compare with the new vector I receive from Kinect and figure out which direction to rotate.
The examples are probably more in the jme3 code (e.g. CollisionShapeFactory, Spatial)… Still, with the methods on Transform, the javadoc and maybe some testing using Arrow meshes you should get it done 
Thanks, I’ll give it a crack tomorrow.
Is using the setUserTransforms function the right idea? I’m assuming that I use the Transform class to calculate the necessary rotation/translation, then simply use this information as a parameter into the function.
Yep Tho creating an animation from the data would also be awesome 
Argh, I thought I could do this, but now I’m completely lost. I get the concept of quaternions and axis-angle representation. But I cannot for the life of me work out how to apply the correct rotations or how combineWithParent() can help with this. I have to use the parent rotations and translations somehow. Converting and applying world rotations from the input data to the local bone rotations seems to be the way to go. I’m guessing the Transform class can help me here.
If anyone can give a simple example step-by-step I’d be really appreciative. For example, I have a model with two bones and I receive two bone vectors as data. How do I rotate these bones correctly?
Look at the class RagdollUtils and search for “combineWithParent”. See how the KinematicRagdollControl uses the RagdollUtils to transform bones at the update method.
OK, I finally worked out how it’s done, so I’m going to post my solution on the chance that someone else is looking to do the same thing.
I have a character model I created in Blender with which I want to control by using data I get from Kinect. The basic problem is to map the bones (represented by vectors) from Kinect to the character model. Of course, it is assumed that the bones in the character model can be matched to the ones in the Kinect skeletal tracker.
- Assign each vector from Kinect to the relevant bone in the character model. There are many ways to do this, I just created a ModelBone class which stores the parent and child joint. The most important thing to understand is terminology. In jMonkey, “bones” are actually points in 3D space with a rotation and translation relative to the parent bone. However, I prefer to think of them as “joints”, and the vector between two of them as a ModelBone.
For example, in my model I have a shoulder Bone and an elbow Bone. Therefore, my ModelBone is simply information about the vector connecting the shoulder and elbow. The name of this ModelBone could be seen as “upper arm” or something similar.
[java]X
X
Bone shoulder ModelBone upperarm Bone elbow
(joint shoulder) (joint elbow)[/java]
I will define the jMonkey Bone objects as joints and the connecting vectors as ModelBones for clarity.
2. Define the initial ModelBone vector, which is simply the initial position of the child joint. This is because the position of a child joint is relative.
Now comes the maths. Because the jMonkey system already moves the joints when the parent joint is moved, there is no need to calculate the translation. You are only interested in the rotation of the joints. To find this rotation, you need to find the quaternion between the initial ModelBone vector and the vector you receive from Kinect. Of course, this isn't so simple because the vector you receive from Kinect will be in its own coordinate system, whereas the ModelBone vector will have its own rotated system according to the parent joint.
3. To calculate the desired quaternion, you must rotate the Kinect vector so that it is in the correct co-ordinate space of the ModelBone parent joint. To do this, you simply multiply the inverses of each quaternion rotation of each parent in the joint hierarchy, starting from the child until you reach the parent, then use this combined rotation and multiply it by the original vector from Kinect.
e.g.
I have four joints, A, --> B ---> C ---> D in my hierarchy. I know the initial ModelBone vector between C and D. Now I receive a vector from Kinect that corresponds to C and D. To transform this vector to the ModelBone vector, I multiply the inverse of the initial rotation of C by the inverse of the local rotation of B by the inverse of the local rotation of A by the vector.
[java]transformed Vector = (C's initial rotation inversed * B's local rotation inversed * A's local rotation inversed * Kinect vector[/java]
This is easily achieved with a recursive function.
4. Now that you have a transformed vector, simply find the quaternion between the initial ModelBone vector and the transformed vector. You can do this using a cross product and dot product of the vectors.I won't go into the details of how to do this, a Google search should give you the answer fairly easily. Note that the initial ModelBone vector will always be the same.
5. This output quaternion is the rotation with which you rotate the ModelBone parent joint using the setUserTransforms function. If it all goes well, it will rotate to the vector in the Kinect world space. That is, if the Kinect vector is straight along the x-axis, the output of the model should be the same, regardless of any rotations in the parent joint.
Remember to always rotate the parent joints before the child joints
This method has worked for me quite well and I can use it on a number of models with no problems. Hopefully this explanation is useful for some people. If there are any questions/advice feel free to post.
Nice explanation :D.
Do you ask for a tutorial or the source code xD.
both of them will be better,
But,first I’ll try first, and now I am still configure my kinect on my laptop, I just got kinect from my brother,hohoho
What libralies will you use?
No man, jkinect no 8). It’s a homemade kinect jara wrapper. The OpenNI itself has a java wrapper built in. In my kinect game template there’s a README that teaches how to install the openni, nite and its driver (sensor kinect). OpenNI is the most complete libraly for kinect, it supports a lot of features. Btw tomorrow I’ll take a look at my mocap control, and if I get to solve that I’ll let u know.
Ok,thx a lot, I’m still on my progress