Animation retargeting, mocap BVH

Development
1

I’m working on animation retargeting lately, and I finally achieved to have a correct result retargeting BVH animations.
For those who don’t know what BVH is, it’s an open file format that can hold skeleton and animation data.http://research.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/BVH.html
The thing with bvh is that there are tons of free repo with animations that you are free to use.
CMU (the most popular, stands for Carnegie Melon University) Carnegie Mellon University - CMU Graphics Lab - motion capture library
The trailers’ park http://www.thetrailerspark.com/downloads/Details/CMU/Library/BVH/
ACCAD http://accad.osu.edu/research/mocap/mocap_data.htm
there are plenty others, just search on google
Here is the incarnation of grace, Ballerina Sinbad
http://youtu.be/r6MJTyZKF88

It’s still a work in progress, it’s not completely perfect and you have to tweak the bones mapping a bit to have correct results…but you can have correct results so I guess that’s a start.

24 Likes
2

Fabulous xD

1 Like
3

Nice stuff! Can you give a little more details?

4

Great! Is the performance the same as the usual animations? How long does it take to retarget the animations? Also, if I have a bunch of different models with the same bone structure to use the same animations, will it only have the memory impact of 1 animation loaded, or one copy for each different model, like it is currently?

5
@8Keep123 said: Great! Is the performance the same as the usual animations?
That's usual animation. Retargeting creates a new animation for the target skeleton with the source animation data and skeleton.
@8Keep123 said: How long does it take to retarget the animations?
Between the blink of an eye and a heartbeat, I didn't really test. It's mostly meant to be an initialization thing, you won't retarget on each frame so I guess performance is fast enough. I can run some perf test for the sport though...
@8Keep123 said: Also, if I have a bunch of different models with the same bone structure to use the same animations, will it only have the memory impact of 1 animation loaded, or one copy for each different model, like it is currently?
Well..it doesn't really work like that. The idea behind retargeting is that you can't apply the same animation data to different skeleton (unless they are exactly the same, meaning structure, bone indices, scale and proportion). So when you retarget, visually the animation is the same, but the animation data is different, so you need one animation data set for each skeleton. For the particular case of strictly identical skeletons, things could be done to share animation data, but, that's unrelated to retargeting.
@TsrKanal said: Nice stuff! Can you give a little more details?
Sure, what do you want to know? :p For now, retargeting works only for rotation and scale, I still have trouble with bone positions. That's why Sinbad stays in place in the video. Also it only works with skeleton with structures that are close to each other. For example if the source skeleton has 2 bones where the target has one, it doesn't work.

Also for rotation there are bone twisting issues. Computed rotation sometime end up in a limb being rotated for PI or PI/2 on one axis. I’ve seen other retargeting softwares having the very same issue, and offering a manual way to fix it. So that’s what I did for now. I think this issue is somehow predictable with math magic, so maybe at some point I’ll find out and users will have less manual configuration on the bone mapping to properly retarget.

I plan to make the algorithm work with most common BVH sources (the one I linked in the fist post), at first. Then will see if other sources can be integrated.

I’d like to make some UI in the SDK so users could import a BVH anim and retarget it to their character skeleton with a view on both skeleton and previewing animation. This will come in handy for the upcoming Cinematic editor, because one will be able to attach more usual animations to a model (I mean other than kick, slash, die, jump :p) thant could be useful in a cinematic.

Still a lot to do :stuck_out_tongue:

1 Like
6

This is so cool.

7
@nehon said: Also it only works with skeleton with structures that are close to each other. For example if the source skeleton has 2 bones where the target has one, it doesn't work.

How do you make the mapping between the source and the target skeleton - manual configuration? You said that there is a problem if e.g. the source skeleton has 2 bones and the target has one - would it work the other way around?

I tried something similar where I wrote a small tool to “normalize” the skeleton (give it the same names) and also the animation. With the intent that I had a few animated commercial models and I wanted to reuse every animation for all the models. But I was not really satisfied with the result because most animations went just weird in some parts… I do not really know much about BVH, apart that its a file format for animations. But is the structure of the skeleton in some way given, so I can use all the BVH animation for my “standardized” skeleton or do I need to create a separte skeleton for every specific animation.

Btw - thx for sharing!

8

Nice!

I’ve done a bit of dabbling with this, in blender.
I found bvhacker to be a really nice tool when trying to create walk cycles etc, since the bvh data can often be very raw on its own.

9

I’m not fond of soccer, but thought it was the good time for this

@TsrKanal bvh files can have different skeletons, there is no “standard” skeleton.
Some have 2 bones for each “classic” bone you’d have in a skeleton, like the forearm has a foreArm and a foreArmRoll (used to allow smooth deformation of the mesh when the hand is rotating and not the elbow for example). in that case the current algorithm doesn’t work, because you have to match 2 source bones with one target bone and somehow combine their transforms. But I’ll add this eventually.

For now you have to feed the anim retarget algorithm with a bone mapping. That’s pretty much mapping bone names from the source skeleton to the target skeleton.

6 Likes
10

Does this just mirror the keyframes to different track/bone names or bake it into the mesh? (there’s very little difference thanks to how the animation system is structured)

Either way, this is very good going considering one of the biggest selling points of Unity 4 was some crappy animation system (which focused heavily on retargeting)
Do you plan to implement features such as name template identification, bone layout identification (automatic characterization), restrictions (e.g. lock in place, rotation locking on base), sequence loop helper? All of these would simply slaughter Unity 4 for animation (the other stuff isn’t really required for most things except dynamic parkour for example).

11
@8Keep123 said: Great! Is the performance the same as the usual animations? How long does it take to retarget the animations? Also, if I have a bunch of different models with the same bone structure to use the same animations, will it only have the memory impact of 1 animation loaded, or one copy for each different model, like it is currently?

I have some code around to use only one animation for several models using the same base skeleton setup. Drop me a line if you are interrested.

1 Like
12

Oh, I completely missed this! Very awesome @nehon! =D

13

wow!!. This stuff will make my life easy :slight_smile:

14
@rickard said: I found bvhacker to be a really nice tool when trying to create walk cycles etc, since the bvh data can often be very raw on its own.
It's nice but it only works under Windows. For others, there are bioviewer and BVH-Motion-Creator.
1 Like
15

Heads up on this, I’ve set up a git repo with my test project.
It’s still in a WIP/mess/draft state, but if you are interested here is the link

6 Likes
16

Ah and how I am interested :slight_smile:

17

Sorry for necro, but I think this thread deserves the attention. Free bvh’s are great and importing them to jme directly doubly so! Thank you nehon!

I tried bvh’s from CMU converted by cgspeed (3dMax-friendly version https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture)

The skeleton imported well except for a minor rotation issue. You see, all the test data you have used had rotation order “Zrotation Xrotation Yrotation”, which was also hard coded in BVHLoader. This was not the case for the bvh I downloaded. The fix was simple, just multiply in order as presented in file, so I made a PR.

Considering the small size of the library and the benefit it provides, I think it should be included by default with jme (either core or standalone).

4 Likes
19

I’ve experimented with using BVH to develop animations for JME models. It’s quite doable, but usually some fixup is required to achieve a good result.

I’ll look into the rotation-order issue.