LipSync / Sound amplitude/data listener

Hello,

i was curious, how could i do lipSync in JME.

seen tools like Annosoft Lipsync tool, but need something simple and free.(it dont need to do exactly how human do)

What i was thinking was doing some simple amplitude or other sound line data to know height/width of mouth to make.

is this maybe easly possible with AudioNode(getAudioData()) class or some other to gather this informations?
or maybe you know some better solution for simple lipsync?

thanks for any help!

For anyone who will look how to do it. Here simple free solution!

Found a nice app, that is generally under MIT.

It works on both Windows and Linux and seems like work very well.

You just do(Windows): .\rhubarb.exe -o output.txt .\tmp\guess-what-2.wav
or(Linux) ./rhubarb -o output.txt ./tmp/guess-what-2.wav

and you get letters where each letter mean some mouth shape(described on github page)
example:

0.00 X
0.05 B
0.19 D
0.40 B
0.54 F
1.03 B
1.31 X
1.46 X

so later just do some code that during sound play, will also change mouth shape keys or change bones of mouth to match letter shape described in tool documentation.

because X is not X letter, its some shape, it mean closed mouth.

here github:

4 Likes