Adding user interaction (hand gestures)

Hello all, I’m a computer scientist/former computer game enthusiast, i recently stumbled across jMonkeyEngine and have been slowly working through the examples trying to get a good comprehension of how it all fits together…seems that there are several different features that jMonkeyEngine incorporates (openGL, bullet physics, ogre, etc). anyhow, firstly i’d like to thank all the people who put work into the project, and whoever made the tutorials, they’re some of the best tutorials i’ve seen for any application, great explanations and full source code examples that you can just copy/paste and have working in seconds, so thanks for that.

now on to my question - one of my favorite games as a kid was dungeon keeper, i liked it mainly for its play style - for those who don’t know, it was more of a “management type game” where you don’t really micromanage your minions, but rather, nudge them in the right direction with slaps and picking them up/dropping them into the battle, etc…then the creature’s ai would usually take care of the rest. the game is pretty much dead these days (came out in like 1995), but i’d like to revive/reimplement it i jmonkey, only with an added twist. in the original game, all actions were implemented with a mouse and keyboard, obviously. if you wanted to pick up a creature, you left clicked on it. if you wanted to slap it to make it work faster, you right clicked. what i want to do is take in data from a webcam about a user’s hand position, and feed that information directly into my game - so instead of using mouse clicks, i want to directly use the user’s body movements, hands in particular, so you’ll be able to slap your creatures with your hand gestures, and pick them up with pinches, etc, instead of using the mouse.

there are tons of image processing libraries in java, javacv for example has wrappers for all the major libraries out there i believe (OpenCV, FFmpeg, PGR, etc) BUT i was wondering if i will be able to use these in conjunction with jmonkeyEngine, and what would be the best way to go about it. I think i want to model the hand as an actual game character (use ogre for this??) and then use that to affect the game in various ways.

also, i was wondering if jMonkeyEngine itself supports any type of image processing … i basically just need access to the raw pixel data coming in from the webcam and i can handle the rest myself… if this was directly accessible through jMonkeyEngine that would be preferable I think. but maybe not, if the image processing libraries are easily integrable with jmonkeyEngine. i can get the webcam feed in the form of a bufferedImage quite easily, would this be transferable to my game in jMonkeyEngine?

thanks,

Russell

If you can get byte buffer with image, or awt Image, you can use it as texture in jme3 without big problems. But this will only help if you want to embed webcam image inside 3d scene, not with image recognition. There is nothing in jme3 which will help you directly with understanding the image contents.

I would suggest taking a look at
http://www.movesinstitute.org/~kolsch/HandVu/HandVu.html
and
http://www.nickgillian.com/wiki/pmwiki.php?n=GRT.GestureRecognitionToolkit

Get one of them working in C++ for test app and after that do a port to java. At this point you won’t need to pass image data to jme3 anymore and will communicate in high-level gesture information. And then we can start getting into actual 3d programming :wink:

And check out “war for the overworld”. It’s a dungeon keeper clone endorsed by the original guy who designed dungeon keeper thats being developed now.

cool, thanks for the replies… i’ll check those sites out.