Hello everyone, I hope the day is treating you all well 
I’ve been working for the past few months on my thesis, a large part of which involves jMonkeyEngine3. Simulated robot vision resulted in a Video capture system for JME3, and now simulated hearing has resulted in an Audio capture system. I thought that it would be good to flesh everything out and give back to jMonkeyEngine3, which has been so fun to work with.
My hope is that with this, it will be possible for jMonkeyEngine3 to support capturing audio and video out of the box. It should be great for demo videos, in-game cut-scenes, and forum posts to show exactly what’s wrong with an application.
audio-send: log
This is a version of OpenAL that allows access to the 3D rendered sound data so it can be saved to a file. It also supports multiple listeners for fancier effects. This is a portable C library with JNI bindings.
More information can be found here: Simulated Sense of Hearing
jmeCapture: log
This is the main capture library in pure java. It provides audio and video capturing. There’s sample code under /examples that shows typical use cases. It needs the xuggle jar files to compile but it does not need them to run.
More information can be found here: Capture Live Video Feeds from JMonkeyEngine
These are a continuation of http://hub.jmonkeyengine.org/groups/contribution-depot-jme3/forum/topic/capture-live-video/. I think it’s much improved since then.
To capture audio:
[java]com.aurellem.capture.Capture.captureAudio(app, audioFile);[/java]
To capture video:
[java]com.aurellem.capture.Capture.captureVideo(app, videoFile);[/java]
The original code had a couple of problems – some effects such as fog wouldn’t record correctly, it relied on Xuggle, and it was difficult to capture the GUInode.
All of these things are fixed now – all effects work correctly, the GUInode is included by default, and there are now two other ways to write video – the open source, pure java AVI video writer by Werner Randelshofer Writing AVI videos in pure Java | Werner Randelshofer's Blog, and a FileVideoRecorder which outputs each frame to a sequentially numbered file. I’ve tested the recording code with all of the demos in jMonkeyEngine3 and there were no problems with either audio or video.
One concern:
In order to support Audio capture, it was necessary to make my own version of OpenAL-soft. This project is contained in audio-send: log above. While the code should compile fine on any system (it’s just openal-soft with some things removed, and one portable C file added), I don’t have the resources to compile it for every OS/architecture that JME3 supports. Is there already a system in place to compile native libraries for all the different platforms?
Please tell me what you think about my new video/audio capture system. I’d like to help in any way that I can to make out-of-the box video/audio capture in JME3 a reality. I’m willing to spend a lot of time to get these projects incorporated into jMonkeyEngine3 if people like them.
sincerely,
–Robert McIntyre
Cool, I suggest making a separate audio renderer for this purpose. Since its never easy syncing the audio samplerate with dynamic video I think it might be easier to go that way instead of presuming that the output occurs in a certain moment according to computations on the passed time. How do you do the syncing now? Do you adjust the audio output later to the video or do you play the audio with varying samplerate? Do you get drift with long sequences?
Nice job.
Almost forgot – there’s two patches to JME3 to use these projects.
[patch]Index: src/desktop/com/jme3/system/JmeSystem.java
===================================================================
— src/desktop/com/jme3/system/JmeSystem.java (revision 8529)
+++ src/desktop/com/jme3/system/JmeSystem.java (working copy)
@@ -299,7 +299,10 @@
clazz = (Class<? extends AudioRenderer>) Class.forName(“com.jme3.audio.lwjgl.LwjglAudioRenderer”);
} else if (settings.getAudioRenderer().startsWith(“JOAL”)) {
clazz = (Class<? extends AudioRenderer>) Class.forName(“com.jme3.audio.joal.JoalAudioRenderer”);
-
} else {<br />
-
} else if (settings.getAudioRenderer().equals("Send")){<br /> -
clazz = (Class<? extends AudioRenderer>) Class.forName("com.aurellem.capture.audio.AudioSendRenderer");<br /> -
}<br /> -
else {<br />
throw new UnsupportedOperationException(
"Unrecognizable audio renderer specified: "
- settings.getAudioRenderer());
[/patch]
[patch]Index: src/desktop/com/jme3/system/Natives.java
===================================================================
— src/desktop/com/jme3/system/Natives.java (revision 8529)
+++ src/desktop/com/jme3/system/Natives.java (working copy)
@@ -122,7 +122,11 @@
extractNativeLib(sysName, name, load, true);
}
- protected static void extractNativeLib(String sysName, String name, boolean load, boolean warning) throws IOException {
- protected static void extractNativeLib(String sysName, String name, boolean load, boolean warning) throws IOException{
-
extractNativeLib(sysName, name, load, warning, false);<br /> - }
+
- protected static void extractNativeLib(String sysName, String name, boolean load, boolean warning, boolean force) throws IOException {
String fullname = System.mapLibraryName(name);
String path = "native/" + sysName + "/" + fullname;
@@ -148,7 +152,7 @@
long sourceLastModified = conn.getLastModified();
// Allow ~1 second range for OSes that only support low precision
-
if (targetLastModified + 1000 > sourceLastModified) {<br />
-
if ((!force) && (targetLastModified + 1000 > sourceLastModified)) {<br />
logger.log(Level.FINE, "Not copying library {0}. Latest already extracted.", fullname);
return;
}
@@ -194,6 +198,7 @@
String audioRenderer = settings.getAudioRenderer();
boolean needLWJGL = false;
boolean needOAL = false;
-
boolean needAudioSend = false;<br />
boolean needJInput = false;
boolean needNativeBullet = isUsingNativeBullet();
if (renderer != null) {
@@ -206,6 +211,10 @@
needLWJGL = true;
needOAL = true;
}
-
else if (audioRenderer.equals("Send")) {<br /> -
needLWJGL = true;<br /> -
needAudioSend = true;<br /> -
}<br />
}
needJInput = settings.useJoysticks();
@@ -223,8 +232,11 @@
extractNativeLib("windows", "lwjgl64");
}
if (needOAL) {
-
extractNativeLib("windows", "OpenAL64");<br />
-
extractNativeLib("windows", "OpenAL64", false, true, true);<br />
}
-
if (needAudioSend){<br /> -
extractNativeLib("windows/audioSend", "OpenAL64", true, true, true);<br /> -
}<br />
if (needJInput) {
extractNativeLib("windows", "jinput-dx8_64");
extractNativeLib("windows", "jinput-raw_64");
@@ -238,8 +250,11 @@
extractNativeLib("windows", "lwjgl");
}
if (needOAL) {
-
extractNativeLib("windows", "OpenAL32");<br />
-
extractNativeLib("windows", "OpenAL32", false, true, true);<br />
}
-
if (needAudioSend){<br /> -
extractNativeLib("windows/audioSend", "OpenAL32", true, true, true);<br /> -
}<br />
if (needJInput) {
extractNativeLib("windows", "jinput-dx8");
extractNativeLib("windows", "jinput-raw");
@@ -256,8 +271,11 @@
extractNativeLib("linux", "jinput-linux64");
}
if (needOAL) {
-
extractNativeLib("linux", "openal64");<br />
-
extractNativeLib("linux", "openal64", false, true, true);<br />
}
-
if (needAudioSend){<br /> -
extractNativeLib("linux/audioSend", "openal64", true, true, true);<br /> -
}<br />
if (needNativeBullet) {
extractNativeLib("linux", "bulletjme", true, false);
}
@@ -270,8 +288,11 @@
extractNativeLib("linux", "jinput-linux");
}
if (needOAL) {
-
extractNativeLib("linux", "openal");<br />
-
extractNativeLib("linux", "openal", false, true, true);<br />
}
-
if (needAudioSend){<br /> -
extractNativeLib("linux/audioSend", "openal", true, true, true);<br /> -
}<br />
if (needNativeBullet) {
extractNativeLib("linux", "bulletjme", true, false);
}
@@ -285,6 +306,9 @@
}
// if (needOAL)
// extractNativeLib("macosx", "openal");
-
if (needAudioSend){<br /> -
extractNativeLib("macosx/audioSend", "openal", true, true, true);<br /> -
}<br />
if (needJInput) {
extractNativeLib("macosx", "jinput-osx");
}
[/patch]
sincerely,
–Robert McIntyre
@normen
In order to perfectly line up the audio with the video, I use the IsoTimer class. It’s a replacement for NanoTimer for use during recording. IsoTimer always reports that the same interval of time has passed, regardless of how much real time has actually passed since it was last called. As long as the frame rate evenly divides 44100 (sample rate of audio), then each video frame will be exactly lined up with (/ 44100 frames-per-second) sound samples. I normally record at 60fps and get exactly 735 audio samples for every video frame. When I’m done recording I have a video and a wave file, and I just have to mux them together and they are in perfect sync.
It may be a good idea for IsoTimer to warn that the audio and video won’t line up if it is created with a frame-rate that doesn’t evenly divide 44,100. I think I will add that to prevent problems.
thanks,
–Robert McIntyre
I see. Well cool, then I’d suggest just adding a set of default output settings that work well (like 25fps/30fps with 44.100Hz/48.000Hz etc.). For the SDK a “Development Application Harness” to facilitate application-SDK integration is planned anyway, it could easily incorporate a switch for this video output renderer.
That would be neat to have an something in the SDK that amounts to “easy record your app.”
So, what is the next step for possibly getting this into jMonkeyEngine3? I’m afraid I’m a little fuzzy on the process here. For something of this size, what would be appropriate?
sincerely,
–Robert McIntyre
We’d make it a plugin for the SDK, it contains all the jars etc needed… It would not be part of the “core” engine I guess, because as you say its probably a bit oversized for that.
Its probably best to make it a plugin yeah. Is there any other formats supported beside AVI? Can it generate WebM files or OGM with Theora video for example? Since people might distribute apps with this library, it is best not to include any patent encumbered codecs (MPEG4, H264, etc)
It can make anything that Xuggle can, including mkv, ogg, flv, etc.
The other two writers are fallbacks if Xuggle is missing. They don’t include any patent encumbered codecs, so there should be no problems with including the library in any apps. It does not require the xuggle jar files to function.
The simplest and most compatible option is to use the FileVideoRecorder class and output a sequence of PNG files, which can be encoded into whatever format is desired.
Since it outputs audio and video as separate files, some amount of post processing is necessary to make the final video.
awesome nice work! thanks
I’ve been reading the documentation for creating a jMonkeyEngine3 SDK plug-in and have four questions:
1.) Is creating the plug-in something I would do myself and then make available here, or is it something that a core developer would make based on the three library jars?
2.) How can I make sure my native code gets compiled for all the platforms that JME supports? I see something that looks like a multi-platform automated build system here: https://www.newdawnsoftware.com/jenkins/. Is this part of JME/is it an appropriate place for my native code?
3.) If the recording system is released as a SDK plug-in, how will do people who don’t use the SDK get access to it?
4.) Does it make sense to make it a plug-in, or would it be better to just provide the jars somewhere, possibly distributed with JME, along with lots of documentation on the wiki?
- Both is possible really but I’d say its easier when you have full control over your software
- No, theres no such thing
- The same way the plugin repository build process gets them, so either via svn or download
- A plugin would contain the jar files so they can be added to projects, for other users see above, they can download the jar just as well
Thanks for your clarifications — I’ll work on making plug-in for the SDK.
How does jMonkeyEngine3 produce the jME3-lwjgl-natives.jar file if there’s no automatic way to compile native code for multiple systems?
sincerely,
–Robert McIntyre
I’ve created a plug-in for the SDK and am ready to contribute it.
The plug-in is just a wrapper for the two jar files that comprise the project, which are also available here:
error-daemon@aurellem:
error-daemon@aurellem:
I used amazon EC2 to compile the GNU/Linux 32 bit and Windows 64 and 32 bit versions of this library. Unfortunately, I don’t have access to a mac, so I could not make native binaries for that platform.
If anyone wants native binaries for mac, please download the code with
hg clone http://hg.bortreb.com/audio-send
and follow the instructions in the README. If someone sends me the mac build artifact I'll include it in the plug-in.
- ☉ Can I get access to the relevant Google code project to contribute the plug-in?
- ☉ Are the patches above which are necessary to support the plug-in OK?
- ☉ Anyone interested in compiling this code on their mac (I'll help if there are any problems)?
sincerely,
--Robert McIntyre
The natives extraction should be done in the audio renderer… Which in turn should be able to be set via the appSettings… If it cannot be set via the class name we’ll add that ability instead, for the project access, just PM me your googlecode mail. Btw, theres a new VideoRecorderAppState in the core engine that uses java only to record avi/m-jpeg video files that are working quite nicely, based on your idea actually. Maybe worth looking into for extending to a complete java-based audio/video encoding system.
I made a patch to Natives.java to support unpacking the modified version of OpenAL from the AudioRenderer.
It gives me everything I need to replace OpenAL from the audio renderer without duplicating any code. It should also enable other modules to unpack their own natives. I had to add the file-length check to support loading a library with the same name as another library. I also changed the un-paramaterized reference to Class while I was there so that Natives compiles without any warnings. That part is not necessary and can be removed if desired.
I figured I’d post it here for completeness. What do people think?
[patch]Index: src/desktop/com/jme3/system/Natives.java
===================================================================
— src/desktop/com/jme3/system/Natives.java (revision 8731)
+++ src/desktop/com/jme3/system/Natives.java (working copy)
@@ -45,7 +45,6 @@
/**
- Helper class for extracting the natives (dll, so) from the jars.
-
- This class should only be used internally.
*/
public class Natives {
@@ -114,15 +113,19 @@
}
}
- This class should only be used internally.
- protected static void extractNativeLib(String sysName, String name) throws IOException {
- public static void extractNativeLib(String sysName, String name) throws IOException {
extractNativeLib(sysName, name, false, true);
}
- protected static void extractNativeLib(String sysName, String name, boolean load) throws IOException {
- public static void extractNativeLib(String sysName, String name, boolean load) throws IOException {
extractNativeLib(sysName, name, load, true);
}
- protected static void extractNativeLib(String sysName, String name, boolean load, boolean warning) throws IOException {
+
- public static void extractNativeLib(String sysName, String name, boolean load, boolean warning) throws IOException{
+
String fullname = System.mapLibraryName(name);
String path = "native/" + sysName + "/" + fullname;
@@ -142,14 +145,15 @@
try {
if (targetFile.exists()) {
-
// OK, compare last modified date of this file to<br /> -
// file in jar<br />
-
// OK, compare last modified date and size of this file to file in jar.<br />
long targetLastModified = targetFile.lastModified();
long sourceLastModified = conn.getLastModified();
-
-
long targetLength = targetFile.length();<br /> -
long sourceLength = conn.getContentLength();<br />
// Allow ~1 second range for OSes that only support low precision
-
if (targetLastModified + 1000 > sourceLastModified) {<br /> -
logger.log(Level.FINE, "Not copying library {0}. Latest already extracted.", fullname);<br />
+
-
if ((targetLength == sourceLength) && (targetLastModified + 1000 > sourceLastModified)) {<br /> -
logger.log(Level.FINE, "Not copying library {0}. Latest already extracted.", fullname);<br />
return;
}
}
@@ -182,7 +186,7 @@
protected static boolean isUsingNativeBullet() {
try {
-
Class clazz = Class.forName("com.jme3.bullet.util.NativeMeshUtil");<br />
-
Class<?> clazz = Class.forName("com.jme3.bullet.util.NativeMeshUtil");<br />
return clazz != null;
} catch (ClassNotFoundException ex) {
return false;
[/patch]
Does the byte-length check as the sole factor of differentiation concern anyone but me? I have a feeling that if someone is using two libraries with the same name in a project then the differences are probably minor. Minor, like, it could be a fix for a typo or something… which would have a reasonable chance of being archived to something with the same byte-length as the original.
Hashing would get around this possibility but would also slow things down quite a bit… This could also be my overly-paranoid side acting up again
I think the changes are fine.
I checked out revision 8846 of jMonkeyEngine from SVN and see that the Natives method extractNativeLib has been made public. However, there’s no checking for the length of the file, which means that it’s not possible for a module to load a replacement for a native library using this method (because whichever one is newest in the jar will always “win” over the other one). Is this an oversight?
sincerely,
–Robert McIntyre