Thanks to @Empire_Phoenix, @abies and @Pesegato I now have a movie playing on the screen! But I am now stuck with sound. I wrote my own movie decoder (no JAVAFX etc) and my choice of AudioRenderer is JME, I think. The problem is the audio playing logic. Audio simply isn’t there always, if we lag behind on the movie etc.
So I went ahead and used regular AudioNode with AudioStream using my MovieInputStream and blocking read() like it says in the JAVADOC. I tried to use PipedInputStream & Output earlier. As the audio frames come from another thread. But both of these ways where not successful. Blocking block the whole application. And stuttering happened on PipedInputStream trials.
What would be the best way to provide audio stream to my movie in JME? Any suggestions?
2 Likes
Hm the video is created by you?
Then its simple:
Add in the video outside the visible are one pixel that blinks every X frames.
Add into the audio a signal at like 50khz (so non human hearable) at the same intervall.
Now you have enough information to sync it properly. By reducing/increasing audio file speed accordingly.
You could also try just storing the aduio a s a wav seperatly.
No, the video is actually from Dungeon Keeper II, playing it for our remake. The biggest video is quite big, so I use “clever” buffering and not expose the whole set of frames at once. This is my interface currently: https://github.com/tonihele/OpenKeeper/blob/feature-31/src/toniarts/openkeeper/video/TgqPlayer.java. I have no experience on creating video player or audio syncing. Currently I’m just starting the video & sound ~simultaneously and keeping the audio data one frame ahead. But I have not yet found a way to play the sound as stated in my original question. I mean, I have played it successfully as AudioBuffer with the full stream.
And here it is in action (don’t mind the artifacts nor the music playing in the background):
Typically audio / video synchronization is done by picking a clock generator and then synchronizing either audio, video, or both to it.
If there’s audio then most players will simply use audio as the clock generator. The currently playing audio position is used to determine the frame that should be displayed.
Because jME3 does quite a bit of buffering on audio streams you will not succeed in determining the audio position from how far jME3 has read from your audio stream. You need to instead determine the exact playback position, which requires intervention from OpenAL …
Now I’m ~syncing against the video, well, making sure that the audio doesn’t get too far ahead. This would also suit my needs. Of course doing it the right way would also be nice. But I’m not able to execute either one of these options. Can you give me some more pointers…? Even if I manage to sync by the audio with the positions and all, how to play it non-blocking, to wait for the next frame…?
Ok, from what I understood I need to access OpenAL myself. OpenAL instance at the point I’m using it, is already created by JME, that I figured out. However I cannot get any sound out, neither any errors. Is this even remotely correct:
private static final int AUDIO_SOURCE = 0;
private static final int BUFFER_COUNT = 30;
private IntBuffer bufferNames;
private int remainingBufferCount;
bufferNames = BufferUtils.createIntBuffer(BUFFER_COUNT);
AL10.alGenBuffers(bufferNames);
AL10.alSourcei(AUDIO_SOURCE, AL10.AL_LOOPING, AL10.AL_FALSE);
AL10.alSourcef(AUDIO_SOURCE, AL10.AL_PITCH, 1.0f);
AL10.alSourcef(AUDIO_SOURCE, AL10.AL_GAIN, 1.0f);
AL10.alSourcei(AUDIO_SOURCE, AL10.AL_SOURCE_RELATIVE, AL10.AL_TRUE);
AL10.alSource3f(AUDIO_SOURCE, AL10.AL_POSITION, 0, 0, 0);
AL10.alSource3f(AUDIO_SOURCE, AL10.AL_VELOCITY, 0, 0, 0);
remainingBufferCount = BUFFER_COUNT;
for (int i = 0; i < BUFFER_COUNT; i++) {
stream(bufferNames.get(i));
}
AL10.alSourceQueueBuffers(AUDIO_SOURCE, bufferNames);
AL10.alSourcePlay(AUDIO_SOURCE);
private void stream(int bufferId) {
int format = audioHeader.getNumberOfChannels() > 1 ? AL10.AL_FORMAT_STEREO16 : AL10.AL_FORMAT_MONO16;
AL10.alBufferData(bufferId, format, audioFrames.poll().getPcm(), audioHeader.getSampleRate());
}
As I mentioned earlier, you have to know the exact playback position of the audio. Without this information, synchronizing video and audio is impossible.
Synchronizing audio to video is a bad idea, because changing audio playback position will cause noticeable audio glitches which will make watching the video unbearable.
In OpenAL, a source’s playback position can be retrieved by multiplying the number of unqueued buffers by the buffer size and then adding the source’s AL_SAMPLE_OFFSET to it. If the OpenAL extension AL_SOFT_source_latency is available, you can also add the source’s latency to that value.
Yes, I’m trying now to play the audio, without ever changing its position as you suggested. Instead probing its playback position and syncing the video frame there. That piece is just a simple test to play the first 30 frames of audio straight up.
But I have problems getting any sound from the OpenAL. I can’t hear anything with the code above, that is all the sound related code now. Is there something wrong with that? That code is run in a separate thread invoked by an AppState.
Do I need to specify the device somehow or something? I tried to look how JME does this (LWJGLAudioRenderer in JME 3.0) and also follow the OpenAL lwjgl tutorials. But I just can’t figure this one out.
It might be a bit awkward, but maybee working,
Have you considered, using the old Java audio system instead for the videos,
as far as I know you can better controll the playback speed ect. there.
(It has not much other features, but when the original videos are your source I guess you kinda only need synchable monotone sound or?)
Yes, I actually considered it once I saw the OpenAL interface 
If I understood correctly they are still part of standard JRE since I don’t need JMF. So there would be no drawbacks… right?
All I need is something capable of playing out my samples, standard PCM audio (stereo 16-bit 22500hz, if I remember correctly, applies to all). Streaming and accurate positioning of course on top.
Java Sound would work here, probably better than OpenAL…
If you’re just trying to get sound out without trying to integrate with the engine, that would be the best approach, as getting playback position from Java Sound is much easier.
Looks like its your lucky day 
I just pushed a new method on AudioSource that lets you get the playback time, which works for both streaming and buffer sources.
Only thing is, I noticed it doesn’t update very often, maybe 30 times per second or so. That should be enough for video synchronization though.
https://github.com/jMonkeyEngine/jmonkeyengine/commit/7393f7916579cf26b7bbdc060756e37bc2c6d61f
1 Like
Great! Thank you both! “My” videos run 15 FPS, so at least for me that wouldn’t be a problem I recon. Definitely a great feature to have.
JCodec has this going on using Java Sound. I’ll try to mimic this. Shouldn’t be too hard with the working source code example.
If I get this to work, I’m never doing this again Just embed FFMPEG, or of course with “normal” video formats the JavaFX road would probably be nice. But all native JAVA & little bit hardware acceleration (the luminance-crominance → RBG) is awesome. Thanks guys.
Trust me you don’t want to embedd FFMPEG in java,
as you need like 6 different version depending on OS ect. And the java wrappers are not for all compiled, dependency problems yadda yadda.
For my Masterthesis I resorted then to Processbuilder and linux only…
Yeah, actually FFMPEG couldn’t even pull this one off it seems. The format specification says 15 FPS, but while syncing the audio it became clear that this particular piece of work uses 25 FPS. Thanks you all once again, and I hope this also says expresses my gratitude:
1 Like
well what works is counting the frames, determining the lenght of the audio signal and then forcing the correct -r rate.
But anyway a pure java decoder is way more flexible.
