Hi fellows,
I’ve noticed that there is actually no support for video playback. A few months ago i developed a solution for jME2 based on
Java Media Components. (Part of JavaFX… Unfortunately the JMC project doesn’t seem to become part of the JDK)
Although my old implementation was weak and slow since i used glTexSubImage2D to upload new video content.
Now after i had a deep look at jME3 i think you guys could really need video support ;). And since i have some time now i want
to contribute a video renderer system to jME3.
This time i’ll use gstreamer-java. A very good project with high activity http://code.google.com/p/gstreamer-java/.
I’ve tested their stuff and it worked great. For texture updates i’ll use a PBO to get direct memory access for fast
texture updates.
Unfortunately i’m not very familiar with the jME3 project structure yet. Maybe you could you give me some tips to create easy to
use interfaces which feel “jME3-LIKE”.
And how can i use PBO in jME3 ? Good, old OGL calls or are there any objects managing this ?
Sorry for my poor english, still trying to improve it 
I hope this project could be useful… What do you think about that ??
Regards,
Chris
Hey Chris, love to hear you want to invest some time in jme3!
Did you see the current video implementation in com.jme3.video? Its a first attempt by @Momoko_Fan to do a video player system, you should get an idea of the jme3 side of things through that. Using OpenGL calls directly is disencouraged in jme3, if you find you cannot do something using the jme3 api, let us know and we will check what we can do to solve the issue. Thanks for getting in contact!
Cheers,
Normen
Hi Normen!
Thanks for your reply. I knew com.jme3.video, i tried it before… In the past i developed a similiar video rendering system using xuggler instead of Jheora. But later i noticed that Xuggler was not able to seek in the movie -.- So i gave that project up because the whole project was based on Xuggler. This time it’s practically just a gstreamer-java wrapper and, of course, the opengl rendering part.
Do you know whether jME3 supports the “OpenGL Pixel Buffer Object (PBO)” ? Because that would be the trick to do texture updates very fast. I would like to use the PBO with glTexSubImage(…). The OpenGL standard gives you the possibility to use the last parameter as PBO offset. So you don’t give OpenGL a byte array to fill the texture but the offset of the PBO which is actually bound to the OGL context. “glTexSubImage (…, long offsetPBO)”
Well, I think I start with searching the WIKI to find the answers, or maybe you could help ?
Cheers,
Chris
jME3 currently does not use PBOs. What I had success with however was creating multiple textures and then queuing them up from the video decode thread into the render thread. This way you gain the asynchronous update nature of PBOs without having to use them.
Ok, that sounds like an idea, but in that case you have to create n-textures per video. Is there no way, to add PBO support ? I mean it’s standard since 2.0 (Am i right ?).
Hi,
after creating a working version for pure LWJGL i want to port it to jME3.
but i really want to use PBO since multiple textures per video sounds more like a work-around and not like a performant solution.
In Addition:: PBO is more than a asynchronous texture update… The update is much faster since there are no copies of the texture content.
If you perform a texture update with the usual glTex(Sub)Image2D you will give your OGL implementation a ByteBuffer which contains the texture content. Usually the OGL drivers make a local copy of the content and use this local copy to update the VRAM. This need a lot of time (1024*1024 image needs 20-30 ms on my Laptop when i use glTexSubImage2D without the PBO).
The PBO is much better since there is no local copy. You can use an existing direct!! buffer and let OGL copy it directly into the PBO which is actually stored in the VRAM. After this you can use glTexSubImage2D to update the texture content from the PBO, which is asynchronous! The first copy DirectBuffer->PBO is not asychnronous but very fast, so there is no delay. The second part is asynch, but since we are using the PBO as source the update is VERY fast, because OGL perform a copy inside the VRAM.
So a few textures to “buffer” the video is very tricky but IMHO not the best way. I would be very thankful if we/you/i? could add PBO support. The result will be 1a video rendering!
I’m uploading a demo vid to youtube. I’ll give you the link when the update is complete…
Regards,
Chris
A video with 720p resolution. On a Q8300 2,5 Ghz 5-7% CPU usage. VLC takes 5%.
If someome is interested in the code, feel free to use it in any way you like 
I hope this could help to implement PBO with jme3. Btw: gstreamer-java plays pretty much every format. MKV… mp4 and even mov files are valid.
Main.java:
[java]
/*
- To change this template, choose Tools | Templates
- and open the template in the editor.
*/
package lwjglvideorenderer;
import lwjglvideorenderer.opengl.lwjgl.media.test.BasicLWJGLTest2;
/**
*
-
@author Kr0e
*/
public class Main {
/**
-
@param args the command line arguments
*/
public static void main(String[] args) throws Exception {
// TODO code application logic here
new BasicLWJGLTest2();
}
}
[/java]
BasicLWJGLTest2:
[java]
package lwjglvideorenderer.opengl.lwjgl.media.test;
import java.io.File;
import java.util.concurrent.TimeUnit;
import javax.swing.JFileChooser;
import lwjglvideorenderer.opengl.lwjgl.media.OpenGLMediaPlayer;
import org.gstreamer.ClockTime;
import org.lwjgl.LWJGLException;
import org.lwjgl.Sys;
import org.lwjgl.input.Keyboard;
import org.lwjgl.input.Mouse;
import org.lwjgl.opengl.Display;
import org.lwjgl.opengl.DisplayMode;
import org.lwjgl.opengl.GL11;
/**
*
-
@author Christopher Probst
*/
public class BasicLWJGLTest2 {
final OpenGLMediaPlayer player = new OpenGLMediaPlayer();
int k = 0;
public BasicLWJGLTest2() throws Exception {
JFileChooser f = new JFileChooser();
f.showOpenDialog(null);
File ff = f.getSelectedFile();
int targetWidth = 800;
int targetHeight = 600;
DisplayMode chosenMode = null;
try {
DisplayMode[] modes = Display.getAvailableDisplayModes();
for (int i = 0; i < modes.length; i++) {
if ((modes.getWidth() == targetWidth) && (modes.getHeight() == targetHeight)) {
chosenMode = modes;
break;
}
}
} catch (LWJGLException e) {
Sys.alert("Error", "Unable to determine display modes.");
System.exit(0);
}
if (chosenMode == null) {
Sys.alert("Error", "Unable to find appropriate display mode.");
System.exit(0);
}
try {
Display.setDisplayMode(chosenMode);
Display.setTitle("An example title…");
Display.setVSyncEnabled(true);
Display.create();
} catch (LWJGLException e) {
System.exit(0);
}
GL11.glMatrixMode(GL11.GL_PROJECTION);
GL11.glLoadIdentity();
float ratio = (float) Display.getDisplayMode().getWidth() / Display.getDisplayMode().getHeight();
//Perspective etc…
GL11.glFrustum(-ratio, ratio, -1, 1, 1, 100);
GL11.glMatrixMode(GL11.GL_MODELVIEW);
GL11.glLoadIdentity();
GL11.glViewport(0, 0, Display.getDisplayMode().getWidth(), Display.getDisplayMode().getHeight());
GL11.glClearColor(1, 1, 1, 1);
boolean gameRunning = true;
//Enable textures
GL11.glEnable(GL11.GL_TEXTURE_2D);
System.out.println(player.getGStreamerPlayer().getPipeline().getClock().getTime());
player.getGStreamerPlayer().setInputFile(ff);
player.getGStreamerPlayer().play();
float scale = 10;
float rotate = 0;
GL11.glClearColor(0, 0, 0, 0);
while (gameRunning) {
GL11.glClear(GL11.GL_COLOR_BUFFER_BIT);
if (Mouse.isButtonDown(0)) {
int x = Mouse.getX();
float percent = ((float) x) / Display.getDisplayMode().getWidth();
player.getGStreamerPlayer().pause();
long dur = player.getGStreamerPlayer().getPipeline().queryDuration(TimeUnit.SECONDS);
long secs = Math.round(dur * percent);
player.getGStreamerPlayer().getPipeline().seek(ClockTime.fromSeconds(secs));
player.getGStreamerPlayer().play();
}
//keyinput
if (Keyboard.isKeyDown(Keyboard.KEY_LEFT)) {
rotate += 0.3f;
} else if (Keyboard.isKeyDown(Keyboard.KEY_RIGHT)) {
rotate -= 0.3f;
}
//keyinput
if (Keyboard.isKeyDown(Keyboard.KEY_ADD)) {
scale += 0.07f;
} else if (Keyboard.isKeyDown(Keyboard.KEY_MINUS)) {
scale -= 0.07f;
}
try {
Thread.sleep(10);
} catch (Exception e) {
}
player.getPBO().render();
float ratio2 = player.getPBO().getRatio();
GL11.glPushMatrix();
GL11.glTranslatef(0, 0, -15);
GL11.glRotatef(rotate, 0, 1, 0);
GL11.glScalef(scale, scale, 0);
if (player.getPBO().getPBOTexture() != null) {
player.getPBO().getPBOTexture().bind();
GL11.glBegin(GL11.GL_QUADS);
GL11.glTexCoord2f(0, 0);
GL11.glVertex3f(-1, ratio2, -2);
GL11.glTexCoord2f(1, 0);
GL11.glVertex3f(1, ratio2, -2);
GL11.glTexCoord2f(1, 1);
GL11.glVertex3f(1, -ratio2, -2);
GL11.glTexCoord2f(0, 1);
GL11.glVertex3f(-1, -ratio2, -2);
GL11.glEnd();
try {
Thread.sleep(10);
} catch (Exception e) {
}
}
// now tell the screen to update
GL11.glPopMatrix();
Display.update();
// finally check if the user has requested that the display be
// shutdown
if (Display.isCloseRequested()) {
gameRunning = false;
player.getPBO().dispose();
player.getGStreamerPlayer().stop();
Display.destroy();
System.exit(0);
}
}
}
}
[/java]
PBO.java
[java]
package lwjglvideorenderer.opengl.lwjgl.media;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import org.lwjgl.BufferUtils;
import org.lwjgl.opengl.GL15;
import org.lwjgl.opengl.GL21;
import sun.nio.ch.DirectBuffer;
/**
*
-
@author Christopher Probst
*/
public final class PBO {
//PBO id
private int pboId = 0;
//PBO texture
private volatile PBOTexture pboTexture = null;
//Mapped byte buffer
private ByteBuffer mappedBuffer = null;
//Do we need an upate ?
private boolean update = false;
//Do we need a resize ?
private volatile boolean resize = false;
//Size vars
private int size = 0;
private int width = 0;
private int height = 0;
//Indicates that this PBO was used at least once
private boolean init = false;
public synchronized boolean update(IntBuffer pixelBuffer, int width, int height) {
//Calc pixel count
int pixelCount = width * height;
//Calc frame size
int frameSize = pixelCount * 4;
//Wrong size…
if (pixelBuffer.remaining() != pixelCount
|| pixelBuffer.remaining() == 0) {
return false;
}
//Adjust size if necessary
if (size != frameSize) {
//Get size
size = frameSize;
//Get sizes
this.width = width;
this.height = height;
//Set update to false
update = false;
//Set resize to true
//VOLATILE - FLUSH!
resize = true;
} else if (!resize) {
//Adjust order to speed up pixel transfer
mappedBuffer.order(pixelBuffer.order());
//At first clear the buffer
mappedBuffer.clear();
//Copy buffer
mappedBuffer.asIntBuffer().put(pixelBuffer);
//Rewind the buffer
mappedBuffer.rewind();
//Simply set update to true
update = true;
}
return true;
}
public int getHeight() {
return height;
}
public int getSize() {
return size;
}
public int getWidth() {
return width;
}
public float getRatio() {
if (getWidth() == 0) {
return 0;
}
return getHeight() / (float) getWidth();
}
public boolean isInit() {
return init;
}
public PBOTexture getPBOTexture() {
return pboTexture;
}
public synchronized void dispose() {
//Remove buffers
IntBuffer ids = BufferUtils.createIntBuffer(1);
ids.put(pboId);
ids.flip();
GL15.glDeleteBuffers(ids);
pboId = 0;
//Destroy buffers
pboTexture.destroy();
pboTexture = null;
//Reset vars
mappedBuffer = null;
update = false;
resize = false;
size = 0;
width = 0;
height = 0;
init = false;
}
public synchronized void render() {
//At first check buffer id!
if (pboId == 0) {
//Generate buffer
IntBuffer ids = BufferUtils.createIntBuffer(1);
GL15.glGenBuffers(ids);
pboId = ids.get();
}
//Create texture
if (pboTexture == null) {
pboTexture = new PBOTexture();
}
//Initialize buffer
if (resize) {
//Create texture
pboTexture.bind();
//Bind buffer
GL15.glBindBuffer(GL21.GL_PIXEL_UNPACK_BUFFER, pboId);
//Initialize buffer size
GL15.glBufferData(GL21.GL_PIXEL_UNPACK_BUFFER, size, GL15.GL_STREAM_DRAW);
//Load image data
pboTexture.loadTexture(width, height, 0);
//Map new buffer!
mappedBuffer = GL15.glMapBuffer(GL21.GL_PIXEL_UNPACK_BUFFER,
GL15.GL_WRITE_ONLY,
mappedBuffer);
//Unbind buffer
GL15.glBindBuffer(GL21.GL_PIXEL_UNPACK_BUFFER, 0);
//Unbind texture
pboTexture.unbind();
//First use!
init = true;
//Now we are initialized
resize = false;
} else if (update) {
//Bind the texture
pboTexture.bind();
//Bind buffer
GL15.glBindBuffer(GL21.GL_PIXEL_UNPACK_BUFFER, pboId);
//Then unmap the active buffer
GL15.glUnmapBuffer(GL21.GL_PIXEL_UNPACK_BUFFER);
//Upate texture with DMA!
pboTexture.updateTexture(0, 0, width, height, 0);
GL15.glBufferData(GL21.GL_PIXEL_UNPACK_BUFFER, size, GL15.GL_STREAM_DRAW);
long d = System.nanoTime();
if (mappedBuffer != null) {
System.out.println("Old address: " + ((DirectBuffer) mappedBuffer).address());
}
//Map new buffer directly
mappedBuffer = GL15.glMapBuffer(GL21.GL_PIXEL_UNPACK_BUFFER,
GL15.GL_WRITE_ONLY,
null);
if (mappedBuffer != null) {
System.out.println("New address: " + ((DirectBuffer) mappedBuffer).address());
}
d = System.nanoTime() - d;
System.out.println("Update time : " + d / 1000000);
//Unbind buffer
GL15.glBindBuffer(GL21.GL_PIXEL_UNPACK_BUFFER, 0);
//Unbind texture
pboTexture.unbind();
//Update is finished
update = false;
}
}
}
[/java]
PBOTexture.java
[java]
package lwjglvideorenderer.opengl.lwjgl.media;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.IntBuffer;
import org.lwjgl.BufferUtils;
import org.lwjgl.opengl.GL11;
import org.lwjgl.opengl.GL12;
/**
*
-
@author Christopher Probst
*/
public class PBOTexture {
public static final int LITTLE_PIXEL_FORMAT = GL12.GL_BGRA;
public static final int BIG_PIXEL_FORMAT = GL11.GL_RGBA;
public static final int NATIVE_PIXEL_FORMAT = ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN
? LITTLE_PIXEL_FORMAT : BIG_PIXEL_FORMAT;
private final int textureID;
private volatile int pixelFormet = NATIVE_PIXEL_FORMAT;
private int width = -1, height = -1;
public PBOTexture() {
IntBuffer buf = BufferUtils.createIntBuffer(1);
GL11.glGenTextures(buf);
buf.rewind();
textureID = buf.get();
}
public float getRatio() {
if (getWidth() == 0) {
return 0;
}
return getHeight() / (float) getWidth();
}
public int getHeight() {
return height;
}
public int getWidth() {
return width;
}
public int getPixelFormet() {
return pixelFormet;
}
public void setPixelFormet(int pixelFormet) {
this.pixelFormet = pixelFormet;
}
public void updateTexture(int x, int y, int w, int h, IntBuffer data) {
GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, x, y, w, h,
getPixelFormet(), GL11.GL_UNSIGNED_BYTE, data);
}
public void updateTexture(int x, int y, int w, int h, ByteBuffer data) {
GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, x, y, w, h,
getPixelFormet(), GL11.GL_UNSIGNED_BYTE, data);
}
public void updateTexture(int x, int y, int w, int h, long pixelBufferOffset) {
GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, x, y, w, h,
getPixelFormet(), GL11.GL_UNSIGNED_BYTE, pixelBufferOffset);
}
public void loadImage(BufferedImage image) {
DataBufferInt ints = (DataBufferInt)image.getData().getDataBuffer();
loadTexture(image.getWidth(), image.getHeight(), IntBuffer.wrap(ints.getData()));
}
public void loadTexture(int width, int height, IntBuffer data) {
this.width = width;
this.height = height;
//Use specific filters
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MIN_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MAG_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_T, GL11.GL_CLAMP);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_S, GL11.GL_CLAMP);
//Set image data
GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL11.GL_RGBA, width, height,
0, getPixelFormet(), GL11.GL_UNSIGNED_BYTE, data);
}
public void loadTexture(int width, int height, ByteBuffer data) {
this.width = width;
this.height = height;
//Use specific filters
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MIN_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MAG_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_T, GL11.GL_CLAMP);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_S, GL11.GL_CLAMP);
//Set image data
GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL11.GL_RGBA, width, height,
0, getPixelFormet(), GL11.GL_UNSIGNED_BYTE, data);
}
public void loadTexture(int width, int height, long pixelBufferOffset) {
this.width = width;
this.height = height;
//Use specific filters
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MIN_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MAG_FILTER, GL11.GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_T, GL11.GL_REPEAT);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_S, GL11.GL_REPEAT);
//Set image data
GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL11.GL_RGBA, width, height,
0, getPixelFormet(), GL11.GL_UNSIGNED_BYTE, pixelBufferOffset);
}
/**
- Destroys this Texture, reclaiming all resources
*/
public void destroy() {
IntBuffer scratch = BufferUtils.createIntBuffer(1);
scratch.put(0, textureID);
GL11.glDeleteTextures(scratch);
}
/**
-
@return Texture ID for this image
*/
public int getTextureID() {
return textureID;
}
/**
- Binds this image
*/
public void bind() {
GL11.glBindTexture(GL11.GL_TEXTURE_2D, textureID);
}
public void unbind() {
GL11.glBindTexture(GL11.GL_TEXTURE_2D, 0);
}
}
[/java]
OpenGLMediaPlayer.java
[java]
package lwjglvideorenderer.opengl.lwjgl.media;
import java.nio.IntBuffer;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.gstreamer.Gst;
import org.gstreamer.elements.RGBDataSink;
import org.gstreamer.elements.RGBDataSink.Listener;
import org.gstreamer.media.PlayBinMediaPlayer;
/**
*
- @author Christopher Probst
*/
public final class OpenGLMediaPlayer implements Listener {
private final PBO pbo = new PBO();
private final PlayBinMediaPlayer gstreamerPlayer;
public void rgbFrame(int width, int height, IntBuffer rgb) {
pbo.update(rgb, width, height);
}
static {
//Init GStreamer!!!
Gst.init();
}
public OpenGLMediaPlayer() {
//Use RGB data sink
RGBDataSink videosink = new RGBDataSink("OpenGLMediaPlayer", this);
//Very important! Use direct buffers, so the slow JVM is not involved!! Jiha :
videosink.setPassDirectBuffer(true);
//Default lateness
videosink.getSinkElement().setMaximumLateness(20, TimeUnit.MILLISECONDS);
videosink.getSinkElement().setQOSEnabled(true);
//Create default media player
gstreamerPlayer = new PlayBinMediaPlayer("OpenGLMediaPlayer",
Executors.newCachedThreadPool());
gstreamerPlayer.setVideoSink(videosink);
}
public PlayBinMediaPlayer getGStreamerPlayer() {
return gstreamerPlayer;
}
public PBO getPBO() {
return pbo;
}
}
[/java]
With best regards,
Christopher Probst
Good work! However remember that you are not permitted to use the DIVX codec freely in your application, as its covered by patents. You will have to pay royalty fees.
This is why the current jME3 video player only supports theora, its the only video codec that doesn’t have a patent pool yet.
Hmm, the gstreamer built i use was compiled using -lgpl. This code only uses gstreamer-java which is just a wrapper. I guess that gstreamer is able to use proprietary codec if they are installed on the OS which you are using. So this should not be a problem. In addition, gstreamer is open source and distributed under the lgpl licence and so is gstreamer-java.
Jiha, found it on the gstreamer website:
Multiplatform
GStreamer has been ported to a wide range of operating systems, processors and compilers. These include but are not limited to Linux on x86, PPC and ARM using GCC. Solaris on x86 and SPARC using both GCC and Forte, MacOSX, Microsoft Windows using MS Visual Developer, IBM OS/400 and Symbian OS.
GStreamer can bridge to other multimedia frameworks in order to reuse existing components (e.g. codecs) and use platform input/output mechanisms:
- Linux/Unix: OpenMAX-IL (via gst-openmax)
- Windows: DirectShow
- MaxOS X: QuickTime
Cheers,
Chris
He was talking about the encoded content that you deliver with your app, not the decoder software.
Well, i think i’m confused now. Do you mean that i could never use stuff like ffmpeg, gstreamer etc because i had to pay for codecs which i’m using ?
EDIT: Or do you mean playing a video is not permitted but delivering the decoder software is ok ?
You are neither permitted to distribute the decoder nor material encoded in that format. So no libraries in gstreamer with decoders except ogg and no mp3’s, mp4’s, h264, divx or whatever content except ogg/vorbis.
Well, and how VLC does this ? I’m really confused now…
Ah well, i think i get you now. After a deep look at gstreamer i noticed that gstreamer does not distributed any decoders for mp3, mp4, h264 but uses installed codec on the OS. But i really need a clear answer, is my solution “legal” or not. Otherwise i won’t spend more time for coding… Sorry that i don’t really understand your answers… 
Regards,
Chris
Writing this software is ok, but published games can only have ogg video and audio if they dont pay.
But if the user uses my program to play one of HIS files, this would be ok ??
Okay just to confirm a few things:
If you sell your application to users, and the decoder code is in your application, then you could get sued for patent infringement in countries that enforce software patents (United States, Japan).
If you use the OS (e.g. DirectShow) then its okay, however note that since Linux is free, it will not include patented codecs by default, that would mean that your application won’t work the same across all platforms.
And yes, users could play their own mp3 files with it.