Video Texture

Kr0e · April 29, 2009, 4:44pm

Hi,

i know it's possible to render a video on a texture using JMF/FOBS…

My question is, how to use it with jME2 … i read this text very carefully

http://www.jmonkeyengine.com/jmeforum/index.php?topic=10561.0

I've loaded the src ( http://www.mediafire.com/file/iumoymbvnje/src.rar ) and did the change from "JOC"

data = new ArrayList<ByteBuffer>(); data.add(ByteBuffer.allocateDirect(size*size*4).order(ByteOrder.nativeOrder()));

Everything runs great without errors but, nothing is displayed on the quad. I hear the sound of the media files,
but the quad is still white... I found the problem:

//tex.setImage(image);

This line is commented but it shouldn't be... Without the slashes the program aborts with this error message :

at com.jme.image.Image.getEstimatedByteSize(Image.java:663)

Looking up this method shows, that the image doesn't have a format... and so there is a NullPointerException...

Now my question:

How can i solve this problem ? Maybe the topic creator could help... it seems to be that he solved this problem....

Greetings

Kr0e

llama · April 29, 2009, 4:58pm

Since Quaddro didn't mind spamming the board with "up up" messages I'd say spam his PM and email to the point where he says what he did

More to the point, try setting a format. Either RGB8 or RGBA8 I think.

Kr0e · April 29, 2009, 5:03pm

Hi,

ok this solved the problem but creates a new one, too.

Now the error message is :

java.lang.NullPointerException

at com.jme.scene.state.lwjgl.LWJGLTextureState.load(LWJGLTextureState.java:486)

I've tested RGB8 and RGBA8…

Any idea ?

PS:

The code which makes problems:

image.getData(0).rewind();

But i don't know which var is null... getData(0) can't be null...

->

data = new ArrayList<ByteBuffer>(); data.add(ByteBuffer.allocateDirect(size*size*4).order(ByteOrder.nativeOrder()));

Kr0e · April 29, 2009, 9:37pm

okay,

i've solved it… I should read the tutorials more carefully the next time…

Now i've registered the ByteBufferRenderer and everything runs !

@Llama:

I'm using Ubuntu 8.10… My framerate is very small… sometimes only 15… (flv format)

I did your patch but this didn't change the speed… tomorrow i'll test it on a windows pc… maybe

it's a ubuntu problem…

Greetings from germany…

Kr0e

llama · April 29, 2009, 9:38pm

So could YOU post the solution this time then? For others to find in the future…

Momoko_Fan · April 29, 2009, 11:47pm

Make sure you're using the native MPEG4 plugins for JMF/FOBS for video decoding (explained in the tutorial).

Other than that, the slow down can come from the software YV12 -> RGB conversion, especially for high resolution video. Profile the code and find where the bottleneck is.

Kr0e · April 30, 2009, 4:05pm

Hi,

what do you mean with "diff" ?

xD

@Momoko_Fan :

I'm using the native libs, otherwise the flv video couldn't be played… (millions of exception are trown :D)

Kr0e · April 30, 2009, 6:56am

Ok, the trick were very simple… The image didn't has a format and the ByteBuffer were null, because the methode "setSize" was never called. After registering the correct renderer (ByteBufferRenderer) by the JMStudio, everythings runs great To register the renderer you have to pack the COMPILED ByteBufferRenderer in a jar file. Then you have to change the start script called "runjmstudio.sh". (You must add the new jar with the ByteBufferRenderer to the classpath)

That's it !

So, your code was completely correct llama. After making a few jme2 related changes, the code runs correct.

Greetings from Germany!

Kr0e

PS: It's not really a trick xD Llama wrote all needed things in his tutorial:

http://www.jmonkeyengine.com/jmeforum/index.php?topic=2908.0

But i didn't read it carefully the first time ://

gouessej · April 30, 2009, 8:23am

Please could you give us a clear diff?

llama · May 1, 2009, 7:08pm

So do you still have the performance issues?

I can think of 2 things… if I remember correctly my default example limits your FPS to that of the video, secondly the new version of FOBS might not be compatible with the extra performance patch anymore (though I think I have described somewhere in these threads how to patch FOBS itself).

A diff is a file that shows all changes you made. However, if you could just package up all the source files in a zip, that'd be a help too…

gouessej · May 1, 2009, 7:25pm

Kr0e said:

Hi,
what do you mean with "diff" ?

I mean this:
http://svnbook.red-bean.com/en/1.0/re09.html

Kr0e · May 2, 2009, 7:47am

Yes, i still have the performence problems… Do you remember, where you create the FPS limiter ?

I use the Fobs4JMF-0.4.1 version of FOBS… the latest version only runs of windows…

I use Java 6… maybe this is one reason for the slow motion

By the way… i 've tested the example on a windows pc with the same result…

I'm new with the monkey engine… where could i upload the zip ?

Greetings

Kr0e · May 2, 2009, 9:48am

Ok,

at first i want to say that i upload the package when i've solved the performence problem…

I removed your "FPS-Brake"… I think it was this part :

@Override protected void simpleUpdate() { if (!image.update(tex, false)) { // if we did not update the texture, wait some time, but when the // next frame is ready, wait no longer. //image.waitSome(3); } }

Now the jME shows more than 450 FPS... Thats ok! But the video still runs with 8 fps and so the "FirstPersonInput" react very "laggy".... The console output tells me , that the program uses the native libs correctly... So maybe there is another brake ??

Greetings

PS:

I find the bottleneck!!

It's this method:

// TODO: use this.dataformat GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, 0, 0, videowidth, videoheight,pixelformat, GL11.GL_UNSIGNED_BYTE, buffer);

I think the problem is the conversion of the formats!!
I hope anybody has an idea !

PSS:

During bugfixing i noticed another problem:

At the moment the update method of the JMEVideoImage class is called in the main loop... So if this update method is slow the whole game will be slow... Now i have the idea to call this update method in a single timer thread... But then i get this exception:

Exception in thread "Timer-0" java.lang.NullPointerException
at org.lwjgl.opengl.GL11.glDisable(GL11.java:995)

I think this is because a GL method is called outside the main thread... But how could i fix this ? I mean... If i want to disable textures or something else outside this method, the whole program will shutdown ...

gouessej · May 2, 2009, 7:56pm

Use the game task manager to post a Callable to execute your task on the OpenGL thread.

llama · May 2, 2009, 8:52pm

… but then you're back in the same thread. I think about the only way to properly do it is by creating a whole new context, you don't want to get involved with that right now.

And while it's true the conversion from one format of pixels to another takes some time, it's only a few percent difference. I think the biggest issue right now is that your computer is busy getting those 450fps and not decoding the video. Try putting a small sleep in the game loop. Another quick fix could be enableling vsync.

Also, how many FPS does your video have?

Kr0e · May 2, 2009, 9:16pm

The video i've tested is a flv video with 30 fps. Your ByteBufferRenderer shows 9 fps … But jME shows 450 fps… But i'm sure that the problem is this method

// TODO: use this.dataformat GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, 0, 0, videowidth, videoheight,pixelformat, GL11.GL_UNSIGNED_BYTE, buffer);

When i comment this method, ByteBufferRenderer shows 29.999 Fps and jME shows 1,5K FPS... But then there is nothing displayed :lol:

A short "System.currentMillis" before and after glTexSubImage2D shows that this method needs more then 50-70 ms... So, i don't know whether this is a "normal" time for converting pixel formats... but i think it'S not the best solution... At the moment i try pixel buffers to accelerate this method... Have anybody experience with pixel buffers ?

GReetings

Momoko_Fan · May 3, 2009, 1:22am

What video card? Latest drivers? What kind of resolution are the videos in? Is multisampling set in the control panel or application? What format is "pixelformat" in that code?

9 fps is too low even if using glTexImage, and I don't think pixel buffers will help you much. I have made a similar application using my own binding to libavcodec and had about 70 fps.

Kr0e · May 3, 2009, 8:11am

Yes, i think you’re right I hate ATI & Linux support… Maybe i have to install windows :’(

I’ll test this method on a windows PC… When it runs faster then 40 FPS i’ll be happy

Kr0e

Kr0e · May 3, 2009, 11:22am

Ok, on the windows pc the method runs great…

@ Llama: Nice work!

Greetings

Kr0e

llama · May 3, 2009, 11:24am

What's your distro/ati drivers/card? Ati/AMD has released some very good open source drivers the last few months… the closed source driver is much better too.