SceneProcessorCopyToImageView Performance problem in JME inside JFX ImageView solution hogs CPU, framerate unstable and stutter!

TypischAndy · October 24, 2016, 4:44pm

I have been using DavidB’s (@david_bernard_31) solution for having a Offscreen JME application displayed in an ImageView in a normal JFX Application. It has been working great but when we put it on a severely slower Client PC we noticed its lagging horribly. On inspection it seems the main culprit is the method copyFrameBufferToImage. Got ~400fps when the app was still JME with JFX HUDs, and now that its the other way around its about 60fps, on the strong development machine. The main problem is that the process is slowing down JavaFX too, I dont care about the JME frames too much as long as I have about 10 frames I am good. But JFX culling sometimes doesnt happen for a few seconds. I believe the culprit to be in all the ColorBuffers, I am using BGRA8 but I think there still is a conversion going on causing the slowdown. And yes I am using the method readFrameBufferWithFormat. But the PixelWriter creates a converter that call getSetter() which gets a ByteBgraPre Format. Know I have little knowledge of ColorBuffers and 3D in general so I am unaware of what the difference really is.

I have been trying for multiple days now to solve it. JProfiler just shows that JFX and JME3 Thread are blocking each other. I tried different JME versions (3.1, 3.1.alpha1, 3.1.beta1) and lwjgl versions 2.9.3 and 2.9.1. Even different JREs (8u66, 8u77, 8u111)

I am at my wits end and any hint or idea would be appreciated.

Edit: There seem to be spikes in CPU Usage but I couldnt find out why. Also even on the developer PC when not putting a framelimit on, the JFX UI is very unresponsive. I get about 60-90 frames but only very few in JFX making it useless.

Edit1.5: Just installed Afterburner and it shows that my GPU usage is jumping from 0-100 with some in between steps. CPU is not bottlenecking. if it where constantly using my gpu i could imagine double the fps atleast.

Edit 2: The main 2 classes I am talking about:

github.com

empirephoenix/JME3-JFX/blob/master/src/main/java/com/jme3x/jfx/injfx/SceneProcessorCopyToImageView.java

package com.jme3x.jfx.injfx;

import java.nio.ByteBuffer;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;

import com.jme3.app.Application;
import com.jme3.post.SceneProcessor;
import com.jme3.renderer.RenderManager;
import com.jme3.renderer.ViewPort;
import com.jme3.renderer.queue.RenderQueue;
import com.jme3.texture.FrameBuffer;
import com.jme3.texture.Image;
import com.jme3.texture.Image.Format;
import com.jme3.util.BufferUtils;
import com.jme3x.jfx.FxPlatformExecutor;

import javafx.beans.value.ChangeListener;
import javafx.scene.image.ImageView;
import javafx.scene.image.PixelFormat;

This file has been truncated. show original

github.com

empirephoenix/JME3-JFX/blob/master/src/main/java/com/jme3x/jfx/injfx/JmeForImageView.java

package com.jme3x.jfx.injfx;

import java.util.concurrent.Future;
import java.util.function.Function;

import javafx.scene.image.ImageView;

import com.jme3.app.DebugKeysAppState;
import com.jme3.app.SimpleApplication;
import com.jme3.system.AppSettings;

/**
 * JmeForImageView create a jme'SimpleApplication viewable into an JavaFX's ImageView.
 *
 * You can manage the wrapped SimpleApplication by calling enqueue (by example to add/remove
 * AppState, Node, Light, to change the AppSettings...).
 *
 * See TestDisplayInImageView.java for sample usage.
 *
 * The usage of the class is optional, it can avoid some

This file has been truncated. show original

normen · October 25, 2016, 10:40am

A lot of work in this case is probably transferring data through the PCI bus. As the image is rendered on the GPU it has to be transferred to the CPU memory for each frame… So memory and PCI bus speed might be the biggest bottleneck. Maybe try lowering the rendering resolution?

TypischAndy · October 25, 2016, 12:28pm

Doesnt help I am afraid. It truly is a CPU bottleneck, altough now I am not so sure what causes it anymore. The copyFrameBuffer method isnt the main culprit anymore now, after I removed the sycnhornized blocks. Risky and probably stupid but desperate situations calls for desperate means. Still CPU is still high, only now LWJGL/JME uses it all.

normen · October 25, 2016, 1:20pm

Hm, if it uses synchronized its probably not a great design anyway. Would probably be better if it used a queue for the images and then dropped images if/when the queue gets too full… That way the threads wouldn’t have to block each other either.

david_bernard_31 · October 25, 2016, 1:34pm

The synchronized block

github.com

empirephoenix/JME3-JFX/blob/master/src/main/java/com/jme3x/jfx/injfx/SceneProcessorCopyToImageView.java#L183


			fb = new FrameBuffer(width, height, 1);
			fb.setDepthBuffer(Format.Depth);
			fb.setColorBuffer(Format.BGRA8);
			byteBuf = BufferUtils.createByteBuffer(width * height * 4);
			img = new WritableImage(width, height);
		}
		/** SHOULD run in JME'Display thread */
		void copyFrameBufferToImage(RenderManager rm, ImageView iv) {
			synchronized (byteBuf) {
				// Convert screenshot.
				byteBuf.clear();
				rm.getRenderer().readFrameBufferWithFormat(fb, byteBuf, Image.Format.BGRA8);
			}
			FxPlatformExecutor.runOnFxApplication(() -> {
				synchronized (byteBuf) {
					if (lastIv != iv) {
						lastIv = iv;
						lastIv.setImage(img);

is required to avoid read or write on partial, using a queue will have the same constraint except if you create image buffer for each entry => lot of memory alloc/desalloc as we’ll not reuse existing.

TypischAndy · October 25, 2016, 1:40pm

Having the sycnhronized only on the block that should run in FX seems to have alleviated the problem a little bit for me. Framerate increased a bit and its also the stablest its been so far. I dont know why or anything but for now its some improvement until I can find the real problem.

david_bernard_31 · October 25, 2016, 1:56pm

Can you share a sample project that reproduce the issue, then we could experiment/work on it ?

Did you remove the limitiation to 30fps/60fps on the jme app ? (a bad idea because it will generate lot of useless copyBuffer,… and will overload the GPU->CPU)

normen · October 26, 2016, 10:35am

You can always use a ring buffer to avoid the alloc/dealloc issue. The Problem with synchonized in this case is a) synchronized isn’t very optimized and puts a lot of strain on the JVM, its basically the sledgehammer of thread safety. Thats why Sun added the concurrency classes. And b) it causes the threads to block each other which isn’t necessary.

TypischAndy · November 2, 2016, 1:32pm

@normen
Sorry I have been away this past few days. Anyways I just read up a bit on RingBuffers but I dont quite know how to implement it in this example. Do i make a queue of whole images or do I make a queue of pixels that is total pixels * 4 (rgba) long? If its not too much of a bother can you show me an example?

@david_bernard_31
The test application (TestDisplayInImageView) that is in the repo actually works as an example. If I crank it up to Fullscreen on my 1080p monitor and start it with jProfiler (which slows it down) it will reduce framerate to about ~40ish and FX gets slow too. I increased camera movement speed to 50 which makes it all the more obvious. And in JProfiler you can see how JME and FX Thread block each other all the time. The setPixels() method is using about 45% and slowly rising of the CPU according to JProfiler.

It seems to me that any kind of JavaFX Thread activity will throw it off completely. I was panning the camera around like crazy when I opened the Background combobox in the test application and it just completely froze the application (was running fairly smoothly until then) and it never quite recovered and kept being laggy from then on for a few minutes until it recovered again.

github.com

empirephoenix/JME3-JFX/blob/master/src/test/java/com/jme3x/jfx/injfx/TestDisplayInImageView.java

package com.jme3x.jfx.injfx;

import java.net.URL;

import com.jme3.app.SimpleApplication;
import com.jme3.asset.AssetManager;
import com.jme3.light.DirectionalLight;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.math.Vector3f;
import com.jme3.renderer.queue.RenderQueue.Bucket;
import com.jme3.scene.Geometry;
import com.jme3.scene.Node;
import com.jme3.scene.shape.Box;
import com.jme3.scene.shape.Sphere;
import com.jme3.system.AppSettings;
import com.jme3.util.TangentBinormalGenerator;

import javafx.application.Application;
import javafx.application.Platform;

This file has been truncated. show original