Integrating Skija/Skia renderer with JME

BFac · January 16, 2025, 12:05am

Hello everyone,

I am building a UI library with the HumbleUI\Skija package, I’m successfully using it while building it out in my game at the moment, but I’m starting to hit performance limits using it with the bitmap rendering back end.

Currently I’m encoding the Skia buffer to PNG then uploading that as a texture to JME, this works but is too slow for complex UI to feel snappy.

Skija/Skia can use an OpenGL backend so I’m now trying to upgrade using this example:

Skia OpenGL DirectContext

apparently Skija’s DirectContext can pick up with the open GL context, thats what the documentation and examples imply?

I’ve put together the basic test case, this renders the blue cube when this line is commented out of simpleUpdate():

skia_surface = getGLSurface(app.getCamera().getWidth(), app.getCamera().getHeight());

But when that line is included, the screen is black, the blue cube does not render, however the debug stats do show the number of objects flicking from between 4/5 as you move the camera with the mouse past where the cube should be, no errors, the debug statements all print okay and the FPS stays at 60.

The test case:



import com.jme3.app.SimpleApplication;
import com.jme3.material.Material;
import com.jme3.math.ColorRGBA;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;
import com.jme3.system.AppSettings;
import com.jme3.system.JmeContext;
import io.github.humbleui.skija.*;

import org.lwjgl.opengl.GL11;


public class GLTest extends SimpleApplication {

    public static GLTest app;
    public Surface skia_surface = null;


    public static void main(String[] args) {

        app = new GLTest();

        AppSettings settings = new AppSettings(true);

        //settings.disp

        settings.setFullscreen(false);
        settings.setWidth(600);
        settings.setHeight(600);

        settings.setTitle("My Awesome Game");
        app.setSettings(settings);

        app.start(JmeContext.Type.Display);

    }

    @Override
    public void simpleInitApp() {

        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);

        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", ColorRGBA.Blue);
        geom.setMaterial(mat);

        rootNode.attachChild(geom);

    }


    public static Surface getGLSurface(int width, int height) {

        // Initialize Skija's context with JME's OpenGL context
        DirectContext skija_context = DirectContext.makeGL();
        //skija_context.
        logStatic("DirectContext ptr: " + skija_context._ptr);

        // Get the current framebuffer ID from JME
        int frame_buffer_id = GL11.glGetInteger(0x8CA6);
        //int frame_buffer_id = GL30.glGenFramebuffers();
        logStatic("frame_buffer_id: " + frame_buffer_id);

        // Create Skija's BackendRenderTarget using JME's dimensions and framebuffer
        BackendRenderTarget render_target = BackendRenderTarget.makeGL(
                width, height, /* samples */ 0, /* stencil */ 8, frame_buffer_id, FramebufferFormat.GR_GL_RGBA8
        );

        // Create a surface for drawing
        Surface surface = Surface.makeFromBackendRenderTarget(
                skija_context, render_target, SurfaceOrigin.BOTTOM_LEFT, SurfaceColorFormat.RGBA_8888, ColorSpace.getSRGB()
        );

        return surface;

    }

    @Override
    public void simpleUpdate(float tpf) {

        if (skia_surface == null) {

            skia_surface = getGLSurface(app.getCamera().getWidth(), app.getCamera().getHeight());
            log("skia_surface initialised");
            return;

        } else {

            /*
            Paint paint = new Paint();

            //paint.setBlendMode(BlendMode.DARKEN);
            paint.setMode(PaintMode.FILL);
            paint.setColor(ColorRGBA.Cyan.asIntARGB());

            Rect box_size = new Rect(100, 100, 200, 200);

            skia_surface.getCanvas().drawRect(box_size, paint);

             */

        }

    }

    public void log(String s) {
        System.out.println("Main: " + s);
    }

    public static void logStatic(String s) {
        System.out.println("Main[static]: ");
    }

}

This outputs:

Main[static]: DirectContext ptr: 1609112682016
Main[static]: frame_buffer_id: 0
Main: skia_surface initialised

I’ve included a bit of skia rendering code in simpleUpdate() thats block commented out, but that would be a valid draw call if the skia surface was set up

I’m at the limits of my knowledge with OpenGL contexts, FrameBuffer ID’s and such, i’m fumbling around not sure if I’m on the right path at the moment… Does anyone have any idea why I get the black screen?

Or is there a better way to do this integration?

Thanks

pspeed · January 16, 2025, 12:14am

If the UI library provides alternate means of intercepting its calls, you might try to look at how Nifty did this sort of integration.

I don’t know if it’s helpful… but trying to coordinate direct GL context access with JME (who assumes it owns it) is probably going to be tricky.

danielp · January 16, 2025, 4:31pm

I have my own UI library built with Skija: Obsidian. I did the jME integration as a standalone module: Obsidian-jME.

I’m going to second what @pspeed said and add that I’ve never worked on a project where sharing a GL context between two different renderers worked out successfully (this is not the only time I’ve done something like this - the other time was an attempt to overlay Qt over the results of a C++ OpenGL geographical renderer). The solution I ended up using for Obsidian-jME was creating a second context set up to share textures with the jME context (provided they are properly constructed, sharing textures between OpenGL contexts is supported by the spec). This means that Obsidian/Skija & jME have their own OpenGL state with no opportunity for interference, and jME and Skija are both able to render hardware accelerated frames without ever copying pixel buffers or GPU → CPU → GPU round-tripping.

If you have any interest in kicking the tires on Obsidian, I’d love to get some feedback on it - I consider it beta quality at the moment. I use it (without major issues) in my own project but haven’t put the finishing touches into a full release yet, so it’s not well known in the jME community at the moment. It has a lot of useful features - fully skinnable without changing component types or UI logic, reactive data-driven rendering, declarative rendering styles (defined in code by default, but would be quite easy to declare in json or xml), etc. All rendering is implemented in Skia/Skija, though at the moment it wouldn’t be terribly difficult to swap in a different backend should the need arise.

BFac · January 18, 2025, 9:02pm

Thanks guys useful info, I will give up on context sharing.

@danielp that’s such a clever approach well done, I will definitely build something with it, I’ve browsed the code for both modules and it’s exactly the kind of OpenGL wizardry i was hoping to find but do not yet understand.

If I had access to mature Obsidian ~3 years ago I probably wouldn’t have decided to self-build, as it seems to offer the fine level of control down to the pixel and directly programmed approach i couldn’t find at the time. This will be a great addition to the JME suite

I’ve found ObsidianDemo - let me know if you have any other resources or complex ui examples tucked away. Did you drop the javafx stuff?

As it is I’m too far down the rabbit hole to switch as my UI is largely built out now, but this certainly demonstrates an answer to the question that I can hopefully pursue.

Honestly surprised at the mileage I’ve gotten out of the bitmap backend, it’s only beginning to choke down on larger windows with lots going on, on the attached screenshot the hover-response only fires in time if you move your mouse purposefully now haha.

That said I’ll include these snippets for anyone trying similar stuff, this is the code I use to render from a Skija canvas to a JME Geometry:

public static Geometry getCanvasAsGeom(Canvas canvas, Material mat) {

        byte[] image_bytes = getImageBytes(canvas);

        int width = canvas.getSurface().getWidth();
        int height = canvas.getSurface().getHeight();

        //dumpImage(canvas, "./button_out_" + System.currentTimeMillis() +".png");

        ByteArrayInputStream bais = new ByteArrayInputStream(image_bytes);
        BufferedImage bi;
        try {
            bi = ImageIO.read(bais);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        //log("BI: " + bi.toString());

        com.jme3.texture.Image jme_image = imgLoader.load(bi, true);

        Texture2D jme_texture = new Texture2D(jme_image);

        Geometry geom = new Geometry("myPic", new Quad(width, height));

        mat.setTexture("ColorMap", jme_texture);
        geom.setMaterial(mat);
        geom.setLocalTranslation(0, 0, -1);

        //log("Built image as png and geom");

        return geom;

    }

public Geometry getCanvasPicture() {
        Geometry geom = SkijaManager.getCanvasAsGeom(sk_canvas, window_mat);
        geom.setName(name);

        return geom;
    }

public static byte[] getImageBytes(Canvas canvas) {
        Image image = canvas.getSurface().makeImageSnapshot();
        Data pngData = image.encodeToData(EncodedImageFormat.PNG);

        //log("Made image bytes, size: " + pngData.getBytes().length);

        return pngData.getBytes();
    }

danielp · January 19, 2025, 10:59pm

It’s pretty close - there’s still a little bit of awkwardness with boundary conditions around how pixels are handled between the two (GL/jME treat a pixel as centered on the coordinate, so if you place a black dot at (1,1) that’s the only one affected, but Skia treats the pixel as centered on the leading edge so you’ll get a neighborhood of 4 pixels shaded grey instead of one shaded black). Obsidian has partially solved this, but has some awkwardness on certain boundary conditions due to the rounding method used.

Unfortunately I do not have any other better examples right now. I did drop JavaFX entirely. The big problem with JavaFX is that (a) there’s no way to hardware accelerate composition into an OpenGL context, requiring copying bitmap images from GPU (JavaFX) → CPU → GPU (jME), and (b) JavaFX has to run in its own thread, meaning that any interaction from UI ↔ app must be thread-safe and accurate synchronization between the two is impossible (even if the image transfer didn’t take multiple frames to do). What this means is that if, say, you’re using JavaFX to draw HUD elements, there will most likely be multiple frames of lag between the game state and what the HUD is showing. For some situations this is fine, for others it’s not. JavaFX is also a bit dated in how it handles logic & state, so I consider Obsidian’s reactive state-driven component rendering to be a significant advantage.

I totally understand that - I was very nearly to that point with JavaFX when I decided to invest the time into an alternative before it was that much harder to switch.

That’s quite impressive!

Elegant and simple! I wish CPU rendering was fast enough to do that for UI everywhere instead of having to mess with hardware accelerated resource management.