SDK Improvements Ideas

[atomixnmc]

I’ve just take a look at the SDK Repo. Really appreciate you guys still keep it up to date, I still like to use the SDK to develop even after a lot of “up and down” with Netbean development recently. But there are hopes:

• Netbean 11 seems a very first good release under the name of Apache.
• It have dedicated Gradle plugin which work pretty well, can start new project with need libraries in seconds, (feels like NodeJs somehow)
• I saw the current SDK did have migrated to Netbean 11 by @MeFisto94 . Congrat

But some plugins need to be fixed and migrated to Netbean 11 also. I can’t find where these things are pointed out beside of Issues tab in Github, even there are many issues listed already, I still think a big refactor needed to make the way to future. Just want to open a discussion here if it’s suitable:

In general

• The structure of the repo is way too complex. It contain many projects that either outdate or not really related to the core. We should keep only the core elements: viewer, composer, project template and assets. This split is very important for me, i will explain below.
• Make things abstract: abstract importer plugin, State plugin, Node plugin, Gizmo plugin, Build pipeline plugin
• We should improve build script, may be ultilize Gradle more, eg: use Gradle plugin to build Netbean module ( GitHub - SGGames/gradle-nbm-plugin: A Gradle plugin to build NetBeans modules and RCP applications. ) , break build script in to serveral maintainable sub-scripts for each modules
• Nifty Editor is out dated → Move to another repo
• Cinematic Plugin is out dated. → Move to another repo
• Each importers should be in one repo
• Each build tool which platform specific like android or ios should be in one repo
• Have new way to manage plugins xml repository, connect to update center, seem like recently jME update center link is broken

Detail components:

• Improve SceneComposer, better UI and UX: split components about Transform toolset, Light, Effect, Physics into different TopComponent
• Improve Importer, each of it has dynamic configurable UI to setup configuration before importing happen and better error handling
• Improve ObjectProperties: corporate with RxJava and Meta widget concepts so very dynamic and flexible to edit, replace Node Properties of Netbean which very complex and un intuitive.
• Improve Scripting: Create Groovy to add editor components on the fly, which can interact with Scene graph nodes or other component.

I will try to do these below in my own fork first. I have my own code base with a lot of improvement over old version of the SDK and engine (version 3.0). If I try to merge it to current master branch at once will be a big diff with many changes… so I try to prevent my self doing a hard fork with the current version of SDK, need the repositories split.

Want to know what you guys think?

[oxplay2]

im happy you want to help cleanup it

but please contact @Darkchaos (@MeFisto94) also on Discord and he could tell if everything is fine with this plan or if something in plan need change.

he is like SDK guru here.

[pspeed]

Of course, he reads the forum, too… and then the discussion will be searchable forever.

[tonihele]

I would be really careful with the Gradle thing. Maven & Ant are officially supported and recently Maven has been kinda promoted as a default for Netbeans platform projects. I tried to convert the SDK to Maven, but I halted this because there is so much going on in the Netbeans itself. @Darkchaos has made amazing job navigating around the various NB platform issues. And since now they more or less work with the SDK, I’m hesitant to bring changes of this magnitude right when everything is working again and JME 3.3 (thus SDK with its support) release is just around the corner.

There are some really outdated 3rd party packages there also that I noticed with Maven. So at very least Ivy on top of Ant would be very welcome

If the Gradle really works… that would be my choice also over Maven.

[Darkchaos]

Welcome back @atomixnmc
So as you know keeping up with the Engine is taking all the available resources already, where your proposal would better have a whole team behind it, but I’ll go into details there.

@atomixnmc

The structure of the repo is way too complex. It contain many projects that either outdate or not really related to the core. We should keep only the core elements: viewer, composer, project template and assets. This split is very important for me, i will explain below.

The repository should mainly contain all the SDK Modules, and as such I cannot understand being outdated, only not being strictly “core”, but even those all fall under your list (e.g. importers, material editor, terrain editor, all are “assets”).

@atomixnmc

Make things abstract: abstract importer plugin, State plugin, Node plugin, Gizmo plugin, Build pipeline plugin

What is the idea behind this? e.g. we already have a JmeNodes class to extend, Gizmos are very specific and only Importers and maybe BuildPipeline make sense as Plugin.

@atomixnmc

We should improve build script, may be ultilize Gradle more, eg: use Gradle plugin to build Netbean module ( https://github.com/SGGames/gradle-nbm-plugin ) , break build script in to serveral maintainable sub-scripts for each modules

As @tonihele has already outlined: Netbeans itself still use ant, the ability to use maven is “brand new”, I don’t think we should risk potential problems for what currently only is a ant.ant("build.xml") statement in our gradle file. Our Gradle build could be improved, but that’d be other spots.

@atomixnmc

Nifty Editor is out dated → Move to another repo

I thought about including it but disabled. Moving to another repo would work, but then most likely we don’t use it anymore.

@atomixnmc

Cinematic Plugin is out dated. → Move to another repo

Any issues with that? I didn’t hear of them

@atomixnmc

Each importers should be in one repo

That’s an engine problem. We basically just load the files with the Asset Manager. We have some files to register the file extensions, but I don’t see the benefit of two files per new-importer-repo.

@atomixnmc

Each build tool which platform specific like android or ios should be in one repo

The problem with this is twofold:

• Making it even less clean than now. Yes the repository contains many many modules, but imagine you would have to search for all those modules under the jme engine. Specifically people looking at the engine don’t find it anymore because there are so many repositories
• The Build: I’ve tried to do modular building (GitHub - MeFisto94/nbproject and the use in GitHub - MeFisto94/jme3-artificial-intelligence), but there is a problem: They all need the full SDK build when trying to use any of the SDK features, so we basically need to build them in-tree, so it only works for plugins like the above, which only wraps a jar.

@atomixnmc

Have new way to manage plugins xml repository, connect to update center, seem like recently jME update center link is broken

Broken because not hosted. There wasn’t much interest in plugins lately, even less so with the new store, so we didn’t solve that problem anymore.

@atomixnmc

Improve SceneComposer, better UI and UX: split components about Transform toolset, Light, Effect, Physics into different TopComponent

How exactly do you envision that? I mean we have Effects as a separate Window, Physics has almost nothing to edit etc.

@atomixnmc

Improve Importer, each of it has dynamic configurable UI to setup configuration before importing happen and better error handling

Error handling has improved with an Exception Reporter Dialog, but also note: Not every importer supports configuration, actually most things are SDK workarounds and rarely a few AssetKey things.
Also the community wants to remove diversion and enforce GLTF as standard.

@atomixnmc

Improve ObjectProperties: corporate with RxJava and Meta widget concepts so very dynamic and flexible to edit, replace Node Properties of Netbean which very complex and un intuitive.

Well the reason of using internal structures is to not have to implement them all by our own.
Like they are themed, support translation, Mouse and Keyboard Interaction etc.
However we can add any given swing component as a custom editor for every type (Vector3f, e.g.).
But that’s a lot of work and needs precise ideas first.

@atomixnmc

Improve Scripting: Create Groovy to add editor components on the fly, which can interact with Scene graph nodes or other component.

That would be cool, I wonder how to do this, though and specifically if we can’t do a thing like right click on any .class file and load it as plugin. But then we’d need a whole API for all of that.

[tonihele]

@Darkchaos

Netbeans itself still use ant, the ability to use maven is “brand new”, I don’t think we should risk potential problems for what currently only is a ant.ant("build.xml") statement in our gradle file. Our Gradle build could be improved, but that’d be other spots.

To still clarify this. The feature, Netbeans platform project with Maven, was officially introduced with Netbeans 7.2 (2012). Netbeans 11 (2019) just bumped the feature so that it is the first thing you see, before Ant. The Maven build NB platform project seems to be very popular and people convert Ant projects to it. I didn’t find any official figures, but overall Ant vs Maven (vs Gradle), Ant is losing ground.

When converting the SDK platform project, I kind of immediately found a problem with the Netbeans installer Maven module. Which was fortunately just fixed 3 days prior to my conversion attempt. I’m not sure which of the problems come from the Maven modules and which from the Netbeans JAR/lib stuff that also affect Ant builds. But this indeed is in a sorry state currently. A bit demotivating to try any big change like this. As the current Ant version has these pretty much now covered. Hopefully Netbeans team get their shit together.

Netbeans 11.3 should bring some fixes to the platform modules, enabling them to be built with newer Java versions. Hopefully some help.

[atomixnmc]

@Darkchaos

The repository should mainly contain all the SDK Modules, and as such I cannot understand being outdated, only not being strictly “core”, but even those all fall under your list (e.g. importers, material editor, terrain editor, all are “assets”).

It will be easier if we have a look at the code, so I’m preparing the fork version in my account, we can compare the pros and cons of the approaches.

I categorize Core is the plugins that essential for development with JME beside of the engine, we need a viewer and a composer and the way to load assets.
The other plugins, may need to move out because they may not be that essentital, or should be moved for the sake of maintain for example:

• Physics may just be used in Physics game.
• Same with Android, iOS (you get the point)
• NiftyUI has a lot of errors right now, I can fix it (i’m working with Nifty for long), but I’d like to have it separately.
• Cinematic plugins, may just be used in game with Cinematic.
• New plugins should use the new Assets Store, have an Update Center there if that’s a SDK plugins.

@Darkchaos

What is the idea behind this? e.g. we already have a JmeNodes class to extend, Gizmos are very specific and only Importers and maybe BuildPipeline make sense as Plugin.

Extend the SDK is very important because people will want to create their own editor not using our if we not allow them to do so, or not easily to extend. Right now it’s the case with Netbean Platform, a lot of new syntax and concepts one have to learn to create plugin with it. What I want to bring to the table is the concepts that already used in other Engine (ahem, Unity) that prove to be easy to extend.

• StatePlugin is an easy way to create new State that connect your SceneGraph to the Editor
• NodePlugin is in turn connect one particular Node to the Editor (by property, by Flag)
• Gizmo is the helper that make transformation or manipulation if Node or Geometry easier. Gizmo is the Shape that present Waypoint, Icons in the 3d scene. We can add more gizmo easily to improve composing and editing task
• Build pipeline is to help corporate with others easier. I might want to have my code automaticly deploy into Android device or in a Mac even when I develop in PC. [More examples to come in another post about this]

@Darkchaos

As @tonihele has already outlined: Netbeans itself still use ant, the ability to use maven is “brand new”, I don’t think we should risk potential problems for what currently only is a ant.ant("build.xml") statement in our gradle file. Our Gradle build could be improved, but that’d be other spots.

This is actually a big move. If we integrate with Gradle, or we introduce another kind of Project call JME Gradle project, we ultilize the Pipeline that Gradle provide.
Pros:

• Pipeline and Plugins
• Flexible pipeline

Cons:

• Big changes from Ant
• New knowledge
• Easy to mess up compare to Maven because its not declarative like Maven

Do it properly will help we solve this, migrate away from this jar tree, mono repo situation we have here.

@Darkchaos

but there is a problem: They all need the full SDK build when trying to use any of the SDK features, so we basically need to build them in-tree , so it only works for plugins like the above, which only wraps a jar.

Mono repo is a sensitive topic, that’s why I tried to bring this up. Developers like to keep everything in one place, that’s where things got clustered. Old code keep hold us to the point we can not roll out new things any more. In the other hand, break things into several repositories we have the freedom and the power of people in community by Opensource model but come with other headache: compatible and oschestration of builds. I will not say I have a clear cut of what need to be keep, what need to be separated, that’s why I will do in my forks first.

I will do in this orders:

• Break into modules for the SDK, all use Gradle to build, (but one Gradle file per module). Then break into several repositories
• Host update center xml staticly in surge.sh or github, So we do not to care about the cost of hosting update center at all.
• Introduce new concepts like Pipeline, Gizmo, Importer in new Plugins
• Have sample code to introduce usage of new Plugins

We speak easier in code and diagram… Let’s do some