Better way to consume and distribute jME libraries?

Hi all,

I’m a relative newcomer to jME, only having picked it up less than a couple weeks ago, but I’ve been using it every day so I’ve gotten enough for some basic first impressions. I apologize if these issues have been voiced before; I looked but could not find them, and would be happy for a link to a discussion if there is one.

My biggest concern so far, as the title suggests, is libraries. Right now, the main (and as far as I can tell, only) vector for delivering libraries from library developers to game developers seems to be as a Netbeans plugin. This is awkward for a number of reasons:

  1. Versioning. I see the Cubes plugin has some major backwards-breaking changes related to BlocksManager in SVN. How does that get released without hosing people? For regular plugins, you aren’t writing code against the API of the plugin, but for libraries…you actually need major/minor/patch versions, and I should be able to model those dependencies.
  2. Too siloed within the jmonkeyengine-contributions repository. You’re telling me that in order to release a library, I a) have to use SVN, b) have to share a repository with dozens of other people, and c) use Google Code…? In some ways it’s convenient for users, but it’s also vastly inconvenient for developers…and users.
  3. Some of the plugins are just wrapped JARs. This seems fundamentally wrong to me – plugins are meant to enhance the developer experience and productivity level, not as a means of delivering JARs to developers and end users.
  4. It makes using other IDEs harder. I use Netbeans the bare minimum amount while doing most of my development from IntelliJ, but in order to avoid compile errors in IntelliJ I end up adding the JAR in the Netbeans plugins directly to the IntelliJ classpath, which is a nasty hack. It works, but then completely breaks down if I try to do any development on another computer (switching between desktop and laptop, for example). I could just copy it into my lib directory I suppose, but…I also have this problem for the core jME libs.
  5. It’s not really “the Java way” – that would be using Maven/Ivy/Gradle or similar. Instead it’s a build process embedded in a particular IDE.

Instead of merely casting stones, I can provide some initial ideas for how to improve the above situation, broken down by category…

Build system
Instead of generating a simple build.xml Ant file, why not use a modern build system, capable of resolving dependencies? Seems like the most popular one these days is Gradle, which I’ve noticed the jME core is using, but really I’d be be okay with anything more than Ant.

Real repositories
Push jME into a real Maven-compatible repository somewhere. I’m not very familiar with this space, but I’ve gathered that it can be painful to sign and push code to Maven Central. It seems like that’s primarily a big “setup” cost and that it could be automated after that, perhaps in a manner similar to this (which admittedly uses SBT). This will make it easier for people to use other IDEs, like IntelliJ.

Besides pushing to Maven Central (or BinTray), you might also set up your own Maven repository somewhere for jME and plugins, though I have no idea what the pros/cons of that might be.

Split up jME-contributions
Not sure how much I need to go into this one – it’d be nice to let each plugin have its own repo for a variety of reasons. Allowing people to use whatever version of source control will only encourage more people to create and share plugins, and people developing on GitHub will be able to more easily incorporate feedback through pull requests and the like. If packaging and distributing JARs is too cumbersome for each individual developer, it seems like there’s ways for game developers to add dependencies directly on git repos. Obviously you’d want to only code against tagged releases, and I’m not sure it’d benefit from the version wildcard (e.g. 2.3.*) resolution that modern dependency systems can handle, but it’s better than nothing.

Plugins
Actual developer-enhancing plugins can keep doing whatever it is that they’re doing. I’m not against plugins, I’m just against abusing plugins for library distribution.

Thanks for reading. What are people’s thoughts? Do these concerns make sense?

Max

2 Likes

Well, we switched to gradle for core and planing to switch the contrib repo (also to Github I guess). This hasn’t be done yet because of the lack of spare time in the team.
I guess once it’s done, you’ll be able to have your own repo on github and have it has a plugin.

There has been work done to automatically turn a github repo into a plugin in the contrib repo. The advantage of the centralized repo is that it’s built nightly and available as an update the next day in the SDK. Maybe @jayfella can tell you more about it.

We are really promoting our SDK because we feel it’s a huge part of the JME experience. I know some hate it, because of this or that, but anyway haters are gonna hate.

We plan to distribute JME as a “classic” dependency through gradle. There has been some initiative to put it on maven by a third party, but we (the core team) don’t want to do it because we can’t afford to support it, because we just don’t know enough about maven.

For now we don’t even have a nightly properly working on gradle. From a user point of view, the switch to gradle just broke some useful features of our distribution system and that’s a shame. From a core dev point of view, gradle can be just what we need, but there is a huge amount of work to get it done properly.

Now since you seem to have some experience in it, if you want to lend a hand, we’d gladly accept your offer :stuck_out_tongue:

The advantage of plugins is that you can supply SDK extending functionality right along with the library, you can’t do that with maven or any other library distribution system. That is SDK editors, wrappers for the SceneExplorer and even manual entries. Even if some libraries don’t do that the possibility should be there. Plus yes, the NB plugin system is already used for that so I don’t think its “fundamentally wrong”.

And nothing stops you from releasing your lib as a jar only or on maven. It just doesn’t get exposure and is in a huge sea of other java libraries (most of which are only relevant to b2b and web development because thats javas main use) and some jME users might wonder why they can’t just install the library in the SDK. Note that not all users (especially schools) want to download everything from the internet each time they make a new project. The plugin repo is a way for us to give the user a set of extensions for his gaming SDK. As nehon indicated, we see jME more as a development platform than a simple java library (see multiplatform distribution etc) We can only supply the user with such features by taking the build process in our hands as well, leveraging IDE features.

Btw, the build process is not “embedded in the IDE”, its a normal ANT build. You can use it in any IDE. Note that ANT (still) is the original java build system, not maven, ivy or gradle.

And please don’t misunderstand this, we see your points, theres no need to try and convince us of what you are convinced of or explain us why doing something else might make sense, its not like we’re stupid or against progress, believe it or not there can be other reasons for us not taking every idea and making it real. Also see this related post:
http://hub.jmonkeyengine.org/forum/topic/maven-hosting/page/9/#post-296024

@nehon, @normen, thanks for taking the time to provide thoughtful responses.

So far, almost everything I’ve seen in jME has been great. I wouldn’t be able to get nearly as far as I have on my own or even using LibGDX (which targets a different group of game developers, IMO). It’s just that what I perceive as the lack of a solid dependency management system seems like the biggest omission to me compared to everything else.

For a bit of background on me, in case it helps – I work as a developer for a Fortune 100 company for my day job. Having reproducible and reliable builds is critical where I work – which is why it’s important to me to be able to use versioned software in general. That said, it sounds like you have your reasons for doing things the way you are and no amount of talk from some newcomer who’s just as likely to disappear off the forum tomorrow is going to change that.

I think for now I’ll try to make do the best I can by bending the available tools to my whim, and if I come up with something novel and undocumented perhaps I’ll write a wiki guide and post it, probably with a disclaimer that it’ll void your jME warranty if you switch IDEs or JDKs :).

@nehon:
re: github -> plugin repo – going to be a bit of a bastard here and say I’d love to see that happen, but not so much that I’ll be doing that myself. Perhaps if I ever want to vend a plugin, then I’ll help make it happen. For now I’ll just make local copies of the plugins I need…
re: “we don’t even have a nightly properly working on gradle” – while I’ve only spent an hour or two fiddling with Gradle so far, I might be able to lend a hand here…I’ll experiment and see how far I get.

1 Like

@nanodeath , take a look at the discussion (http://hub.jmonkeyengine.org/forum/topic/maven-hosting/) linked by normen, there are several “workaround” scripts to deploy jme 3.0 into a maven repository (local cache or server).
If you want to work border-edge with jme 3.1 the ‘gradle install’ command do the job (at least for me, no native build).

I lend a hand to help, too.

@nanodeath said: @nehon, @normen, thanks for taking the time to provide thoughtful responses.

So far, almost everything I’ve seen in jME has been great. I wouldn’t be able to get nearly as far as I have on my own or even using LibGDX (which targets a different group of game developers, IMO). It’s just that what I perceive as the lack of a solid dependency management system seems like the biggest omission to me compared to everything else.

For a bit of background on me, in case it helps – I work as a developer for a Fortune 100 company for my day job. Having reproducible and reliable builds is critical where I work – which is why it’s important to me to be able to use versioned software in general. That said, it sounds like you have your reasons for doing things the way you are and no amount of talk from some newcomer who’s just as likely to disappear off the forum tomorrow is going to change that.

I think for now I’ll try to make do the best I can by bending the available tools to my whim, and if I come up with something novel and undocumented perhaps I’ll write a wiki guide and post it, probably with a disclaimer that it’ll void your jME warranty if you switch IDEs or JDKs :).

@nehon:
re: github -> plugin repo – going to be a bit of a bastard here and say I’d love to see that happen, but not so much that I’ll be doing that myself. Perhaps if I ever want to vend a plugin, then I’ll help make it happen. For now I’ll just make local copies of the plugins I need…
re: “we don’t even have a nightly properly working on gradle” – while I’ve only spent an hour or two fiddling with Gradle so far, I might be able to lend a hand here…I’ll experiment and see how far I get.

But what dependencies would you have to solve? Everything is included and the plugins are compatible to the version you can download them for. E.g. you cannot use NiftyGUI 1.4 with jME as the adapter classes are not compatible but you don’t get that issue because the compatible version is included. I can imagine that one thinks “Oh its java, I know java, but why doesn’t it use the typical maven system to download jar files” but my question would be what your actual problem is.

Would the problem be catered for if you had the option to enter a package name of some external library (not one that is included already) in a jME SDK projects settings which then is pulled from maven? I somehow have the feeling its just for the sake of it?

Btw 3.1 isn’t border edge, its borderline using that. It WILL break BIG TIME, SOON.

@david.bernard.31
Thanks, I will. I was a bit turned off by the prospect of going through several pages of a thread looking for a workaround, but if you tell me there’s one there, I’ll check it out. To be honest, I’m not super interested in using the tip of jME development – I’m okay with using the stable 3.0 stuff. I’m more interested in the versioning of the plugins and libraries. That said, versioning jME core would help too, if you ever wanted to release a 3.0.1 patch, for instance…

@normen
You’re asking if this is a theoretical problem or a real one, this dependency situation. This actually is a real problem I’ve come across. I pulled in Guava so I could use ListenableFutures and Optionals, which (in my particular case) required downloading the JARs, creating a “library” dependency manually in both Netbeans and IntelliJ, and adding the downloaded JARs into that dependency. Sort of a pain when it could just be adding one line to a Gradle file! I also wanted to use Logback instead of j.u.l., so I had to repeat the process again there. I’m not sure how many other libraries I’ll be pulling in, perhaps just those two, but it’s still not a process I’d wish on anyone else, and if anyone wants to work on my project with me (or if I want to switch computers), there’s a chance I’d have to do it again on every machine I use.

For the Cubes plugin specifically, a bunch of backwards-incompatible improvements were made recently, but there’s no way for me to “pin” the version I’m on to prevent my IDE from updating to that version when it’s released other than to not update plugins, nor for the author to only release the update as a major version bump, which communicates that APIs will be breaking. As a workaround, I just made a local fork of the entire library so I could be insulated from future problems (and make tweaks). Doesn’t seem like a good solution, though. Because I’m using git, and the plugin repository is a monolithic SVN repo, there’s really no way for me to replay my changes on top of the upstream code. Sounds like that situation might change in the future, which will be nice.

As I think I mentioned above, while installing a plugin representing a library in the jME IDE is easy, for me I have to install it in Netbeans, then I have to find the JAR on disk so I can manually add it to IntelliJ. I know, it’s my fault for using IntelliJ, but it’s still an avoidable problem.

I know you didn’t ask, but in a perfect world, it seems like the dependencies for Nifty and its corresponding jME plugin would be organized into three parts. One for Nifty, one for the adapter, and finally the actual plugin. Nifty could be something like nifty-1.3.0.jar, the adapter might be nifty-jme-1.3-1.jar (the 1.3 corresponding to the version of nifty, the last -1 would be for versioning the adapter itself), and then the plugin (just responsible for generating XML, effectively) would be compatible with any version of Nifty, so long as you configure it to generate XML for the version you’re using. Or there could be a completely different plugin for Nifty 1.3 and 1.4, doesn’t matter.

Glad to hear 3.1 is coming out soon! Though I’m not exactly sure how that’s relevant to the discussion :slight_smile:

FYI, today I’m playing with bintray (to host my first gradle plugin). It seems to be possible to create public repository. So I’ll explore how to host an UNOFFICIAL/community driven repository for jme, extension,…

More info soon.

@nanodeath I said that the development version of 3.1 will break soon, not that its out soon. And I say that because david suggested that you are on the “border edge” by using the development version of jME while in fact it will give you BIG headaches of the type you already complain about. One of the next commits might completely break your app if you use the development version.

Anyway so you basically just want a way to more easily integrate external libraries that are available though maven. Heres a doc on that: FaqMavenAntTasks - NetBeans Wiki Using this method you can “just add one line” as well.

As for Cubes, the developer would have to make proper versions in maven too, its more an issue of how the developer organizes (breaking) changes or if he makes versioned releases at all. Normally the versions in the plugin center should be compatible to the jME version that can access them and not introduce breaking changes but you have to see that some of the plugins are in development still. However you can circumvent your actual problem by following the solution for the next problem (see below).

Furthermore, you don’t need to copy everything to IntelliJ manually, just set the SDK to store the needed jar files in the lib folder (Project preferences → Libraries), then IntelliJ can find all jar files though the ANT build file, just open the project as an ANT project in IntelliJ. You will have to remove and re-add libraries that have been updated through the plugin system though but that could help with your other issue (see above).

As for Nifty, jME 3.0 is only compatible with the version it ships with, the adapter would have to be called “nifty-1.3-jme-3.0” if we did what you suggest and that would mean much more work in maintaining and updating all the possible version combinations… I don’t think thats a good idea.

So as you see, maybe it makes more sense to ask how you can circumvent actual problems you have instead of suggesting that everything is changed to a development workflow you are used to or know from somewhere else :slight_smile:

@david.bernard.31: neat, keep me posted :slight_smile:

@normen: Sorry, I interpreted “breaking soon” as in the “breaking news” sort of usage. I wasn’t expecting you to say that something that wasn’t released could break, so thanks for clearing that up. I’m also confused by the usage of “borderline” in this thread, and I think by “border edge” I think bleeding edge was meant. Anyway, enough about word choice.

I really have no interest in converting my project into a Maven project – anything that involves more build logic encoded in XML is out. I’m only interested in consuming Maven artifacts…since that seems to be the main/only generic way for library distribution in Java.

Re: Cubes, I agree that he’d need to do versioning regardless of whether he was using Maven or not, but because it’s distributed as a Netbeans plugin, he has no choice – only release breaking changes as often as jME has releases. That seems unfair. Library developers obviously shouldn’t be making breaking changes often, but forcing them to wait months or years to release a change seems unreasonable.

Dumping a collection of JAR files into the lib directory and scooping them up in the other IDE seems dubious to me, but perhaps a reasonable short term solution, if I can figure out how to integrate Gradle into Netbeans. I hadn’t seen that option before; thanks for pointing it out.

Re: Nifty, if we only include versions of jME and Nifty released in the last two years, that would just be nifty-1.3-jme-3.0.jar and nifty-1.4-jme-3.0.jar – hardly unreasonable (assuming Nifty follows some semblance of semantic versioning best practices). And when jME-3.1 comes out, you could always not update the nifty-1.3 adapter, so it’d just be nifty-1.4-jme-3.1.jar. Which you’d probably have to create for the new release anyway, even without this proposed versioning system.

Re: “So as you see”…first off, you didn’t really clarify anything except to give a couple of semi-workarounds, certainly nothing to warrant a know-it-all condescending tone. For another, I wasn’t going by the hideous version management system I know from work – I’m going by how normal projects with Ruby/Bundler, or Node/NPM, or Android/Gradle are organized. SLF4J has a successful pattern going for them with the API JAR and the binding JAR and I thought it would make sense to do something similar for Nifty, that’s all.

@nanodeath said:

certainly nothing to warrant a know-it-all condescending tone.

I think that’s just his German accent. :wink:

@pspeed: ha, ah…can’t tell to what degree you’re joking :wink: I’m just looking for some validation that just maybe, at least part of my constructive criticism is valid. So far most has been met with a “haters gonna hate”/defensive mentality instead of “potentially interesting idea to explore eventually” or “we don’t have time right now” or “we looked into that before, here’s why we’re not doing that” with a link. I’m trying to have an honest discussion and get feedback about some things that could make the project better and end up getting shut down, repeatedly. Not fun.

@nanodeath said: @pspeed: ha, ah...can't tell to what degree you're joking ;) I'm just looking for *some* validation that just maybe, at least part of my constructive criticism is valid. So far most has been met with a "haters gonna hate"/defensive mentality instead of "potentially interesting idea to explore eventually" or "we don't have time right now" or "we looked into that before, here's why we're not doing that" with a link. I'm trying to have an honest discussion and get feedback about some things that could make the project better and end up getting shut down, repeatedly. Not fun.

A similar discussion has come up every couple months for the last three years. So it’s well trod territory and some of the answers will be a little short. Also, while I’m a big producer of text-walls myself, I’m not sure many will read the whole thing. (I know few read all of my posts.)

Personally, I’ve developed in both environments: “lib/jars checked into SVN” and “full up maven nexus repo with all artifacts”… and I’ll let you guess which one had 100% reproducible builds and which one didn’t. I appreciate maven artifacts in the wild, but I’m not saddled to them, either. Every gradle project I ever create has a ./lib folder added to the dependency mappings so I can shove non-maven-repo jars in there as needed. (Actually, it really bugs me when the only way a project gives you to get jars is with maven and then I have to manually cut and paste URLs into my browser to grab the jars if I just want them for some reason, ie: JME-based projects. :))

Publishing artifacts to a maven repo is not any core developers priority.
a) we don’t really have time to deal with it (evidenced by the fact that gradle conversion isn’t even done)
b) we have a system that already works fine enough
c) over plugins, it doesn’t really solve the versioning issues. (Projects who want to support 3.0 and 3.1 will still have to produce two versions… as far as I know nothing about the plugin system prevents this at all.) It may be a little cleaner in POM but the major work is still entirely on the plugin developers shoulders… which button the push to publish is largely irrelevant.

@nanodeath said: @pspeed: ha, ah...can't tell to what degree you're joking ;)

P.S.: I was being humorous but I wasn’t joking. 9/10 times if you think normen is being mean then it’s really just being blunt. I think it’s safe to say that’s a very German thing and it can be helpful in putting the proper spin on things. I can tell you that in offline discussions it cuts through a lot of fluff.

@nanodeath: Exactly, David implied that you’d be on the bleeding edge, I tell you that its borderline to do that because its NOT the bleeding edge, its a building site and stuff is gonna hit your head.

You didn’t seem to read the wiki entry I linked. You don’t convert to maven, you add this to the normal jME BasicGame ANT project and then you can consume maven libraries. Maybe could add this to the BasicGame by default as to not confuse people who are used to maven.

The point of maven is not that somebody resolves the dependencies like we do but that maven does that, as we already do that adding maven to this mix would only get us more work to do, as in the Nify example.

If somebody wants to release a breaking version of a plugin he can do so as a separate plugin, e.g. “Cubes v2”. Thats not “unfair” I think.

And yeah, I don’t mean to be condescending but if I wanted I could take your “teachings” on how maven works (which we know) as condescending too. I mean you come to people who released a software used by tenthousands and tell them how to set up their build and dependencies… :slight_smile:

FYI: I start an other topic about jme3stuff-a-dedicated-maven-repository-for-jme3

Sorry for duplicate post, but I didn’t know if Max subcribe to topic Maven Hosting

Due to circumstances beyond our control theres now an official maven repo for jME3 available, see this post: http://hub.jmonkeyengine.org/forum/topic/official-maven-repo-for-jme3-0-stable-available-please-test/

Sorry for the late respone, but as has been already said before, plugins are being automated but at this moment in time there isnt enough data to test it thoroughly: http://hub.jmonkeyengine.org/plugins/

In a nutshell - you submit your repository, one of the team will either accept it or reject it, and that it, job done. It has all been written but just needs more projects of various types to test all of the various results.

3 Likes