Build systems & development conventions

1 - What have we got so far

2 - What more do we need / what needs to be changed

3 - Is there any reason to treat jME2/3 differently in this respect?



Another useful read would be the thread that discusses unified build process.

sbook said:

...  I'd say any issue tracker will beat out the unused one described in Mark's sticky here: http://www.jmonkeyengine.com/jmeforum/index.php?topic=370.0


I've edited the sticky to point to the correct issue tracker.

Here's my best answer to the questions:



We've got a powerful engine, another one in development, and a user base.  The problem here is that what we have as far as support on the site vs. people using the engine in the wild seem to be two different things. I often see first or second time posters who seem to know a good deal about the engine and it's workings just popping in to ask a question…  That seems all well and good, but those are folks that could be on the forum helping out and being helped.  That of course leads into what I think we need:



I think we need a more present user base where more folks are cruising the forums and wiki throwing out tidbits of advice and helpful hints.  We've got such an awesome 'User Code' section that I'm sure a lot of us find useful that can be A: more helpful to others not on the forum and B: larger due to the larger support/user base.  This would also lend itself to what we really need:  A more consistent form of documentation…  There's a lot of good stuff floating around, but i think a lot of those "hot technologies" that people want require more digging than should be necessary (i.e: video display, GUI, etc)…



Should jME2 be different from jME3?  Well assuming we're following the roadmap from another thread saying that jME2 would continue community support while jME3 is in active development, I think we should make a concentrated effort to make some really good documentation on the workings of jME3…  jME2 has a really good PDF floating around explaining the scene graph… We need more of that…





And that's my not-so-concise piece

I am also completely in the dark about what these different build systems really offer. Here some things that puzzle me:



Both Maven and Hudson are apparently used for builds, but there exists tight integrations between the two: What are the core differences between the two in terms of use, and how would a tight integration be beneficial? (I suppose maybe it’s got to do with Hudson’s automated tasks?)



Ivy is apparently not a build system, but they claim that when combined with Ant, it outmatches Maven. So that’s another option to consider… But how important is dependency management for jME compared to build management/system(s)?



If you combine And & Ivy, is it still easy/necessary to apply Hudson on top of that? If you take Ant out of the picture, will Ivy and Hudson together still cover every need?



Then there’s Buildr, which represents an interesting alternative (there is also the very similar Raven which seems to have been discontinued some years ago). Pure ruby, it claims to match the feature-set of both Ant and Maven. I don’t know how it matches up against Ivy + Ant though (I don’t see them mentioning dependency management). One significant drawback is the lack of official support with Hudson, but I believe the two can still work together with a little bit of extra work done.



Could Hudson by itself serve every need in (dependency) management and perpetual builds? Also, why are automated builds (Hudson) in such demand? Why is it such a necessity for this project? What problems would it solve?



What if we have a look at one of our neighbors and see what Ardor3D is doing? :smiley: I know they have Hudson deployed; anything else in use, like Maven? Are they satisfied with this solution?

All very good points on development convention skye, but what about build systems?



I need some answers from people with in-depth overview of jME's current organization and current tools in use. I believe Maven has been deployed for jME2; are there other tools or special approaches (best practices) in use that are not immediately obvious?



If new developers find it strikingly apparent what to do once they've pulled the source and compiled it, well then probably nothing should change.

erlend_sh said:

I am also completely in the dark about what these different build systems really offer. Here some things that puzzle me:

Both Maven and Hudson are apparently used for builds, but there exists tight integrations between the two: What are the core differences between the two in terms of use, and how would a tight integration be beneficial? (I suppose maybe it's got to do with Hudson's automated tasks?)


I am not a Hudson user, but it looks like it's strengths are server-side.  For automating unit testing, distro pushes, clover reports, etc.  I welcome a Hudson expert to correct or refine what I say, but I think your average user would not install or run Hudson, though they could benefit by the outputs, such as regular reports, regression tests, up-to-date distros, etc.


Ivy is apparently not a build system, but they claim that when combined with Ant, it outmatches Maven. So that's another option to consider... But how important is dependency management for jME compared to build management/system(s)?

If you combine And & Ivy, is it still easy/necessary to apply Hudson on top of that? If you take Ant out of the picture, will Ivy and Hudson together still cover every need?


It would not make sense to use Ivy for jME without Ant.  UPDATE: I stand corrected about Maven and Ant.  The Maven compiler plugin no longer uses Ant.  You used to have to monkey with Ant and Jelly to customize Maven compilations.

Then there's Buildr, which represents an interesting alternative (there is also the very similar Raven which seems to have been discontinued some years ago). Pure ruby, it claims to match the feature-set of both Ant and Maven. I don't know how it matches up against Ivy + Ant though (I don't see them mentioning dependency management). One significant drawback is the lack of official support with Hudson, but I believe the two can still work together with a little bit of extra work done.

I wouldn't waste time with anything that hasn't been maintained in a year.

I would only consider the most popular tools, because people are much more willing to learn tools that they have heard of and expect to come in useful for future professional or leisure work.

As this is a Java project, I would discourage from using a tool which requires knowledge of another language to adjust, including Ruby and even Bourne shell scripting.  You'd have to justify any extra skill set beyond Java and Ant.


Could Hudson by itself serve every need in (dependency) management and perpetual builds? Also, why are automated builds (Hudson) in such demand? Why is it such a necessity for this project? What problems would it solve?

What if we have a look at one of our neighbors and see what Ardor3D is doing? :D I know they have Hudson deployed; anything else in use, like Maven? Are they satisfied with this solution?


They use Hudson on their server.  If you want to work with Ardor3D on your workstation, you get the source and build with Maven.

As you sort of understand, Ivy is for managing libraries and distribution management.  Maven does everything that Ivy does plus a bunch of other stuff.  jME could benefit a lot by proper, declarative library management.  Get the binary libraries out of Subversion and allow developers to specify which jME subproducts they need, and which third-party products and versions they need.  Example:  A FPS may need jME core, terrain, audio with the latest LWJGL.  This app should not be loaded down with everything in jME, JOGL, etc., and their builds should not change because somebody at jME decides to update a thirdy party binary library in svn.

Maven adds on tons of higher-level functions, but you don't have to use them.  It can automatically download everything needed for an Ant build, generate distributions and docs, etc.  Maven can be easy if you convert everything to comply with its default expectations for directory layouts, annotation assumptions, etc., but can cause a lot of extra work if you want to customize it.  I don't see much in Hudson that Maven can't do, but I think it has Gui management features and has a real,dedicated scheduling system.

I've redesigned a lot of build systems.  I go with the simplest system that can accomplish the goals.  Are there problems with server-side builds because of changing library dependencies?  Are the Sourceforge downloads stale because the jME Sourceforge Admins don't have the expertise to do proper builds manually?  Don't answer.  I'm just trying to point out that if a new system doesn't satisfy a need, then implementing it will just add complexity and will narrow your potential administration base.

Hi everybody! Hey Erlend!



Hudson is a Continuous Integration "tool". The main purpose would be to automatically build the code base of a project in a given interval, usually once a day for big projects, the so called "nightly builds". That also includes running the tests, producing the documentation, updating a website doing some checks and statistics with Quality Metrics tools and also publishing the build to maven repositories or ivy or whatever you choose. It also connects to issue tracking systems. It forces the project to be stable and helps by having a working updated build always, preventing "innocent" commits that break the build go unnoticed. It's really not hard to setup, it might not be quite clear right now if it's worth but trust me, it will keep many doors open and in the long run it will prove useful.



I personally don't know about Ivy, Buildr or Raven. I'm sure Hudson strikes a good balance between features and usability and I survived the Maven hell and came back to tell the tale.



Yesterday with the help of some people who made a custom Maven plugin to handle natives and an Eclipse plugin to further integrate it I managed a supreme configuration of a maven project, capable of handling natives as a dependency hosted on a maven repository, creating a deploy ready to run in unix and windows and integrating the project seamlessly with Eclipse and Netbeans allowing running or debugging the project or individual files without having end-user configuration. Yes, it works, seriously.



For the end user who wants to work on jme for example it would only mean to checkout the code open it with the IDE and compile it and run it as she wishes. For projects depending on jme it would mean adding a dependency to jme in the maven config file and that would also be it. No configuration or natives handling.



Actually for Eclipse one would have to install the svn plugin, maven plugin and the maven-natives integration plugin, for Netbeans it just works (I will let you decide which is best ;)), Intellij IDEA remains to be tested but considering I would have to pay for it or make a run during the 30 days trial I'm not sure it's worth it.



I've offered my help on this matter before in this forum and I'm also helping out with Ardor, if you consider this useful I will gladly help out otherwise I really don't feel like trying to convince people that this is a good idea.



PS: Last post came just when I was about to post this. BTW I'm pretty sure Maven doesn't use Ant, it could use it if we as users needed it but doesn't depend on it. As clarification Hudson can be configured to issue commands to Maven, so Hudson would be the director that also directs many other tools, maven would be the executor that actually does stuff. Both are not interdependent but interact quite nicely.

@blaine, Tomygun: Thanks a lot for your input guys. Tomy, glad you could make it ;D

blaine wrote:
I wouldn't waste time with anything that hasn't been maintained in a year.
Well, according to the changelog the latest release actually dates back to late April 2009. Not that it really matters much, because it seems largely unknown anyways, which equals unsupported, like you said.
Tomygun wrote:
I've offered my help on this matter before in this forum and I'm also helping out with Ardor, if you consider this useful I will gladly help out
Maven is what we got and so far there seems to be no one urging to reconsider. I would like to give it a little more time to give more people a chance to speak up, but yeah, assuming there's no change of direction, your help would be greatly appreciated.

Assuming we go on with maven though, there are some new points to consider. We got two main branches, namely jME2 and jME3. I will assume Maven would have to be deployed separately for these. Based on that assumption:

- Would two deployments mean double the burden of maintenance?
- Maven for jME3 can be deployed from scratch whereas jME2 got it already, however without this handy new plugin. If we want to upgrade it, would it just be a matter of applying this new plugin to what we got set up with jME2 already?

- Would Hudson be installed on our own webserver?
- Could we have one installment of Hudson to govern both branches, or would we need two of that as well?

i also think we need:

  • maven repo with those artifacts
  • hudson to have nightly builds of jme2 and 3
  • hudson plugins (checkstyle, sonar) to monitor source quality
  • trac is nice, but the google issue tracker would fit also i think.



    I don't know why there should be a switch to Mercurial, svn seems to work fine :slight_smile:
erlend_sh said:

- Would two deployments mean double the burden of maintenance?


Well, yes and no, they will probably use pretty much the same configuration. The thing with maven and hudson is that the hard part is making it work. After that it shouldn't require too much maintenance.

erlend_sh said:

- Maven for jME3 can be deployed from scratch whereas jME2 got it already, however without this handy new plugin. If we want to upgrade it, would it just be a matter of applying this new plugin to what we got set up with jME2 already?


"Mavenizing" a project is done through a single file, the pom.xml. Adding a plugin to a maven project is just adding a few lines too that file and then maven will do it's magic when the build is executed and download the plugin and all of it's dependencies recursively. I already have poms working and completely figured out for Ardor and Slick projects (both use lwjgl as jME does). Adding it to jME2 would only require updating that pom.xml.

On the other hand, Eclipse plugins should be handled by the end user, it's quite simple to do and most user should be used to it. They are forced to add SVN plugins because they don't come out of the box. Just to be on the safe side we should make a tutorial but it should be quite easy. As of the recently released Netbeans 6.7 maven is already installed and no other plugins are necessary, I managed to configure it through a companion file to the pom, the nbactions.xml that should be added to jME repository.

erlend_sh said:

- Would Hudson be installed on our own webserver?


Yes, Hudson should be installed as well as a Maven repository manager to host jME builds, Nexus seems the best chocie.

erlend_sh said:

- Could we have one installment of Hudson to govern both branches, or would we need two of that as well?


Yes, Hudson has this things called "jobs", each job is an independent unit of work so there won't be any problem, sometimes you may even have more than one job for a single project. Hudson "connects" to maven to figure out what to do so most of the work would be in the poms.


As I said before, maven is only one file, if any user doesn't like using maven, even though I think we solved all the problems and there are so many benefits, they can just ignore it and do it the way they like.

If later on we want to use Quality Metric tools like Checkstyle, PMD, FindBugs, or test tests coverage with Cobertura or connect to Issue Management tools like Trac or Bugzilla a little work should be necessary either on the maven config or Hudson. These tools should be discussed if jME wants to be as professional as possible.

I will begin working on an archetype, which is a maven "thingy" that bundles a set of files and can magically setup a new project, it's a project prototype that can be used when creating a project that uses jME. The average user won't know how to do a complete maven configuration for a brand new project but by providing an archetype they can simply set the name of the project and see it magically unfold into a completely configured project that will automatically download jME and all dependencies and build into a working example that can be bundled into a zip ready for deploy. This would also eliminate all the lengthy setup tutorials and the hundreds of forums posts asking for help with the setup.

EDIT: I would gladly do the configuring of Hudson and the poms since I already have devoted lots of time to maven and read a couple of book on the matter.

I don't think Maven or Hudson would be poor choices either way.  One thing that has kept me weary of auto-build systems, as Erlend knows, is cross-platform compatibility.  The beauty of jME, of course, is that we're building for Java and don't need a ton of different build hardware floating around.  The idea of a Trac of Bugzilla system I think is an essential need as the project grows (which is an aspiration that I believe all hold at jME), such to say that assuming Maven doesn't miss out on any huge features offered by Hudson (or vice-versa), I think ease of integration and maintenance with an issue tracking system should be a central concern.  Let's also remember that there are snapshot builds for most while auto-builds are there for whoever is up to it, so much to say that the auto-builds will probably have enough bugs as is, let's make sure the build-system doesn't become it's own section in the troubleshooting sub-forum…



That all said, I think this offers great potential (and forces more consistency) and have been impressed by such a system before (cough, Maven, cough)

Tomygun said:

Yes, Hudson should be installed as well as a Maven repository manager to host jME builds, Nexus seems the best chocie.
Got it. I checked out the available options and it seems like Nexus is pretty much a no-brainer pick. I guess Skye will get to work on adding both when the time is right (and by that I mean probably after the site migration) and he can consult with you should he encounter anything unfamiliar, yeh?
Tomygun wrote:
I will begin working on an archetype (...) This would also eliminate all the lengthy setup tutorials and the hundreds of forums posts asking for help with the setup.
This sounds like a brilliant idea! I love everything that makes jME more accessible. The lower the entry-point the better!

EDIT: I would gladly do the configuring of Hudson and the poms since I already have devoted lots of time to maven and read a couple of book on the matter.
I can give you administrator access on google code if needed. As for Hudson, will you need server access (might be tricky) for that or could Skye (sbook) just set you up with an admin account for Hudson and Nexus if he's able to install them by himself?

sbook said:
The idea of a Trac of Bugzilla system I think is an essential need as the project grows (which is an aspiration that I believe all hold at jME)
No doubt about it on both accounts. I have much more experience in this area than with build systems and whatnot, so I'll have much more on my chest for this one ;) Will bring it up once some of these other tasks at hand have settled.

Big question: Would a lot have to change if we decided to switch from SVN to Mercurial? I've read enough about revision control to understand that SVN is a thing of the past, and a DVCS like Mercurial is the future. Now that Mercurial is coming to Google Code, I think we should be the first to jump on the wagon. They seem to have stalled it for quite some time as it's not gone public yet, but when it is it should already be very stable and ready for large scale adoption.

So, say we set up Maven and Hudson all done and ready for use once the migration is done, and a couple weeks after that Mercurial goes public and we want to make the switch. Would that cause some problems with the configuration? (Btw I know Hudson & Maven are perfectly compatible with Mercurial aka Hg already)
erlend_sh said:

(...)
I guess Skye will get to work on adding both when the time is right (and by that I mean probably after the site migration) and he can consult with you should he encounter anything unfamiliar, yeh?
(...)
I can give you administrator access on google code if needed. As for Hudson, will you need server access (might be tricky) for that or could Skye (sbook) just set you up with an admin account for Hudson and Nexus if he's able to install them by himself?


Sounds good. It's up to Skye I guess :D. SVN access would be good so I could start a branch for testing and work with it until ready. Hudson and Nexus admin accounts should do. I'm guessing server access is out of the question for obvious reasons.


About Trac vs Bugzilla vs Google Code. Integration with svn, hudson, IDEs.

Trac works with SVN well and Hudson should be no problem either as for IDEs Netbeans doesn't and Eclipse perhaps?
Bugzilla has it all.
Google Code no Netbeans and the rest really no idea :?.

BTW will code be still hosted on google or will you setup svn in the server?


About Mercurial, big NO!  :D. Let me explain. When Netbeans code base migrated to Mercurial I thought the same as you do "awesome! some really cool new thing that beats svn" so I switched over to mercurial. It's a great tool but it's great for really big projects with many parts and many teams working on different parts. When working with a repo you have to make a local repo, then you work with that local repo as you would with a svn one, committing, updating etc, you can do whatever you want with that repo but then you have to commit your repo to the official repo if you want everyone to see your changes. For small projects that you just want to commit what you did you end up doing 2 steps every time. For bigger projects where lots of people work long time until a module is complete and when it's absolutely functional it's committed to the official repo is great. Otherwise I don't think it offers any advantage over SVN.

Please correct me if I'm wrong about this and if there is a way to bypass the 2 steps. This was my experience at work a year ago and we ended up switching to svn.

EDIT: Agree with Core-Dump

No idea on Bugzilla, only worked enough with Trac. If we are going to use google code the I guess using google tracking system should be ok.



Hudson it is apparently, no one seems to have another opinion. Let me know when it's ready so I can tinker with it }:-@.

Gonna be setting that up this weekend :smiley:



hardware is/will be a Pentium-D box with 2 gigs of RAM for the sole purpose of jME server(website, forums, Hudson, et al)… You've got an e-mail inbound tonight concerning the setup of Hudson



As for Google Code… It's issue tracker seems to work, and we know it's SVN definitely does.  I'd say any issue tracker will beat out the unused one described in Mark's sticky here: http://www.jmonkeyengine.com/jmeforum/index.php?topic=370.0

sbook said:

As for Google Code.. It's issue tracker seems to work, and we know it's SVN definitely does.  I'd say any issue tracker will beat out the unused one described in Mark's sticky here: http://www.jmonkeyengine.com/jmeforum/index.php?topic=370.0
What do you mean? The issue tracker described there is the google code issue tracker. For a stand-alone tracker I would recommend Bugzilla, because it does bugs only, and does it damn well. I've worked with Trac and it's okay, but has many more features than what we need. I still don't know if either of these can be hooked up with remote repositories. Anyone got any idea?
erlend_sh said:

sbook said:

As for Google Code.. It's issue tracker seems to work, and we know it's SVN definitely does.  I'd say any issue tracker will beat out the unused one described in Mark's sticky here: http://www.jmonkeyengine.com/jmeforum/index.php?topic=370.0
What do you mean? The issue tracker described there is the google code issue tracker. For a stand-alone tracker I would recommend Bugzilla, because it does bugs only, and does it damn well. I've worked with Trac and it's okay, but has many more features than what we need. I still don't know if either of these can be hooked up with remote repositories. Anyone got any idea?


Look above you, Cristoph already fixed the link ;)  I couldn't find a definitive answer for Bugzilla, but Trac does not..
Nothing I have read about Bugzilla tonight leads me to believe that it would be able to either though..  We could probably use svnsync, but I think that may create only a read-only mirror...  that could be ok though, since we could just commit to google and svnsync would update the bugzilla accordingly..

That all said though, that's a pretty big run around... and I think you'd need a certain version of svn (and possibly/probably the same) or higher on both servers.  We could suit our own end, but I'm not sure what Google's on (I'm sure they're on a relatively recent release)

Yeeeh… Major run-arounds like that ain’t too tempting. There’s this new bugtracker in development called JotBug which I’ll inquire more about. I know for sure the developer is highly competent, as he’s a former developer of Mambo, the CMS Joomla! branched out from.



In any case, I think we’ll be better off trying to push Google for additional features to the tracker should we need them. One particular feature-hole I’d say is being able to set dependencies between tasks. I’m sure there’s a lot of requests around for it already, but I’ll check it out.

I appreciate the goal to stick with Google services as much as possible.



If Google can't cut it for bug and issue tracking, I suggest that you look into Jira.

blaine said:

I appreciate the goal to stick with Google services as much as possible.

If Google can't cut it for bug and issue tracking, I suggest that you look into Jira.


Looked pretty cool until I hit this: http://www.atlassian.com/software/jira/pricing.jsp

Good Lord, that's a lot of bread.