Ideas for making JMonkey game engine more unit testable? (e.g. w/ DI)

Hi All,

I would like to discuss ways to make JMonkeyEngine more testable. I’m a TDD guy from the C# world, so I almost never write code without unit tests. The lack of unit testing is a major drawback to most game engines, so I really feel like progress in this area would be instrumental to future growth of this framework. The problem with unit testing seems to revolve around the difficulty with updating the bulletAppState from a unit test method. I’m wondering if dependency injection would be the appropriate way to solve this problem, but I haven’t worked with any of the Java DI frameworks yet. Any thoughts or ideas?

Sorry, I really wanted the title of this thread to state “Ideas for making game engine more unit testable,” but it looks like I cannot modify the title. If there’s a mod out there, please update the title of this.

@jMethyl said: Sorry, I really wanted the title of this thread to state "Ideas for making game engine more unit testable," but it looks like I cannot modify the title. If there's a mod out there, please update the title of this.

Add /edit to the topic url in the browser to edit

And I’d love to hear peoples ideas on this as well.

I also remember considering dependency injection for dealing with assetManager. That was way more complicated than I expected. There needs to be a way to access the manager classes from outside of main, perhaps with the Repository pattern. Thoughts on this idea?

@t0neg0d said: Add /edit to the topic url in the browser to edit

And I’d love to hear peoples ideas on this as well.

I don’t see an option to “Edit” the topic like I see for all of the other posts. Is there something I’m missing?

@jMethyl said: I don't see an option to "Edit" the topic like I see for all of the other posts. Is there something I'm missing?

Your not missing anything… just click here:

http://hub.jmonkeyengine.org/forum/topic/ideas-for-making-game-more-unit-testable/edit

1 Like
@t0neg0d said: Your not missing anything... just click here:

http://hub.jmonkeyengine.org/forum/topic/ideas-for-making-game-more-unit-testable/edit

Thanks!

1 Like

Several things about unit tests. JME Tests results are graphic results, that’s very hard to test with a unit test. Only unit testing we could do is run all the test and see if they don’t crash (I’m pretty sure some tests are crashing…). That wouldn’t guarantee that everything is working as it should.
At some point we had nightly (we’ll have it back) and some dutiful monkeys were testing for us. That was working pretty well.

Now about dependency injection,

  1. the brute force approach is to pass around everything you need from the Application. Say if you have a Manager class, you just make a constructor with all that you need as parameters.
  2. Another approach is to have some kind of AppContext class that you initialize in the simple init and feed it with whatever you need, then make a singleton type static access to its instance so you can access it from anywhere AppContext.getInstance().getWhatEver(). Not completely pretty…but it works.
  3. Else…you can use something like Spring that is the kind dependency injector in java…but IMO that’s just using a nuclear bomb to kill an ant.
  4. My prefered approach is to use AppStates. Appstates have an init method that is given the Application as a parameter so you can grab whatever you need on it (most of the important things have an accessor on Application or on SimpleApplication). then you can combine with 1 if you have classes that are not appStates
2 Likes

@nehon, I agree that Spring is way overkill, and I really don’t think a framework like that would solve the problem anyway.

I fully recognize that graphic results are hard to test. Luckily, however, it is not uncommon for incorrect behavior to still be possible to unit test.

Here are some examples:

  1. If a physical object is passing through some other physical object when it should not, tests for collisions might allow the impossible states to be detected.
  2. The values of vector computations can be checked.
  3. Properties of objects can be tested to ensure that they are not null.
  4. Method calls can also be tested to ensure they return expected values.

The key is that unit tests would need to not only test for correct values but also incorrect values.

The tricky part of designing for testing, however, is to break away from depending on specific classes, such as the Application class. Otherwise, all of the tests will technically still be integration tests, so changes to the Application class can still break tests even when the code is actually working fine.

I have often used seams to break dependencies. A seam is basically an interface that is used by a unit test to decouple implementations and isolate the specific parts of the code under test. In my mind, the key dependency to break is the Application itself. But since almost all of the important methods on the Application class are not defined on an interface, there is no clean way to fake the Application instance. The best we can do is inherit from Application and override all of the methods to provide fake implementations; but this is incredibly ugly, it is still very brittle to changes in the Application class, and it doesn’t cover a significant number of edge cases.

One of the reasons that AppState is so useful is because it is an interface. It would be even more helpful if it depended on something like an IApplication interface instead of the Application class.

Thoughts?

I will instantly revert any interface checked in that has an I prefix. Silliest convention ever… though not surprising that it came from the Microsoft camp… likely as a hold-over form rampant hungarian notation. Either that or we should go through and be total about it. I for interface, C for class, A for abstract class, etc… just carry it out to its ridiculous conclusion. (Sorry, big pet peeve of mine.)

The problem with turning things like Application into an interface is that it’s totally artificial in this case. Sometimes TDD goes one step too far and confuses a class hierarchy just for the sake of some testing convenience. Users are confused enough by these classes as is without introducing artificial interfaces whose sole existence is to facilitate some unit testing of marginal value.

Really, the kinds of things you’d need an Application interface for testing in core aren’t really unit-testable components anyway. As far as user code, A MockApplication that extends Application is not so ugly. Much uglier things have been done in the name of testing things and at least it doesn’t confuse the public API otherwise.

2 Likes

I completely agree with Paul, changing the API with only reason that it will make testing easier is just a mistake.
That’s the part I hate about TDD…thinking “how can I test my API?” before “how can I use my API?” can’t lead to something functional.

I don’t say that tests are not useful, we could have some, but IMO we’ll just test so basic things that we’d instantly detect the problem with conventional testing and usage of the API anyway.

IMO nightly and monkeys is all that we need.

1 Like

Hi, welcome to the community

Yes, JMonkey engine is unit testable even with current implementation!! Just like other Java application…

[ I’m also admit that I’m a fan boy of google java tools because you may later recognize it, let’s name Guava, Guice and Caliper. But here come the goods…]

I see you are a new comer and relatively new with Java tech, JME3 tech so i assume you asking for serveral questions about concepts, how tos and why should i together…

1)What should be tested in a Game for god sake? What is a Testcase look like?
2)How do I make something from outside? How do I inject data, assets, codes and my drugs…?
3)Why should I use this techs but not other.

1)What should be tested in a Game for god sake?.. What a Testcase look like for empty brainer…
Predicates

Predicate is some question you may want to know if your game is or is not, a condition which at one state the game's objects may sastify. Ex: In this area there should be more than ten monsters?

Functions
Functions are Predicates usually work with a formal system. It usually look like if you finish StateA with a condition then should be StateB.
Ex: You may want to know if a you exchange an item for 5$, you should have 5$ in your pocket. After MainMenuState should be InGameState

Behaviours
Behaviours are more attribiary things, which can be composed by several Predicates and Functions together. Behaviour usually need a Coordinate system or a Specification to resolve.
Ex: Your character should turn left if you hit Left key. But it stay still or jump up.

Graphics
Graphics behaviours and graphics aspects are the most difficult thing to be tested.
Ex: Your character should turn blue when you hit the blue bubble.

When it come to graphics, the test case and the way machine (by instructions of developer) actually do to capture the state of the game may be various and pretty ridiculous. For example, if you want to check if your character turn blue right after the hit, you capture the screenshot for the specific moment and save out as name “Check if character is blue”. A Tester (human or machine) later come to the server and just check and report the issue back.

That’s said in general, a Test can be about any thing… but here I will share some of my experiences working in a large scale game issues tracking system, from that you may have an idea of what you want as a result of a Test:

  • The Tests are scheduled incremental with development phases and involve peoples and machines.
  • There is already quite tons of specifications and checklist for a game to check to.
  • The issues if found also be reported back in a big server with images, sounds, description and even data dump to help the tester, developers visualize or reproduce the issue again.

Now talking about code side only, and unit test only, about functions for example:
“Did the StateA change to another StateB with my input?”

-What you may already had are JUnit and Mockito.
You may need Guice for DI (instead of Spring in the name of holy monkey…) and may be also Capiler.

Steps:

@Test
public void stateChangeTest() {

class StateA{
private ActionListener actionListener = new ActionListener() {
public void onAction(String name, boolean keyPressed, float tpf) {

  if (name.equals("changeState") && !keyPressed) { 
       stateManager.remove(StateA.getInstance());
       stateManager.attach(StateB.getInstance());
  } 

}

};
}
SimpleApplication app=new SimpleApplication(“MyApp”){

 public void simpleInit(){
      StateA firstState=StateA.getInstance();
      stateManager.attach(firstState);
      inputManager.addMapping("changeState",     new KeyTrigger(KeyInput.KEY_SPACE));
      inputManager.addListener(firstState.getInputListener(), "changeState");
 }

public void simpleUpdate(float tpf){
     assertTrue (stateManager.hasState(StateB.class));
}

app.start(JmeContext.Type.Headless);
}

This kind of TestCase are quite boring and to some people it’s even ridiculous. What you may see it’s not touch real-time aspects and graphics of a game yet… Later, now just move on

  1. How do I make something from outside? How do I inject data, assets, codes and my drugs…?

First, better get some patterns in JME3 straight.

Hooks:
you should learn that the Application is embeded in a Context and obey a cycle and timing ticks from the SystemListener. Your code better hook within Applications update, State’s updates, Control updates…
https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:advanced:update_loop

Compose things:
A lot of things are Spatials, composed Spatials result in a Node (extends Spatial).
https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:advanced:spatial

Load things:
Assets are loaded via AssetManager via
https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:advanced:asset_manager

Now if you want to do something from the outside. Yes, I’m talking about Dependency injection. DI somethings are cool but not very suitable with Application as the Target.

My “milion dollars advice” is to make your own Context class and inject things into it instead, and later interprops with JME3 system.

[java]
class MyGame extends SimpleApplication{
class Context{
@Inject
EventBus eventBus;
@Inject
LoadingCache<AssetKey,Object> assets;
@Inject
Set<Services> plugins;
@Inject
Set<Manager> managers;
}
}
[/java]
In this example:

I have a working Application which is ready to use and fully function at the time of injects.
I use Guava Cache and Guice to setup assets.
I use Guice to locate Services with Multibindings
I use Guice Scope.Singleton inject to setup my Managers.

Those Guice Module of course using Application facilities like AssetManager, StateManager, InputManager, RenderManager underlying.

Futher more you can rig up Input with Annotations inject, Make Cinematic with Event inject and unit test those…

IF you still insist to wrote up an application which use DI by default, learn an sostiphicated DI framework like Guice, you can see a lot of similaries of abtractions phases of Application with Guice internal phases to rig those two together. If you go along this path, you did exactly what I did with my game framework for large scale game called Atom framework :).

Note: Some JME3 phases may look alike, work alike and abstractional similar to those in DI and IoC phases but not for general Java usage, it work with JME3 specific data types. So you see Bootstrap, Loading, Injections, Composing… with different names only.

Anyway, hope this help you in getting your head around the problem. Nice day.

2 Likes
@pspeed said: I will instantly revert any interface checked in that has an I prefix. Silliest convention ever... though not surprising that it came from the Microsoft camp... likely as a hold-over form rampant hungarian notation. Either that or we should go through and be total about it. I for interface, C for class, A for abstract class, etc... just carry it out to its ridiculous conclusion. (Sorry, big pet peeve of mine.)

I just want to point out that those silly Microsoft people also created C#, a language with delegates, lambda functions, events (as an actual keyword, not just interfaces with a naming convention), the Common Intermediate Language specification, expression trees, LINQ, extension methods, operator overloads, named parameters, nullable types, generic type inference, and async/await, none of which Java supports (with only a partial exception to generic types and a toy version of lambdas). Oh, and by the way, it also seamlessly interoperates with F# (the first true functional language with real-world utility) and everything else on the .NET platform (e.g., WPF, WCF, T4 metaprogramming, etc.). Those silly Microsoft people…

@jMethyl said: I just want to point out that those silly Microsoft people also created C#, a language with delegates, lambda functions, events (as an actual keyword, not just interfaces with a naming convention), the Common Intermediate Language specification, expression trees, LINQ, extension methods, operator overloads, named parameters, nullable types, generic type inference, and async/await, none of which Java supports (with only a partial exception to generic types and a toy version of lambdas). Oh, and by the way, it also seamlessly interoperates with F# (the first true functional language with real-world utility) and everything else on the .NET platform (e.g., WPF, WCF, T4 metaprogramming, etc.). Those silly Microsoft people...

Yeah, they had the benefit of hindsight after trying to sabotage Java for so many years and finally having their butts handed to them. So they made a Java like language that looked a little like Visual Basic and had every feature and its brother thrown in without regard to if it actually made the language better or worse. Sounds pretty much like every single other thing they’ve written. Sorry I Should Have Written That All Like This. It’s The Visual Basic Way And Microsoft Loves It. :slight_smile:

Ooops… I meant:
Sorry, PI should VHave VWritten PThat All Like PThis… :slight_smile: PIt’s The AVisual NBasic NWay…

2 Likes
@pspeed said: Yeah, they had the benefit of hindsight after trying to sabotage Java for so many years and finally having their butts handed to them. So they made a Java like language that looked a little like Visual Basic and had every feature and its brother thrown in without regard to if it actually made the language better or worse. Sounds pretty much like every single other thing they've written. Sorry I Should Have Written That All Like This. It's The Visual Basic Way And Microsoft Loves It. :)

I just want to point out that C# featured generics, the switch statement, lambdas, closures, method references, and delegates before Java added them. If you look it up, you will derive the same conclusion. (Those are just a few examples.)

Here’s a good comparison of creating an asynchronous http request:
How to create an asynchronous http request in Java
vs.
Creating an asynchronous http request using C# async/await.

Here’s an example comparing Java to F#:

"Consider the task of filling a hash table with 10,000,000 associations from integers to single-precision floating point numbers. This may be accomplished in Java as follows:

package hashtablebenchmark;

import java.util.HashMap;

public class Main {
public static void main(String args) {
int n = 10000000;

    for (int j=0; j&lt;10; ++j) {
        long startTime = System.currentTimeMillis();
        HashMap hashtable = new HashMap(n);

        for(int i=1; i&lt;=n; ++i) {
            hashtable.put(i, 1.0f / i);
        }

        System.out.println("m[100] = " + hashtable.get(100));
        long time = System.currentTimeMillis() - startTime;
        System.out.println("Took: " + time / 1e3 + "s");
    }
}

}

The equivalent program in F# is not only shorter but also 17× faster:

let n = 10000000
let m = System.Collections.Generic.Dictionary(n)
for i=1 to n do
m.[i] <- 1.0f / float32 i
printf “m[100] = %f\n” m.[100]

Specifically, Java takes 6.967s initially and 5.733s steady state whereas F# takes only 0.414s.

In fact, F# completes this benchmark so quickly that we would like to give it more work but Java cannot handle any more work without running out of memory on this 4Gb machine" (http://fsharpnews.blogspot.com/2010/05/java-vs-f.html).

Anyway, we’re digressing.

I’m hearing three main reasons for not creating a seam for Application (w/ an interface):

  1. It will confuse people too much.
  2. The benefits do not outweigh the work involved.
  3. There are other ways to test things, though not necessarily in any automated way.

@atomix, thanks for the write-up. I am indeed familiar with JUnit and Mockito. The Guice framework looks very interesting. It appears that Guice can be used for method interception, which may be extremely useful in this situation.

For the other framework you mentioned, are you referring to
https://code.google.com/p/caliper/ ?

I would also like to hear from @normen on this topic. (After all, he’s usually the one to fix the problem anyway.)

Hi,

Dont start the C# vs Java flame war please.

Caliper is for mirco benchmarking… but it also abstract enough to work a basic skeleton for doing incremental tests for performance in real time enviroment.

Instrument. (Definition.) Examples: microbenchmark, macrobenchmark, allocation, footprint, arbitrary measurement.

Measurement. A fact (usually a quantity) measured empirically by Caliper, corresponding to a specific scenario. Each run of a benchmark will record one or more measurements per scenario derived. Usually a measurement is of elapsed time, but it is possible to specify alternate instruments to gather measurements of other types.

Guice of course are good :)) And you will find JME3 abtraction are good too because it fit lend it self very well in a lot of situations.

Application facilities for loading: AssetManager is an interface.
Input and RawInputListener are 2 points you can rig for user interactivities for testing purpose.
AppState is the way to hook activities.

With these 3 above, almost every UnitTest can be rig up.

If you don’t mind doing futher research, I also recommend:

Which will take some time to start up but allow you to rig everything (yes!) for unit test (including Application if you still insist) from interfaces and protocols.

P/s: If you do Java, please learn a Java way, and I personally appreciate Java as much as C# and Haskell. No one is better, stop the nonsense thing and focus in your game plz…

Well first of all I like the IInterface naming schemata, as well as the EEnum and the AAbstractClass.
This is just a matter of taste, and thats it. Btw they have some things I would wish java had as well, eg a unsave logic to deactivate all array bound checks in my core loop.

Then for any game based, the testing is different
-> Around 50% are at max texting for bug freenes
-> The rest should be usability testing, do not underestimate the need for this.

Now the thing is, for the time someone needs to actually write and maintain unit tests, you could always create a low level job for manual testing instead. Especially for any larger enviroment, doing manual integration tests is nearly not avoidable.
Eg gui testing -> if the controller gets the right input everything works for a unit test, but manual testing is only working if it also renders correctly, and is understandable.

SO in tl;dr

I find unit tests a highly waste of time for anything that is not a algorithm framework.

@Empire Phoenix said: I find unit tests a highly waste of time for anything that is not a algorithm framework.

In games, certainly. 100% agree. I might feel differently if I was writing a 1 million user MMO or something but any hobby game dev who thinks they are doing that is actually in a loony bin somewhere. :slight_smile:

Most game development exists on the sharp edge of ‘functional prototype’. In prototype land, full unit tests tend increase development time by a factor of three on average (you have to count all of the code you unit tested and then threw away during the prototype process). It can actually even be quite a bit higher.

The other thing to consider is that unit tests aren’t really unit tests if you don’t have a spec that you are working from. Otherwise, this is the common scenario:
-change something
-notice unit tests break
-think to self “Yes, but I meant this behavior to change”
-fix unit tests to match code.
-meanwhile 1,000 users scream into the night

That’s not really a unit test. That’s a speed bump. If desired behavior is not well documented then there is no way you can test for it.

TDD presumes that desired behavior is well known in advance.