Speaking of neural networks

In response to this thread.



Although there doesn’t seem to be much interest in this topic, I’d like to add a few comments.



A common misconception about ANNs is, that, because its based on a model of a real brain, you get something that behaves like an organic brain. For starters, typical ANNs consist of one type of neuron and one type of synapse and are mostly build as a straight forward network. A real brain is more of a conglomerate of too many subsystems to count with lots of different, specialized neurons and synapses.



Also, think of the training time you needed from birth to posting in this forum… you don’t want to spend years parenting your game-ais until they finally go to the restroom by themselves… You will never (ever!) reach an abstraction capability comparable to a human brain within the time period of programming a game-ai at the current stage of NN usability.



If you want to go for “intelligent planning” in an environment as abstact as a game, I would suggest planning systems like the very nice goal oriented action planning (GOAP) by Jeff Orkin et al. (sry, had to!). This might be worth to start a new thread btw.



You have to look at NNs more as a black box function. You have an n-dimensional input and an m-dimensional output. You know some of the functions values (your training data) and you tell the box that thats what you want to have. No something magical happens (or call it back-propagation) and after some time, the wheels inside the box stop and it has found a way to express your function values correctly.



Now the big question is, what does it do for inputs different then your trained inputs-output pairs?



In naive game terms:

You trained a NN to classify enemies according to the form of their 3D mesh (not that you actually would do this, just as an example). Now you import some sweet, little bunny meshes to make your enivroment more family-friendly… You start your game and suddenly all your monster run from the bunnies, because their NN gives a wrong, unforseen output to their mesh. You take a deep breath and start to remodel/retrain the NN. Again. On the other hand, if you are good at setting up NNs or just get lucky, the NN could classify them correctly.



A trained NN converges to the state which minimizes the error of its output compared to your given training data. When working as a classifier, it kinda finds the best borders between inputs that lead to different outputs. For example, a 1-dim NN could classify between “dangerous” and “not dangerous” simply by the size of the model of the entity to classify. Lets say the training data is the following:



Size Dangerous?

100 no

200 yes



Based on this abundant data, the best way to classify would be the same as the hardcoded “if size > 150 then dangerous, else not”.



Since the input in a NN tends to be high dimensional, its very hard to imagine the m-dim hyperplane that maximizes the margin between the inputs of your data in n-dim hyperspace, even if your output is just a bit. Try to extend the above example for more dimensions… Spatial imagination can only take you to a 3-dim input space, where the border would be a plane. After that, it starts to get confusing. If you start to map complex mixtures of states to other complex mixtures of states, you can forget about imagining it and have to look at it very mathematically. It is very hard to know what a complex NN will classify in the end.

You also need a training set that is versatile enough to train a classifier that accounts for all unseen inputs. And preprocessing is very important.





I think, NNs can excel in very specific tasks. One I’d like to mention is simultaneous localization and mapping for robots. Its about the problem to find your way in an initially unknown environment. One of the better performing algothithms for this is RatSLAM, a system based on the model of a rodents brain navigational functions. It uses visual cells to map to spatial cells, which give an abstract memory of the environment (in form of a specialized NN) and can in turn be mapped to an experience map, describing the remembered locations in real space. There are some google resources to the matter if you are interested in more details.



edit: on second thought, ratslam might be cool to create movement nodes in a newly build fps level by letting a virtual robot drive around in it to learn the map. you would only store the experience map as a sparse tree with the leafs being chokepoints etc… It would give a very fast pathplanning and length/time calculation.



I tried to use a NN to decide, if the tip of a scanning tunneling microscope changed while recording an image based on feeding the

image to the input of the NN. It kinda worked but almost always overlearned the data set I used for training (giving the right output for all images in the set but wrong/random output for new, unseen images). The heaton research site mentioned earlier helped me a lot to get started.



So… in my opinion, their advantage is, that you get a very fast classifier that you dont have to worry about how it works once the network is trained propery and actually classifies what you want to (see the tank example in a post above). Their main backdrawn is the same sentence, starting from “once”.



You also lose some of the advantage when it is time-consuming to preprocess the input to the NN from a complex world state. And you definitely don’t get “This is my world state - now do what a human would do with all possibilities the game has to offer.”



And a whole different aspect is the point of implementing a parallel computing scheme on a serial processor…

As you sort of hinted to yourself, it is worth making a new thread out of this, and so I did. I edited your post to reflect that.



What I am curious about is to what extent you can let an end user train a NN.



Dancing bear:

To use dancing bears as a cruel but brief example; one way to train them is to make them stand on a hotplate while you play the flute. The bear will only register the flute in its sub-conscious, because the pain takes precedence. Now whenever you play the flute for the bear again, it’ll dance.



Reward & punishment

Another basic example is the reward & punishment of a dog, usually marking the end of a small training sequence/routine.





With ANN all the way at the bottom and maybe some extra communication layer on top of that to make it work with a straight forward (end-user’s) GUI, do you think a game could be made where players could train their pets like this?

Actually I meant GOAP to be a good topic for a new thread, but thanks anyway  :wink:



Hm, there were some attempts at making such "petting"-games, like Creatures or Black & White? I'm not too familiar with them, so I don't know, how well they succeeded and how strongly they rely on machine learning or NNs more specifically.



A quick search gave:

"The program is one of the few games (as of 2008) to employ machine learning." - pretty vague…

and

"In the games, the Norns have their own simulated biochemistry, haploid genetics, and neural network to serve as a brain. From Creatures 2 onwards, Norns also possess organs determined by their genetics; mutation and injury can render these defunct, with typically fatal consequences."



Also sounds like evolutionary style algorithms. So probably a mixture of a lot of methods… Are you working on something like that?



The dancing bear seems like a good example of a poorly explored state-space or overtraining… Depends on the learning method, I guess. While training a NN, you usually reduce the stength of your adjustments over time. So if you only train it with flute sound while standing on the fire, it could keep that dancing behaviour without the fire too. There are different scenarios for different kinds of learning.



It's hard to imagine how a bear actually correlates sound patterns and pain experience, but lets just keep it simple as flute pattern on/off as a strong input to the network. The flute acts like some redundant/unnecessary input signal to the bear NN that opens up badly explored dimension of the input space and creates the wrong impression, that, when there is a strong input of the flute sound pattern, dancing is a good idea. In a way it relates to the confusion of the correct boundaries in classification I referred to in the first post. The hyperplane to classify along the temperature/action-dimensions is alligned correctly, but the flute/temp-part is screwed up.



The "magic function" of the bears NN is like (Temp,Sound,Action) -> Cost (Pain)



A Monte-Carlo bear would vary his action randomly to find the one which gives least pain. Assuming he will always explore other states of "Action", he could eventually find out that you can even chill to flute music. A NN bear would learn from the unseen input-output pairs and find a wrong correlation between sound and action. The dancing without fire is then a training error where some of the synapses interferring sound and action to the output have badly trained weights.



It might go away later, when there is more data connecting flute sounds to no-pain patterns and the network has time to relax (optimize its weights). It will never go away, if you anneal the learning parameters (e.g. reduce the learning rate) to a point where the network can't learn (change its weights) anymore from the later data. I guess most applied NNs (and generally learned controllers) in use recieve new learning data as general maintenance. The Tokyo subway e.g. has fuzzy-controlled trains that get new data at night by humans driving the trains. This helps to explore state space and avoid overtraining.



To make the training work in a game you would have to map pain to cost and any kind of petting to reward (or the other way around for a masochistic pet…) If you want a NN, you are all numbers. So you have to find a way to translate user input into numbers from which it makes sense to learn. You obviously won't get intelligent behaviour out of mapping mouse cursor coordinates… or maybe… :slight_smile: No, but this abstraction might be the biggest problem when trying to implement NN-learning of any kind of abstract in-game behavior. Maybe subsystems of it, would have to think about examples. I think the dancing bear might be kind of a discussably useful bug of a bear NN. Too bad evolution won't answer the bears detailed bug reports…



Reinforcement learning might be a more direct approach to really train a pet in a game. There you have the typical agent and an environment. On each update, you choose an action and in turn get a reward from the environment (user action, has to be translated to a numerical reward somehow). There are also scenarios designed for delayed rewards. Then you map world states or action states to expected rewards. E.g. the dog dances for you because this was rewarded with a bone earlier. You include some factor that gives you the greediness of the character. If its ~1, it will always do what gave the best results earlier. If its less, it will get more explorative, choosing actions that might give less optimal results (exploitation vs. exploration). If the random explorative action would be to bite you and you do something you regret later, the strong negative reinforcement would still lead to a very low frequency of bites in the future (depending on greediness, a very "explorative" dog might never learn not to bite you)…



It seems a little easier to apply, because you can work with an abstract action space, so it should translate easier to a game.



Hm, I was a little bit focussed on the learning aspects… will think about your pet training game some more. Do you have any more ideas about it in mind?

Hm, the network mapping was a little bit incomplete/wrong. Better to map State -> Action and use input-output pairs or a costfunction that will minimize State -> Pain to train the network.

No I'm not a programmer, but I'm an aspiring game designer and I also study psychology. Neural networks involves enough theory that I can still keep up with what really boils down to very advanced programming in this context.



Thank you very much for your elaborate explanations; I'm sure I'll be regularly coming back to this thread to re-read material or see if you're still available for consult :wink: Right now though I'm all out of (time for-) ideas.

That reminds me, if you are interested in neural networks with Java, there is a three part article on dzone about this topic. It also explains what you mentioned here, regarding training (you have to train it first, it doesn't learn during the game.) there is a code sample for a free network library. 



http://netbeans.dzone.com/articles/neuroph-smart-java-apps-neural-1

(the intro contains links to part one and two.)

Interesting article. I didn't look into all the options of the framework yet, but, while naming a lot other aspects, it seems to be focussing on image recognition?



I like the application to DotA they mention in the article. There they use the image recognition to monitor the state of the game and inform the player about certain things going on by capturing frames and feeding it to the NN. Some details can be found at:

http://dota-autoscript.com/blog/neuroph-neural-networks-dota-image-recognition/



Also this one about NN in Colin McRae:

http://www.ai-junkie.com/misc/hannan/hannan.html

I looked through some interesting papers on the subject of using NNs for decision making tasks.



This one is on using an evolutionary approach published last year:



Abstract:

Evolution of neural networks, or neuroevolution, has been a successful approach to many low-level control problems such as pole balancing, vehicle control, and collision warning. However, certain types of problems – such as those involving strategic decision-making – have remained difficult for neuroevolution to solve. This paper evaluates the hypothesis that such problems are difficult because they are fractured: The correct action varies discontinuously as the agent moves from state to state. A method for measuring fracture using the concept of function variation is proposed, and based on this concept, two methods for dealing with fracture are examined: neurons with local receptive fields, and refinement based on a cascaded network architecture. Experiments in several benchmark domains are performed to evaluate how different levels of fracture affect the performance of neuroevolution methods, demonstrating that these two modifications improve performance significantly. These results form a promising starting point for expanding neuroevolution to strategic tasks.



----



Again evolutionary with interesting learned behaviors of a bunch of monsters fighting against a single player:



Abstract:

Neuroevolution with multiple objectives allows for populations of agents to evolve different solutions based on various trade-offs inherent in any given domain. In particular, these methods can be used to evolve complex NPC behavior for games involving multiple objectives.



Multi-objective neuroevolution has currently been applied to a battle game domain in which a player fights against several attacking monster agents by using a bat as a weapon. As work on this project continues, multi-objective methods will be applied to increasingly complicated/difficult games with more objectives.







Feature on aigamedev.com about Creatures.