Learning AI

[timmey_o_tool]

Hi,

I don't know anything about game programming, but I have some knowledge in AI algorithms.
So my question is, why do the developers not use any "learning algorithm" for the AI (or do they during the testing ?) ? With the huge "open beta testing" we have now, the algorithms should converge quite rapidly. And as the AI is structured in different "levels" you can train them separately to avoid to much complexity...
Correct me if I am completely wrong please and help me understand.

 

[Sephi]

There have been a couple of attempts to implement "learning AI" in Games in the last decade but non of them have been really successful to my knowledge.

There are reasons for this. When a game is produced, it is typically finished only last weeks/days before release. And then there are even games that are released unfinished and that need major patches I have heard. So in the worst case your AI would need to always relearn the game. I am sorry that I cannot give more specific answer.

What kind of learning algorithm would you use for a game like CIV 5?

 

[Beef Hammer]

There have been a couple of attempts to implement "learning AI" in Games in the last decade but non of them have been really successful to my knowledge.

There are reasons for this. When a game is produced, it is typically finished only last weeks/days before release. And then there are even games that are released unfinished and that need major patches I have heard. So in the worst case your AI would need to always relearn the game. I am sorry that I cannot give more specific answer.

What kind of learning algorithm would you use for a game like CIV 5?

The one they used for Galactic Civilizations 2, probably the hardest AI that has ever been made for a TBS game, I believe it "learns" as well.

 

[timmey_o_tool]

To Spehi :
Yes, when the game is updated, the AI needs to be updated to. But isn't this easier with a learning algorithm ? I mean, after every new patch, it does a new run and with the thousands of players ciV has, even if only a fraction would agree to send some data it should converge rather fastly.

The algorithms I know best are the genetic one, but I assume that all learning algo. just do update the values based on some probability densities. Do you know any powerful which should/could be used ?

Or is there any deep reason why it cannot be used which I don't know about ?

TO Beef Hammer :
Can you be a bit more precise ? Do you know if they really used learning algos. or did they just do very good testing ? (I've never played the game)

 

[Deep_Blue]

As far as I know learning AI is very rare in computer games, when Far Cry was released it was advertised with learning AI but I didn't test the game and I don't know if it truly had a learning AI. I don't know other games that claim of having learning AI.

Learning AI algorithms are built upon Neural Networks and Fuzzy logic principles, everything is controlled by weight vectors with feedback and feed-forward processes where the output is not settled before a certain number of trials or past memories. You can also search for something called Genetic algorithms. I am a programmer I tried to explore this field but It is complicated and I dont have the time to start learning it from scratch.

 

[DaveGold]

There are number of problems for a CIV style learning AI, but the essentially the many small variations in past events make it difficult to learn how to act in a new situation which may or may not be similar.

For example, if an AI is thinking whether it should upgrade its warrior to a spearman does it need to compare previous events with a similar date, a similar treasury, a similar income, similar barracks production, similar policies, similar unit mix, similar war conditions, similar barbarian activity, similar resources, similar unit promotions, similar unit health, or use other learnt behaviour? A human mind can filter all this to pick out the relevant data for a decision. A learning AI finds it very difficult to pick out the required data and have it all recorded correctly from previous games, even before it can start to learn.

 

[Beef Hammer]

TO Beef Hammer :
Can you be a bit more precise ? Do you know if they really used learning algos. or did they just do very good testing ? (I've never played the game)

O man I really don't remember where i read all about the AI for that game, been a long time. I wanna say it was something along the lines that they were able to "teach" the AI advanced tactics used by players or something.

 

[CivFanaticMan]

If we make it learn to hard it could take over the world...

 

[Rasma]

If we make it learn to hard it could take over the world...

You mean actually understand how to conquer the game... Egads!
It might become self aware and nuke us... curse the icbms.

 

[timmey_o_tool]

There are number of problems for a CIV style learning AI, but the essentially the many small variations in past events make it difficult to learn how to act in a new situation which may or may not be similar.

For example, if an AI is thinking whether it should upgrade its warrior to a spearman does it need to compare previous events with a similar date, a similar treasury, a similar income, similar barracks production, similar policies, similar unit mix, similar war conditions, similar barbarian activity, similar resources, similar unit promotions, similar unit health, or use other learnt behaviour? A human mind can filter all this to pick out the relevant data for a decision. A learning AI finds it very difficult to pick out the required data and have it all recorded correctly from previous games, even before it can start to learn.

You are right, but that is for me exactly one reason why we could use learning algos. Now a human has to decide in a limited number of situation what is best to do and set the AI values. But this will never take into account all possibilities, far from it.

For any algos. even if learning, the human has to put a lot of prior knowledge in it, even if just by the way the information is coded. So, even with a learning algo. it wouldn't be something magic were the AI just "learns" everything, but it is a way to find the right weights.

Does actually anyone know if neural networks, fuzzy logic graphs or genetic algos. are used for the AI in normal games ? In previous Civ's iterations ?

 

[CivFart]

There is no such thing as a "thinking" AI in a game like civilization. There is too much balance work that is absolutely neccesary for it to be a thinking AI.

The AI cant think in a game like this, that would totally screw up the game. It is a mix of cognitive thinking (In very small quantity) combined with alot of math, balance-work and such.

 

[timmey_o_tool]

That is precisely what a learning algos. does do : math and balance-work to quote you.

 

[ian.w.lloyd]

Judging by some of the XML and descriptions of the way the AI operates, they might be using a (non-learning) neural network for some of the decision making, which does make sense. It would also contribute to some of the problems around "why" the AI does things, in particular the diplomacy. Neural networks are notoriously difficult to diagnose or understand and they can be prone to taking (what appears to be) non-logical responses to stimulus.

But the biggest problem with using a pure learning algorithm is the garbage they produce at the beginning. Its not uncommon to find that neural networks or genetic algorithms produce responses that turn in circles or run away from everything before they figure out how to actually play the game properly. The huge number of details in a game of Civ5 make it difficult (not impossible!) to build a learning algorithm, but until the AI learned enough, it would be totally useless, and the time needed to train it would be mind boggling.

What they could do, though, is collect save games and analyse them for self-defeating behaviours and traits. This would fall under "machine learning," specifically data mining. For instance, if they collected a few hundred save games they might find that Napoleon takes a non-optimal action when he has hundreds of gold saved. The developers could manually modify the AI to correct that behaviour. Its a kind of learning algorithm, just one step removed!

 

[Deep_Blue]

If learning AI is implemented in civ then each new game the AI will be harder to beat, it is like the game has memory and personality, we dont want to see this in Civ game because it is unnecessary first and will screw difficulty levels (imagine noble AI becomes smarter game after game), and is very hard to balance. Also fixing AI problems in the future will be harder because developers will not be able to debug the game in the same way as step by step programs.

For civ a balanced piece of code with weights, probabilities, flavors will do the trick.

 

[SHaW1986]

it would be a bad thing game after game, but if everything the ai learned was erased after each game it would be cool. imagine if the ai learned during the game and the amount capable for the ai to learn or how fast they learn was determined by the difficulty level.

 

[Deep_Blue]

Thats cant be done, the AI needs large number of trials to learn something, 1 game is not enough to learn anything. The AI has to play at least 20 games to start learning things that are as basic as learning how to walk. Unless the developers trained the AI before releasing the game to a level that they feel is balanced then they shutdown the learning process and releases the game at that level.

 

[timmey_o_tool]

If learning AI is implemented in civ then each new game the AI will be harder to beat, it is like the game has memory and personality, we dont want to see this in Civ game because it is unnecessary first and will screw difficulty levels (imagine noble AI becomes smarter game after game), and is very hard to balance. Also fixing AI problems in the future will be harder because developers will not be able to debug the game in the same way as step by step programs.

For civ a balanced piece of code with weights, probabilities, flavors will do the trick.

Funny, I don't wanna be disrespectful, but it seems to me that you have a somehow weird knowledge of learning AI's. On one hand, you know that they need several data to converge (although your cited at least 20 games is a completely random number), on the other hand you really think that AI wold become really intelligent !! You should know that no AI will ever (well for several years/decades at least) get that good that a human can't beat it (except very "simple" games, like chess)...

 

[timmey_o_tool]

Judging by some of the XML and descriptions of the way the AI operates, they might be using a (non-learning) neural network for some of the decision making, which does make sense. It would also contribute to some of the problems around "why" the AI does things, in particular the diplomacy. Neural networks are notoriously difficult to diagnose or understand and they can be prone to taking (what appears to be) non-logical responses to stimulus.

But the biggest problem with using a pure learning algorithm is the garbage they produce at the beginning. Its not uncommon to find that neural networks or genetic algorithms produce responses that turn in circles or run away from everything before they figure out how to actually play the game properly. The huge number of details in a game of Civ5 make it difficult (not impossible!) to build a learning algorithm, but until the AI learned enough, it would be totally useless, and the time needed to train it would be mind boggling.

What they could do, though, is collect save games and analyse them for self-defeating behaviours and traits. This would fall under "machine learning," specifically data mining. For instance, if they collected a few hundred save games they might find that Napoleon takes a non-optimal action when he has hundreds of gold saved. The developers could manually modify the AI to correct that behaviour. Its a kind of learning algorithm, just one step removed!

Hmmm, very interesting answer... I agree with you that pure learning algorithms are probably not a good way to go, a mixture is always best. But, still, even if the convergence time is high it could be used for "minor features" : actually I am thinking of exactly your second example. If you analyse over several games, it should/could become obvious that Napoleons strategy is non-optimal. You suggest that after some data mining a human corrects manually the numbers, but could't this be codded, and therefore we would get a learning machine ?

 

[Sephi]

To Spehi :
Yes, when the game is updated, the AI needs to be updated to. But isn't this easier with a learning algorithm ? I mean, after every new patch, it does a new run and with the thousands of players ciV has, even if only a fraction would agree to send some data it should converge rather fastly.

With a large open beta it could work, yes. My point was just that there are arguments against learning AI (how valid these arguments are in a very different matter) and that it's standard in Game Development not to use learning AI.

The algorithms I know best are the genetic one, but I assume that all learning algo. just do update the values based on some probability densities. Do you know any powerful which should/could be used ?

Or is there any deep reason why it cannot be used which I don't know about ?

I was thinking about a genetic algorithm as well to optimize the weights in a decision making process.

However there are two big questions.

Does the algorithm converge at all? You may end up in some oscilization problem. AI techs fast -> AI get rushed -> AI builds more military -> AI gets outteched -> ....

And how fast does it converge? You want a correlation between AI success and different weights used. But there is a lot of background noise that influences AI success.
- Civilizations are different (easy to eliminate if you only optimize one weight, but typically you must optimize several weights at the same time). Different Civilizations have different optimal weight patterns.
- random maps (optimal pattern on a food low map might be very different than on a production low map)
- A lot of randomness in the AI decisions (war declarations for example, atleast in CIV4). This is necessary to make the AI less predictable and the game more dynamic.

I am sure you can filter the background noise with fairly easy statistical methods but I think it could lead to a very low converge rate, even more if it is implemented by someone who does not have much experience at this.

 

[Fabien]

It's unbelievably hard to teach an AI something as complex as Civ 5 and simulating this neural Network would eat up all possible ressources anyway. You'd need pobably thousands of games just to teach them how to connect cities and hook up ressources in a meaningful way.

You probably wouldn't like to play such an AI as well, because they'd all decide to mop that strange random and cheating thing up (you, the player) because it usually wins the game aynway.

They could however let the AI play against each other with the different flavours they have; i'm sure some flavourcombinations work better than others.