Technical commentary on Civ/game AI

[Xmonger]

I'm thinking there should be a saying, "it wouldn't be a Civ forum if every other post wasn't bashing the AI". Warning, long post.

Anyhow, I work in AI and have been in the field for many decades, through it's ups and downs, and am versed in the classical techniques and all the new exciting developments with DNN's (Deep Neural Nets) you're all hearing about. Given the amount of commentary on the Civ AI I thought it might be interesting to hear some thoughts on the technicalities of Civ and game AI in general (in a personal side company I also develop games).

In AI today there are basically two main techniques which I'll call "Classical" and "Connectionist" (any other practitioners out there just hold your nose, I'm simplifying). Classical is all the technologies (algorithms, concepts) that are developed as what we think of as a algorithm. An algorithm is just a recipe, and a venerable game AI algorithm is A* which is used for path finding still used in games today (it's how a mob finds it's way over to bash your head in).

Key idea: with classical techniques we (humans) tell the computer how to be smart, which assumes that we're smart enough to do it.

Connectionist has been around for a long time too but has been slow in coming because it requires lots of compute power, which (thank you gamers) is now available with our high end GPU's (there also was an important missing theoretical development). With this technique (also known generally as Machine Learning) we teach the computer how to teach itself, then feed it lots of data. For example, we set up the computer to detect things in pictures, but don't have to tell it what a cat is, but if we feed it lots of pictures of cats it will learn how to detect cats, really well. As good or better than humans now. This is the power behind all the recent advances you've seen such as image and voice recognition (and AlphaGo from DeepMind).

Key idea: with connectionist techniques we teach the computer how to learn, and the hardest part we have is getting enough data for it to do so.

Punchline: Civ and most games use classical techniques, which are fragile, break easily (and when they fail they usually do so spectacularly) and exhibit little spark that helps us get over the uncanny valley. Further they take an enormous amount of work and as the game develops they take more, and more work. You could spent a billion dollars on the Civ AI and get something that would still be fragile. This has nothing to do with developers, it's the nature of the problem and available technologies. But there's hope.

Connectionist techniques are ones I think we'll start to see more of in games because they're more robust (however when they fail they also do so spectacularly, but they just fail less), and they are generally easier (and trickier if that makes sense) to do. But they take a lot of data. Like, a lot. Take whatever you're thinking of and quadruple it.

Warning, techie bits here.
So what about Civ, connectionist sounds pretty good, why not do that here? Well let's see. We'd need about, oh say 100k to 500k full games as training data to start, let's just generate data from AI vs AI games. Then we'd need to define the network, what's the cost function? In other words how do you define the 'value' of a move? Well that's a great question, like what's the numerical value (from 0-100) of making a builder? It needs to be a formula, how would you express that? Be careful and don't turn it into a classical system by putting too much thought into it, if it's complicated it probably won't work. But what about the hundreds of other things you could do at any moment of the game? What's the cost function for that? Finally, what are the input neurons and output? Given the number of orthogonal actions at any point it kind of seems like the network would be bigger than what even our modern CPUs/GPUs could handle. Brrrrr .... I'd love to get some time to look into this, but at first glance it's not an obvious solution.

So what does Civ actually do? Some combination of classical techniques, yes A* probably for moving pieces. The main AI is probably a type of expert system, which is just a database of rules of what to do in various circumstances. As I said these are fragile and error prone, and when you push them (higher level games) they'll break. But Civ does have an advantage which is the system already is an expert system of a form. The game already knows what is possible and legal at any point in the game, so the AI can leap off of that. It's probably something like review all possible actions and then consult a database of likely choices, then modified by things like agendas and leader traits.

Conclusion: Sorry folks for the length but I needed to get that off my chest. Complaining about game AI is like complaining about the weather because a) you're not a weatherman and b) there's nothing the weatherman can do about it. Having said all of this in my tiny opinion the Civ AI is excellent, I'm continually surprised at the smart moves the AI's are making. Though I've played the game forever I'm not a good player (too busy but I like VI so much am keeping a game going most of the time), but in my games I appreciate how the AI Civ's appear to have distinct personalities and generally make good decisions.

 

[unpossible251]

Thanks though! I dont work anywhere near the industry, but broadly aware of in principle how these "connectionist" techniques work.

When I was reading about them, civ in particular (but strategy games in general) seemed like an obvious application.

 

[ashendashin]

Complaining about game AI is like complaining about the weather because a) you're not a weatherman and b) there's nothing the weatherman can do about it.

Really? Nothing can be done just because the system will always be too fragile? What about those who did manage to do quite a bit, to the extent of overhauling the 'weather'?
I can't understand how you can call the Civ VI AI excellent...that's not even funny.

 

[Boris Gudenuf]

Really? Nothing can be done just because the system will always be too fragile? What about those who did manage to do quite a bit, to the extent of overhauling the 'weather'?
I can't understand how you can call the Civ VI AI excellent...that's not even funny.

It's 'excellent' given the state of what's Possible in economic and technical terms At This Moment. Subject to change as the technology gets better. To put it in perspective, Self-Driving Vehicles are getting a lot more resources and money thrown at them, and arguably have a much less complex set of 'problems to solve than the Game AI, yet they still are only marginally effective at playing their 'game'. If you want an Opponent as good as a Human Opponent, you still need to play against a Human.

 

[Siesta Guru]

Heya, I love these AI discussions so thought I'd butt in again. While I don't work in AI, I've made several game AIs and regularly compete in bot writing competitions where I tend to do fairly well. I've also made the AI+ mod for civ 6, and have modded civ 5's AI in the past too (+ studied the code), so know the Civ system fairly well.

While I completely agree with your main points, which are that 'connectionist' systems would be a terrible idea to use (too risky mostly) and that classical techniques are somewhat fragile, I don't agree that this means there's no room for complaining.

People are complaining because they feel the AI is one of the most fundamental aspects of strategic games, and that the weak AI significantly limits their ability to enjoy a game they otherwise love. Bad AIs with bonuses encourage abusing the AI, rather than actually engaging with it in the way you would with humans.
By voicing opinions on what they find most important, they help shape what a company does in the future (if the company actually listens). Civ 6's AI seems to be mostly build by one guy, while the civ team is relatively large. People don't appreciate that the team is skewed in this manner, and would probably have preferred it if the company prioritized funding better AI over say leaderheads or tech quote voice acting. The latter two are executed well, but just aren't as important to game enjoyment as AI is to a lot of people. To have only one person allocated to such a difficult to implement and fundamental aspect of the game feels strange.

This former point wouldn't be fair if the AI was actually really good, and it would be technically impossible to make it much better, but it just isn't. The AI makes many extremely obvious and dumb mistakes, plays unoptimally overall, frequently plays against common sense and doesn't engage at all (or extremely rarely) in some aspects of the game like aircraft/nuclear weapons. The game frequently still feels easy enough on immortal/deity to be able to autopilot, despite ridiculous AI buffs. These weaknesses are not technical limitations or limitations due to their choice to use a classical system, but limitations due to a lack of high quality code in many areas of the AI execution.
Classical AI systems actually have a lot of potential power. The stockfish chess engine brutally murders any human and is based on fairly conventional AI techniques. Chess is an easier game, but there are also games that are in theory harder for AI, where it yet ends up giving players a much better challenge. Take for example the by now 20 years old age of empires 1, just a random game by no means especially acclaimed for its AI.
The AI in that game feels significantly more interesting and challenging than the civ AI does, while using less cheats, while built for much older computers, while operating in continuous space (harder for AI), while taking up less processing time, and without even resorting to abusing stuff AIs are good at (like insane micro). And this AI was build 20 years ago, when we knew less about AI algorithms and when google and stackoverflow werent available.
Sure, you can come up with some arguments that Civ should be harder to make good AIs for too. But the reality is just that in terms of AI, it feels weaker than something we had 20 years ago, that's not likely to impress people.

And coming from my background as a competitor in bot coding tournaments, the top 50 or so bots in these competitions easily blow the civ AI out of the water, while build by hobbyists in their spare time in timespans as little of 1-3 weeks. The Civ AI for example frequently just ends the turn on units/cities that can fire, and it frequently ignores cities at 0 health with adjacent melee units. The Civ AI frequently has 50%+ of its units just stand around doing nothing while a war is ongoing. It will miss the opportunity to capture civilians or make its own civilians walk out right into visible enemy units. It settles a tile away from rivers instead of adjacent to them. It declares peace with others at no cost while surrounding the enemy capital at no health. It will send high promotion units in to die for no good reason. It builds wonders it has no use for, picks policies and beliefs with no benefit. None of these issues are by any means hard to solve using classical AI systems.
With errors like these, you couldn't even hope to compete in a tiny bot coding competition, but somehow we're expected to consider this normal for a big budget studio which had years, and which delivers a game bought by millions. It's just an incredibly dumb business decision to not focus more on this, because it should be obvious by now a lot of people are not buying into the game at all, or aren't buying into DLC/expansions because the way games get stale without good AI (and because people aren't recommending games they don't really enjoy). They could've easily gotten 5-15% more revenue by just putting a second skilled employee on AI full time, which would've paid that employee back many times over.

 

[ashendashin]

It's 'excellent' given the state of what's Possible in economic and technical terms At This Moment. Subject to change as the technology gets better. To put it in perspective, Self-Driving Vehicles are getting a lot more resources and money thrown at them, and arguably have a much less complex set of 'problems to solve than the Game AI, yet they still are only marginally effective at playing their 'game'. If you want an Opponent as good as a Human Opponent, you still need to play against a Human.

In technical terms a far better AI has already been achieved for Civ V, and the performance isn't bad...When it comes to cash, it's really down to hiring the right people and letting them work on the game. They've got one dude right now and I doubt their competence.
And no one is talking about playing against an AI that's just as good as a human.

 

[Karmah]

I'm thinking there should be a saying, "it wouldn't be a Civ forum if every other post wasn't bashing the AI". Warning, long post.



Conclusion: Sorry folks for the length but I needed to get that off my chest. .

Good thing you did ! As a fellow practicioner for decades too , I heartingly applaud ,you summarized everything I think everytime the discussion on AI rises. Especially you made clear that yes it is tricky to set up models that could help the AI but it is really doable and even hard to understand why it hasn't been made so.
If Fireaxis launch an initiative to collect gameplay data from volunteers for an AI project, I'm pretty sure they would enough participants here to start working soon enough.

 

[Xmonger]

Really?

Yes. This is the point of the post - there's not a lot Firaxis can do, nobody has solved the Expert System approach, it will always easily break. Which by definition means somebody can mod in an improvement, but I guarantee that breaks too depending on the game circumstances.

One possible correction, I realized that there is a cost function, the Firaxis team has already developed it which is used in the Score win. So now the problem with doing a DNN is the model you'd choose and it's architecture. A conv net probably won't work ... not sure, maybe plain old gradient descent? I'll back burner it.

 

[stealth_nsk]

Ok, some bits to consider:

1. This post assumes developers have a goal of making as strong AI as possible. They don't. The AI is a gameplay system which should:
- Provide players with impersonated opponents for role-playing.
- Provide some forms of challenge (victory race, settle race, military threat, wonder race). Actually civ game doesn't require opponents at all - I could imagine a civ-like game where you need to launch space ship till some turn, without any opponents and the game could be challenging.
- Fuel some parts of gameplay mechanics, for example alliances.
Actually the last part is quite opposite to AI playing to win (see classic "Playing to loose" about civ AI). No machine learning could help here, because it optimizes AI for winning, usually by a single strategy. This conflicts not only with the third point, but also with second. For example, it's likely self-learning AI will decide Wonders don't worth their value and will not provide any challenge for wonder race (if it will decide the opposite and never let human player to build wonders, it would ruin this part of the game just the same).

2. The complains about poor AI are about AI looking poor, not functioning. There were a lot of complains about Vanilla Civ5 AI before developers added some announcements made by AI about their decisions. Without any actual changes to AI logic, it started looking better for players, because they understand it better. Any AI based on machine learning will look dumb, but cheating from player perspective. Not because players don't understand the game, but because they see only part of AI actions and no AI reasons behind them.

3. Actually the definition of "connectionist" (or machine learning) includes very broad range of approaches, all with their individual problems and benefits. For example, supervised learning is way more effective than neural networks, but the amount of human work required is likely to be more than just implementing algorithms classical way.

In short, I don't see any place for machine learning in games like Civilization. There are other game types, where it works, though - classical board games like chess, simple arcades and so on.

 

[ThunderLizard2]

a) you're not a weatherman and b) there's nothing the weatherman can do about it.

You don't need to be a weatherman to know which way the wind is blowing (credit BD). In this case, you don't need to be an AI programmer to know a bad AI when you see one. From all the specific posts I've read the AI makes too many really dumb decisions and can't put together a decent tactical attack or defense.

Why was the Civ IV AI much better? I think one answer is the mechanics were simpler and the 1 upt is much harder on an AI than a human who can see and process a situation at a glance. Civ IV BTS was released over 10 years ago and computers were maybe a 1/10 of todays power.

 

[isau]

I am not as experienced an AI coder as some other people here. I have written some very basic AIs for simple games of the platform and tactical RPG variety.

Part of the problem I think with Civ AI is overthinking it. Some things really just need to present a facsimile of intelligence and the elements thought more of as like the turtles and goombas in Super Mario. After all, you aren't really trying to create an AI player equivalent to the human. What you're trying to create is a formidable obstacle with some vestiges of intelligence about it.

Based on that fact, my take on this is that:

• Game rules should in general be straightforward enough for an AI to navigate in the first place
• Creating an obstacle is more important than creating realism

A good example of this is air units. There doesn't need to be any real complexity here. The AI should:

1. Build air units
2. Use them

There are a couple of questions like "Are air units more important than other units?" But really the answer to that is that the unit systems should easily be able to identify perhaps 5 units viable at any particular point in the game, and stick to those. That's minimum level intelligence but practical for most scenarios. That's also pretty much how Civ 4 AI seems to judge which units to build. They did build and use planes in Civ 4. And 5.

In 6 they don't and I find it very odd.


Part 2, changing game rules to accomodate AI rather than the other way around.

There are some rules that are far outside the AIs ability ever comprehend. The Domination victory, for example, is extremely nuanced and well outside of the AIs capability. That is why in Civ 4 the actual game rule was different. Own a certain percentage of world tiles. It's easy to teach an AI to do that in comparison. Just teach it the more tiles it gets the better. Then it presents a real credible threat. It doesn't need to navigate to 3 specific cities in the world and take them. They can do their thing anywhere, and the player better be ready to stop them wherever they are doing it.

A second example is Settlers being capturable. There's so much nuance that goes into this rule that there's just no legit reason it should even exist in a game with AI. Settlers should just not be capturable. Problem solved without any changes to AI. Keep the rules realistic and the expense of building complicated AI goes down considerably.

 

[ashendashin]

Yes. This is the point of the post - there's not a lot Firaxis can do, nobody has solved the Expert System approach, it will always easily break. Which by definition means somebody can mod in an improvement, but I guarantee that breaks too depending on the game circumstances.

Good thing Civ5 was rebuilt so it wouldn't easily break due to base mechanics. Also you're very vague about what it means for the AI to break. I've played Civ 5 with Vox Populi for over a year and there's never been anything to completely 'break' the AI.
Through development something bugs out, sure, but that can always be addressed so there's no point in not trying. The majority of bugs in the AI were left by Firaxis, by the way.

Despite your claims, Firaxis does at least put some work into improving the AI. Poorly and slowly. It needs to be overhauled starting from base because it doesn't even work with the game mechanics, so yes in a way you're right to say that the system is vulnerable, because the base is a crap system. This definitely isn't a case where nothing can be done, because what has been done was a crap job to start with. I've followed Vox Populi's development enough to know that Firaxis built Civ 5s DLL poorly and I have no doubt it's the same case with Civ6.

You're telling this to us non-weathermen, but I'd much rather see you address Siesta Guru, who does actually know what he's talking about.

 

[Abaxial]

Great thread, and I appreciate the comment by Siesta Guru that Firaxis have under-funded the AI coding relative to other features.

One thing I have noticed, and I think I saw it in Civ V also, is that some CSs seem to be magnets for the AI. In my current game Vilnius, which is tucked away at the N end of a continent, miles from anywhere, has been attacked by three different AI powers in succession (and liberated by me each time). There must be some rule in the AI that decides that certain places are good targets, even if you have to assemble an armada to get to them.

 

[phoinix]

I'm thinking there should be a saying, "it wouldn't be a Civ forum if every other post wasn't bashing the AI". Warning, long post.

...

So what does Civ actually do? Some combination of classical techniques, yes A* probably for moving pieces. The main AI is probably a type of expert system, which is just a database of rules of what to do in various circumstances. As I said these are fragile and error prone, and when you push them (higher level games) they'll break. But Civ does have an advantage which is the system already is an expert system of a form. The game already knows what is possible and legal at any point in the game, so the AI can leap off of that. It's probably something like review all possible actions and then consult a database of likely choices, then modified by things like agendas and leader traits.

Conclusion: Sorry folks for the length but I needed to get that off my chest. Complaining about game AI is like complaining about the weather because a) you're not a weatherman and b) there's nothing the weatherman can do about it. Having said all of this in my tiny opinion the Civ AI is excellent, I'm continually surprised at the smart moves the AI's are making. Though I've played the game forever I'm not a good player (too busy but I like VI so much am keeping a game going most of the time), but in my games I appreciate how the AI Civ's appear to have distinct personalities and generally make good decisions.

Really appreciate these kind of posts as I think are a breath of fresh air relative to the constant "AI sucks" posts that dont offer any new insight. I agree that the AI problem is difficult but I believe Civ AI has a lot of room to improve.

Im not a professional in AI but I have dabbled, taken a few AI courses and I am following the developments in the field.

I have been thinking of the problem of CiV AI and how the "connectionist" approach can help for quiet some time. It's true that the game rules can be quiet complex and hence a monolithic network would be too difficult to train and troubleshoot if things go wrong. Not to mention that choosing a proper cost function may be as complex as the the AI itself.

After some thought I remembered the old divide and conquer approach: "If a problem is too complex, cut it in smaller pieces". Sure a global optimized solution might be "smarter" but it would also

1. take forever to run
2. take forever to tune and
3. is next to impossible to debug/maintain.

I came to the conclusion that a hybrid connectionist/classical approach has the most promise. The main ideas are:

• Independent small networks/AIs handling all the partial decisions the AI has to make. Examples (the questions each of those AIs must handle are in parentheses):
  • Settler AI (where to build a new city to maximize its usefulness as a production/commerce/etc city)
  • City Build AI (what thing to build next in city X, which tiles need to be flagged for improvement),
  • Policy, Science, Civic AI (self explanatory),
  • Tactical AI (how to "win" a battle, how to take over a city),
  • War AI (whats a good war plan: which cities to take, should they be razed?),
  • Diplomacy AI (given the circumstances, what is a good deal),
  • Strategic AI (Which victory condition? should I go to war? Which opponent must be stopped? whom should I ally? should i expand?).
• Hierarchical architecture with both bottom up and top down paths (feed-forward and feed-back): the Strategic AI sits at the top, takes input from lower level AIs (such as Tactical) and sets biasing (or weights) on the lower level AIs to influence their decisions.
  
• The types of those independent AI pieces can vary (classical approaches like algorithms or connectionist methods such as networks -CNN, recurrent etc-). If a sub-problem is easily solved by algorithm xyz, by all means use it!
• Strive for "good enough" approximations, not optimal behavior or 99.99% accuracy.

The advantages of such an approach are:

1. Individual AIs are smaller, easier to train/tune/maintain, run faster (also suitable for parallel implementation).
2. Robustness coming from networks.
3. Hierarchy imparts a feel that AI individual decisions are synchronized, all working towards a overarching strategy to achieve victory.
4. Is not "optimal" in a way that makes the game annoyingly hard (not that this has ever been an issue in Civ but still...)

I have some experience in programming so I think I can at least try implementation but that would depend on how much of the game Firaxis opens up for modding. If the only tool given to modders is to tweak some values to get the existing CiV AI make the decisions I want it to make, I'll pass.

...

Fully agree that AI needs more attention from Firaxis than a single engineer!

 

[Siesta Guru]

If the only tool given to modders is to tweak some values to get the existing CiV AI make the decisions I want it to make, I'll pass.

Yeah that's basically it right now. Unless you figure out a way to change AI behavior through lua. My own investigation failed to deliver anything worthwhile on lua side, but I could've missed something. Annoyingly it does contain methods to move units, which could've been awesome, but it doesn't seem to contain anything to overrule AI behavior and it disregards movement points, so it ends up moving units and attacking twice etc. Haven't found a way around that yet

 

[Karmah]

3. Actually the definition of "connectionist" (or machine learning) includes very broad range of approaches, all with their individual problems and benefits. For example, supervised learning is way more effective than neural networks, but the amount of human work required is likely to be more than just implementing algorithms classical way.

In short, I don't see any place for machine learning in games like Civilization. There are other game types, where it works, though - classical board games like chess, simple arcades and so on.

Uh ? Neural Network is by essence a supervized approach , arguably the first historical supervized approach in its traditional 80's form before SVM took over
Now I'm more than aware that they are now unsupervized components in any any complex deep learning based trainer (if only as a speed up component if not at the core of the learning) , it just feel strange to oppose supervized learning and neuron networks

 

[nzcamel]

Chess is an easier game, but there are also games that are in theory harder for AI, where it yet ends up giving players a much better challenge. Take for example the by now 20 years old age of empires 1, just a random game by no means especially acclaimed for its AI.
The AI in that game feels significantly more interesting and challenging than the civ AI does, while using less cheats, while built for much older computers, while operating in continuous space (harder for AI), while taking up less processing time, and without even resorting to abusing stuff AIs are good at (like insane micro). And this AI was build 20 years ago, when we knew less about AI algorithms and when google and stackoverflow werent available.
Sure, you can come up with some arguments that Civ should be harder to make good AIs for too. But the reality is just that in terms of AI, it feels weaker than something we had 20 years ago, that's not likely to impress people.

I gotta say...is real time strategy really harder for AI? Or is it that the limitations of what the human can see and do at any time compared to the AI (which can "see" and move all it's units at once) which gives it a different kind of advantage?
Cos the issue with turn based games (esp SP) is that the human can take their time to optimise the best move or choice in every case. In real time strategy we don't have that luxury. I'd suggest that real time AI gets made to look better by the very nature of putting time pressure on humans.

 

[stealth_nsk]

Uh ? Neural Network is by essence a supervized approach , arguably the first historical supervized approach in its traditional 80's form before SVM took over
Now I'm more than aware that they are now unsupervized components in any any complex deep learning based trainer (if only as a speed up component if not at the core of the learning) , it just feel strange to oppose supervized learning and neuron networks

Yep, I didn't dive that much to terms. I meant modern neural networks, which are nearly completely are about deep learning.

 

[Xmonger]

1. This post assumes developers have a goal of making as strong AI as possible.

No it doesn't, the post was an attempt to write a short explanation about why the AI is the way it is, not because Firaxis is necessarily incompetent, but that it's a difficult problem.

2. The complains about poor AI are about AI looking poor, not functioning. There were a lot of complains about Vanilla Civ5 AI before developers added some announcements made by AI about their decisions. Without any actual changes to AI logic, it started looking better for players, because they understand it better. Any AI based on machine learning will look dumb, but cheating from player perspective. Not because players don't understand the game, but because they see only part of AI actions and no AI reasons behind them.

Yes, I wonder if a lot of the complaints is because people aren't understanding the reasoning behind the AI decisions

Great thread, and I appreciate the comment by Siesta Guru that Firaxis have under-funded the AI coding relative to other features.

Maybe. A developer or artist costs about $180k/year (depending on overhead), 10 is a very small team and is 2 million being burned year after year. I have a small game company (consisting of three people) and I doubt I'd throw $400k/year at at AI if I was the size of Firaxis - one dedicated developer is generous IMO.

I gotta say...is real time strategy really harder for AI?

Yup

 

[Victoria]

A second example is Settlers being capturable. There's so much nuance that goes into this rule that there's just no legit reason it should even exist in a game with AI. Settlers should just not be capturable. Problem solved without any changes to AI. Keep the rules realistic and the expense of building complicated AI goes down considerably.

Non capturable settlers is just a nightmare, see the opposition settler walk around my city and settle on the other side? Its one thing to have a great musician wondering around but its quite another to have a unit as subtly devastating as a settler. You may not think it so but if the civ also has a few thousand gold then its another story.

Yes, I wonder if a lot of the complaints is because people aren't understanding the reasoning behind the AI decisions

A good example of this is joint wars, everyone complains, thread after thread. Someone gave me the logs of game and asked why Germany was friendly but still joint warred and so I went through it. (link in my signature) He agrees the reason for the joint war now made sense and in fact the real evil behind it was Spain who had been spamming civs for a joint war for a while. Its easy to complain without all the facts.
Also to me the thing the AI does not have that will not make it as good is a decent memory, in particular of military movements.