Game AI & net based machine learning

[Deggial]

Okay, this is sort of a thought experiment.

We all know, that Civ's AI (and especially Civ 6's) is lacking. We also know some of the reasons, mainly that programming a good AI is expensive and doesn't necessarily sell more copies of the game.

But it is actually still necessary to program such an AI?

Deep learning is the hotttest Moderator Action: stuff nowadays. 'Alpha Go' defeated one of the best human Go players, but was defeated itself by 'Alpha Go Zero'. (The difference between those two networks is, that the previous was taught by analyzing recorded games played by humans. The later was merely taught the rules of Go and learned everything else by itself!)

I know that there was an idea (by the creators of Alpha Go themselves, not some fantasizing fans, that is!) to create a Civ 4 playing neuronal network. Unfortunately, I don't know, whether or not this idea came to any fruition.

What, if Firaxis would make this idea real for their game? What, if they actually trained a deep network and let it loose on us fans? The beauty of it would be, that no conventional programming would be needed at all to get very good results. (I am not claiming that setting up such an AI wouldn't involve any work, of course.)

Deep networks calculation times can currently outmatch "classical" algorithms by an factor of 1000. They don't run on CPUs, but GPUs nowadays - which we all have in our computers.
Not everybody has the most powerful GPU though and calculation time for the AI is not available for the GPU's actual purpose: the rendering of pictures. Additionally, I don't know, how many layers such a game AI network would need to achieve satisfactory results - and whether or not such a deep network could be calculated by our single PC GPUs at all.

But even for this, there could be a solution: cloud based deep networks, for example the Google Cloud Platform. Just send the game data to the cloud, let it compute the moves and display the results on your computer. (Or run the whole game in the cloud, of course; but this is a different topic.)

Google doesn't give the service away for free, of course. There would be a cost involved. And again, I have no idea about the related prices.
But WoW players pay 15 bucks (or so) per month to play their game. I would expect using Googel's cloud based network be in the same ballpark.

I am pretty sure, Firaxis could create a huge PR gain out of "the first true AI in the history of 4X strategy gaming". I even believe, that DL specialists would be very willing to work with them and examine the technologie's capabilities to handle such a "fuzzy" rule set (Go is very complex, but it's rules are very, very simple!)

So, would you guys be willing to pay a monthly fee for a really good, DL based AI?
What, if such a subscription would be optional? ("Classic" rule based AI for free, DL as an additional service.)

What are your thoughts about this topic?

Could (cloud based) neuronal networks be the (near) future not only in picture recognition etc., science (the graviton wave detector uses neuronal networks to filter the signals) and other "serious" fields, but also in the gaming industry?

Moderator Action: Replaced inappropriate language in your post, please remember this is a family friendly site -- Noble Zarkon
Please read the forum rules: http://forums.civfanatics.com/showthread.php?t=422889

 

[universecreep]

Great ideas but I think Firaxis/2K wouldn't go for this as they're in it to make money and the return given the investment would be low.
I suspect this type of thing would probably have to be initiated by a collection of fans probably starting with Civ V since it's been out for a while. I think the source code is available for that game as well although I'm not sure if that would be needed.

Maybe they'll try something like this for Civ 7 or if some other company with a broader vision for Civ-type games decides to make a Civ game of their own - I'm talking to you Stardock!!

 

[True_Candyman]

This is a sure fire way to demonstrate exactly how terrible the mechanics that sit behind this game are.

One week after any kind of machine learning is implemented, all AI on all difficulty levels will be swordsman rushing you to crap.

It’s theoretically a good idea, but until there’s a balanced game for it to operate in, it will 100% turn this into COD modern warfare 5000

 

[Josephias]

I'll be less optimistic here and say that the complexity of civ VI would require quite a big neural network... yep, cloud solutions could be an option, but still, they require the machine power somewere - Bitcoin works by "using" computer power of all Bitcoin miners in example. In the same way the SETI program (where has it gone, btw?) used computer power of all subscribers: remind: the clould is real: ethier rented servers somewere, or tiny bits of power on multiple computers.

Not to say, neural networks can develop quite a good AI, but it will take some cycles of being quite a bad AI: it is "learning" afterwards, and to learn you have to fail a lot, so you know all the places you do not have to go. Maybe (and it is just a weird speculation) they are using already deep learning with Civ VI, with steam transferring all our games to a Server in 2K headquarters, where the AI is still trying to underestand why this settler that send ten tiles away to found a city is captured the 75% of times (while the other 25% of times builds a city that rocks). ¿Which is the better option counting the cost of the settler? ¿How many tries will it take to consider it a final GO/NO GO solution and not a path still open for experimentation?. (And, supposing the new AI code is only delivered by patches, ¿how many patches will it take to be provided an AI that has learnt enough to challenge a player?)

 

[bbbt]

There's a machine learning AI for Freeciv (the online Civ 2 clone): https://arago.co/ai/freeciv/. It also has nice comparison showing how much more complex Freeciv (which is simpler than 6) is than chess or go. Which more complexity = more data needed. They actually have prizes for playing against it/'training it', which doesn't speak so well against the number of willing participants (granted I'm sure Freeciv's general pool is small): https://freeciv-leader-board.hiro.community/train

The issues are one: It would have to be online, and would require a significant number of people to be willing to use it to train it properly.

Another issue for civ: two, it would probably have to be un-modded (though UI mods would presumably be fine). The more 'rigid' an AI is, the less well it's going to be able to adapt to mods (i.e. one change to the rules, or new mods, could send the AI back to square one in some respects).

Finally, three: as mentioned by folks, the way the AI might learn to play the game in the 'optimum' way (rushing, or maybe chopping it's way to a turtled science victory), might not be perceived as fun by most of the players.

Having said that, I think there might be some potential for isolated segments - like something that is just training the tactical engine for instance, with some manual input. But it's not a simple undertaking at this point.

 

[ShallowSeas]

There are some problems:



1.Firaxis has no real incentive to develop such a system even if they could. It is true that there are a lot of people who complain about how bad the AI is, but you have to remember that these people represent only a small segment of the overall playerbase. The average Civ player would probably struggle badly on emperor difficulty. So why would casual players want to play against this hyper intelligent AI that would just stomp them even quicker? Would they even be good enough to appreciate the intelligence of the AI?

2.Civilization is a far more complex game for an AI to learn than either Chess or Go. It took years and bucket loads of money to develop an AI that could beat the best human Go players. Where would Firaxis find the money and the expertise to not only equal Google's efforts, but surpass them?

3.Many of the people who complain about the AI are subject to the Dunning Kruger effect and believe that the only reason Firaxis doesn't create hyper intelligent AI is because they are Lazy. Such people would be unlikely to pay $15 a month for a feature they think should already in the game.

4.If such a service existed it is inevitable that there would be longer turn times (AlphaGo takes a lot of time to “think”) and lag. So, gamers who were willing to pay $15 a month to flesh out one of the games features would have to suffer through these problems also.

5.I'm a little out of my depth here, but I doubt $15 a month would be enough to cover operating costs, let alone turn a profit. Again, I don't have a lot of technical expertise, but I think it is safe to assume that running a bunch of WoW servers is cheaper and easier than what you're proposing.

6.I don't even think it would be good from the PR standpoint. Civ is not an E-Sport, there are no Civ champions to pit against the AI. When AlphaGo beat the best Go player in the world it was a big deal. It would not be a big deal if a hyper advanced Civ AI beat some random, fifty sub, Twitch streamer.

7.By far the biggest problem of all, is that someone who is desperate for an opponent that is as smart as a human could just play against actual humans in multiplayer and avoid wasting $15 a month.



I agree that its fun to fantasize about hyper intelligent AI in our games, but, for the moment, it isn't realistic. Perhaps we should revisit this discussion in ten years.

 

[Kyro]

I think a better example would be Elon Musk's OpenAI that managed to beat professional Dota 2 players in certain settings with just 2 weeks of learning.

The limitations about AI "learning" in such instances is the sheer traffic of iterations required in order for it to arrive at the optimal choices and the greater the possibilities and choices in the game the larger the number of iterations required at an exponential increase rate. (Learning through pure trial and error)

The other limitation is that the true hallmark of intelligence which is flexibility cannot be created this way as this "intelligence" only figures out what is "Meta" in the game and the moment you create a unique scenario where flexibility is required the AI immediately fails.

 

[aimlessgun]

But it is actually still necessary to program such an AI?

Interesting idea, but even if you had the resources and the go-ahead from Firaxis leadership, you'd probably still need to do a lot of programming. Because the objective of the Civ AI is not just to win. You also want your AI to give players fun games, be immersive, have some personality, stuff like that (not saying the current AI does any of those things either). AlphaCiv figuring out the optimal meta and beating you with that strat every game would have none of that.

So you'd have to layer a bunch of weights and special conditions and stuff on top, which is a bunch more work and I'm not sure if there are examples of successful machine learning AIs which had objectives beyond just 'win'.

Maybe the technique could be deployed vs specific situations though, like given an army and a target city, use deep learning to train the AI to mount an effective attack.

 

[stealth_nsk]

There's no goal to create strongest game AI. The goal is to provide best user experience. Just to say:
1. Imagine Firaxis made an AI matching top players. This means best players will be able to win only 20% of the game with 5 civs on the map. Weaker players will be unable to win at all. Game will fail.
2. AI needs to provide some kind of immersion. Remember rant about vanilla Civ5 AI backstabbing? Believe me, with optimal playing AI it would be much worse. This would cause outrage.

Not to mention technical difficulties:
1. The database required for this will be huge. It will take a lot of space on user computers and a lot of traffic to exchange game analysis.
2. Self-taught AI works well if rules doesn't change. With more or less regular patches (and during development new versions come out weekly or biweekly) the AI will not have time to learn. The decisions of the AI after each patch will be much more weird than they are now and it will be impossible to predict how any game change will affect AI behavior.
3. The development cost is actually huge. It's not something like "let's just add tensor flow to the system". It needs to have ins and outs configured for every game mechanic.
4. Each user will be required to have small personal supercomputer. Neural networks learn deep, but resources they require are monstrous.

Overall it's one of the worst ideas I've heard about civ game.

EDIT: Forgot one point to the first list:
3. One of the roles of AI in games like civ is demonstrate various mechanics to player and force players to use as much of the game as possible. "Effective" AI will use what's effective instead, probably ignoring game elements which doesn't fit the strategy.

 

[TheMeInTeam]

This is a sure fire way to demonstrate exactly how terrible the mechanics that sit behind this game are.

One week after any kind of machine learning is implemented, all AI on all difficulty levels will be swordsman rushing you to crap.

It’s theoretically a good idea, but until there’s a balanced game for it to operate in, it will 100% turn this into COD modern warfare 5000

It's dangerous to make assumptions. In a 1v1 it might use demographic data and warrior choke you to death or something if you underbuild military. Since it won't need much time for demographic analysis and can learn the implications of them given enough iterations it could pattern-react with near map-hack levels of efficiency.

I'm not convinced a sword rush always wins in such a scenario. Alpha zero does some moves in games that even professionals shy away from doing, I've seen several of its chess games vs stockfish engine.

1. Imagine Firaxis made an AI matching top players. This means best players will be able to win only 20% of the game with 5 civs on the map. Weaker players will be unable to win at all. Game will fail.

* assuming no bonuses or penalties to anybody.

Nothing about Civ 5 AI was actually in the realm of "trying to win" effectively. It's never existed in a civ game, let's not pretend it has as a way to refute it.

All that said, such an AI in civ isn't practical. Cost is still an object.

3. One of the roles of AI in games like civ is demonstrate various mechanics to player and force players to use as much of the game as possible. "Effective" AI will use what's effective instead, probably ignoring game elements which doesn't fit the strategy.

Any time that happens it's a cop-out by the design team and should be treated as such. If game elements in a strategy game are *consistently* not effective to the point that a theoretical optimal player doesn't use them, it's an irrefutable indictment on the design/implementation of said game elements.

False choice should not be celebrated in strategy games.

 

[Lazy sweeper]

I like the Freeciv deep learning stuff, I played it on Linux machines a while ago, I must check that out, but the first considerations is that such a system must not have enormous costs, given is a free game for Linux users.
The idea to try build such an AI network for CIV V online is compelling, but once that AI has learned how to play all various leaders (Or should we think that multiple AI are needed? That seems less difficult than programming one AI to adapt to certain aggression/expansion/trade/science/etc levels). So One AI with more variables or multiples AI, that is, then Players would have to check this Online AI only as a difficulty option, so it's actually like going Multiplayer on a private server, but just you and 4/5...9/ computer AI.
Another consideration is that many players have good GPUs these days and lot of computing power, so allowing only certain system to establish a peer2peer deep learning network, does sound actually right, that AI will live
only in this online world, and will share and evolve on other players systems as long as they are connected to the network, sort of everyone would need to establish a private server and mantain it, or it would not be sustainable. In any given moment, there must be at least one peer active to mantain the AI alive, but it's actually shared and multiplied on every system... I don't know if this makes sense, if this is just how OpenAI works.

Secondary thoughts on AI:
How does AI get to know how many military You have? Where are they located? Does they need scouts or it immediately knows all of your unit positions on the map? I still prefer CIV III to IV/V/VI for the actual gameplay, but AI would get straight to attack the city in your empire with just one warrior defending it, or stalk your privateers wherever they are as if they had military satellites always active, spy networks you can only dream of. CIV VI gives me the impression AI knows the same way everything, then denounces you (piss off Rome, you can't conquer a city of mine with 30 units, trebuchets, etc, and I have just walls and a Bowman defending...)

Back on track:
Funding the project for an Open AI network: Good idea or no?
Would Firaxis start a campaign for Kickstart an Open AI network? No.
Can someone else who knows the subject do it? Yes.
Would people fund it despite it's unofficial? Yes, but not if it would require a monthly subscription.
Would it require a special server to set it up? Yes or not. See above for peer2peer network.
Would Firaxis buy the AI and the network model if this would actually work? If it is an OpenAI couldn't them just get it and replicate it? Yes/No/Maybe
Would the kickstarters lose the money they got in the first place? Not at all

Of course this is just personals thought, but it's actually a good advice to put some questionmarks on the board wether none had been set before.

(btw, where are my Templars?????)

 

[kryat]

Ignoring development cost and run-time issues, there’s still some other problems to this idea.

But first, a clarification. Deep learning does not happen on the spot. You build it ahead of time. The GPU’s would be owned by Firaxis to build the beast, but the beast could then run on your home computer.

Ok, onto the problems

1) The first problem is knowing how we want the AI to behave. This is likely to be divisive. Do we want an AI that achieves any achievable victory as fast as possible? Do we want a military-focused AI that isn’t interested in winning quickly, but winning mightily? Maybe both? We can’t specify tell the AI what to do unless we know what we want it to do.

2) Deep learning relies on having a very large, quality training set. This means it can only be trained by models we want the AI to mimic. Let’s say we can gather all saves of every VI game as a pool for training set, and narrow down from there which players it should mimic. Which ones do we use? Only the PvP games? Those tend to skew military. Would it be fun to play a game where all you’re doing is playing defense against a human-like player? What about diety SV rushes? Those often use exploits, which could be removed by the developers at anytime. Now let’s try to find only the games where the player never used an exploit. Good luck sifting for those. What about faith or culture? Or score even? Building a training set is far too complicated. Are any of these strategies valid against a fair player?

3) Most players would like a difficulty scale. There aren’t very good ways to do this. Beyond handicapping resources, and maybe setting calculation time limits, it’s really hard to make a computer play dumb, but we could mimic worse players. Let’s say we build a monster that beats players 99% of the time. Call that one diety. Do we try to train one that beats diety-level players only 50% of the time for immortal? 5%? Where do we get the training set for it?

4) A machine learning AI player is unlikely to be inventive, just mostly unbeatable, until it’s not. There’s no training for creative plays. I’m not saying you’ll see the same behavior every time, but it’s not going to usher in a new paradigm of play. Once humans figure out how to beat it (assuming that’s possible), then it’s boring again, unless we retrain it again. And the cycle repeats.

5) Overfitting the data is a huge problem in this field. Since the beast is copying the behavior of players, some quirks would arise. Few high-level players rush Stonehenge because they know the current AI can beat them to it, so it’s unlikely the new AI will try to build that as well. Now the assumption isn’t valid, and we’ve just found an exploit.

One more thing, that’s not a problem, but a side note. We probably also want to preserve the agenda systems, so they feel more like we’re playing against mildly interesting historical figures, not a human. That’s going to be difficult, but not impossible to incorporate. Mostly, it would probably involve building a training set based on humans role-playing the agendas.

 

[Josephias]

1) The first problem is knowing how we want the AI to behave. This is likely to be divisive. Do we want an AI that achieves any achievable victory as fast as possible? Do we want a military-focused AI that isn’t interested in winning quickly, but winning mightily? Maybe both? We can’t specify tell the AI what to do unless we know what we want it to do.

Indeed that's one of the biggest quirks. Hardcore strategist here seems to prefer winning in the least turns possible (as it is the most objective right now).

But the motto of Civilization game has always been "building a legacy that whill support the test of time", and I'd say this is not the least turns possible. Indeed you'd want a city with wonders (built by you preferably, but also conquered, tall cities, high yield output,.... It should be underestood that the current victory conditions are just "killer" moments in this regard.
Current scoring system is just not capable to manage all this (and, altough it will probably be easy to improve it, will it be so easy to take it to AI management level?. Due to the varied systems on the game, you'll want to factor in many things, and there you can mess a lot with weights... weight to much culture yield, and AI will stop building units... weight to much number of cities, and AI will spam them mindlessly... fine tune would be necesarry, but then you will need to re-learn.

 

[Pythakoreas]

My suggestion would be to start programming an AI that doesn't roll around in Industrial Era with Ancient/Classical Era units. Within the investment Firaxis can afford to make $80 CAD games they should be able to fix the simplest of mistakes. Many of the older RTSs I've played have AIs that can upgrade their units and those are from the 90's and early 00's. In Civ5 I played through some actual invasions, with Civ6 they took a step back and I rarely come across a warmongering AI. Given Firaxis' disregard for AI I don't think they'll even consider Deep Learning in the near future.

But if DL comes to life that would be awesome. I might end up playing against an AI that's genuinely competent rather than one that has a bunch of bonuses stacked against me. After AlphaGo beating the Go champions I will undoubtedly tell you that an AI competing in Civ is possible, it's only a question of spending. Go is infinitely more complex than any iteration of Civilization - I've seen forumers argue otherwise when there was Elon Musk's OpenAI discussion, clearly not familiar with Go - in fact, it was thought to be impossible for years to create an AI that can compete with the top players in Go given the sheer scale of possibilities. On the other hand it is definitely possible to make a good AI for Civ games without even relying on Deep Learning.

 

[TheMeInTeam]

Indeed that's one of the biggest quirks. Hardcore strategist here seems to prefer winning in the least turns possible (as it is the most objective right now).

But the motto of Civilization game has always been "building a legacy that whill support the test of time", and I'd say this is not the least turns possible. Indeed you'd want a city with wonders (built by you preferably, but also conquered, tall cities, high yield output,.... It should be underestood that the current victory conditions are just "killer" moments in this regard.
Current scoring system is just not capable to manage all this (and, altough it will probably be easy to improve it, will it be so easy to take it to AI management level?. Due to the varied systems on the game, you'll want to factor in many things, and there you can mess a lot with weights... weight to much culture yield, and AI will stop building units... weight to much number of cities, and AI will spam them mindlessly... fine tune would be necesarry, but then you will need to re-learn.

The AI should not be designed or programmed to throw. The game has stated win conditions. The AI should pursue whatever it evaluates as the best path for victory. Low turn count should only get priority insofar as that makes victory more likely. Cutting defenses to rush a win then losing to invasion = no win, should be preferred less.

In practice this likely means military victory, because the game isn't actually balanced to make all of its VCs viable when everyone on the board is trying.

Do we want a military-focused AI that isn’t interested in winning quickly, but winning mightily?

The goal is to reach a win condition. Doing other things on top of that is gravy. How fast doesn't matter. How much style doesn't matter.

Which ones do we use? Only the PvP games? Those tend to skew military.

If a theoretical optimized AI consistently picks military like PvP, other options are a false choice. To fix this, you need to fix the design, not AI, or conclude it isn't broken and center the game on war outright. Pretending it isn't a war game while placing incentives in a way where war is the obvious top choice is disingenuous.

A machine learning AI player is unlikely to be inventive, just mostly unbeatable, until it’s not. There’s no training for creative plays. I’m not saying you’ll see the same behavior every time, but it’s not going to usher in a new paradigm of play. Once humans figure out how to beat it (assuming that’s possible), then it’s boring again, unless we retrain it again. And the cycle repeats.

Still better than present AI, despite sharing some of its problems. If you even get different behavior that's viable it's a major step forward.

Most of the agendas are fundamentally flawed as a concept and effective AI should not employ them. You could somewhat lessen this issue by making more options competitively viable than they are at present.

 

[kryat]

The AI should not be designed or programmed to throw. The game has stated win conditions. The AI should pursue whatever it evaluates as the best path for victory. Low turn count should only get priority insofar as that makes victory more likely. Cutting defenses to rush a win then losing to invasion = no win, should be preferred less.

In practice this likely means military victory, because the game isn't actually balanced to make all of its VCs viable when everyone on the board is trying.



The goal is to reach a win condition. Doing other things on top of that is gravy. How fast doesn't matter. How much style doesn't matter.



If a theoretical optimized AI consistently picks military like PvP, other options are a false choice. To fix this, you need to fix the design, not AI, or conclude it isn't broken and center the game on war outright. Pretending it isn't a war game while placing incentives in a way where war is the obvious top choice is disingenuous.



Still better than present AI, despite sharing some of its problems. If you even get different behavior that's viable it's a major step forward.

Most of the agendas are fundamentally flawed as a concept and effective AI should not employ them. You could somewhat lessen this issue by making more options competitively viable than they are at present.

My point wasn’t that the AI can’t be programmed to win more efficiently, just that deep learning is not a vehicle that can get us there in a way that would make everyone happy.

 

[Gedemon]

After AlphaGo beating the Go champions I will undoubtedly tell you that an AI competing in Civ is possible, it's only a question of spending. Go is infinitely more complex than any iteration of Civilization

I'm curious, how it is more complex ?

 

[TheMeInTeam]

My point wasn’t that the AI can’t be programmed to win more efficiently, just that deep learning is not a vehicle that can get us there in a way that would make everyone happy.

"Making everyone happy" is closer to "moving humans at near lightspeed" than it is to "actionable on a budget with time constraints".

 

[Pythakoreas]

I'm curious, how it is more complex ?

The sheer scale of possibilities. There are 361 spaces on the Go board so every time a stone is put down the possibilities go like this:

361x360x359x358x357...

do this 200 times (until you come close to x160)

it is not even mathematically comparable to Civ games where you have a limited set of moves and it comes down to managing what you have to maximize your output

 

[TheMeInTeam]

The sheer scale of possibilities. There are 361 spaces on the Go board so every time a stone is put down the possibilities go like this:

361x360x359x358x357...

do this 200 times (until you come close to x160)

it is not even mathematically comparable to Civ games where you have a limited set of moves and it comes down to managing what you have to maximize your output

You're stuck working with incomplete information, and "possible moves" and "viable moves" have a large gulf between them in both games.