AI beats top human player. Yet another AI discussion

[Karmah]

Thats already what everyone is saying I think. The question raised by the thread is exactly on the estimation part. Either its done by a logic based algorithm like in civ5 and civ6 and we only wish it was better made or we go an eentirely different route where the AI is trained to do this estimation. Computation of the weights is intensive but that would be done on Firaxis side not by the player. The player would have only access to the end result that is being updated each patch. For something like buildings the networks would not be very deep or have that many inputs and vectorized calculations may be not slower than the currentfor loops Firaxis uses anyway.

However all this raises quite a lot of questions when it comes to modability of the AI.

Actually I'm pretty sure that the future of AI in games lies in data collection from the game played with level estimation and go machine learning. As you said the training part would be on the dev sides , clients would just get some implementation based on the output model.

 

[Kurnn]

Actually the future of AI is kinda scary, but good for games like this .
"Learning" are all about cause and effect and understand the "function". In machine language that's "Sub-rutines".

Example: Drop an item and it falls down = learning. Computers can do this, today.
To understand why the item falls down and not left, is understanding the "function". Computers can not do that, yet.
However, computers can handle over 100.000 sub-rutines per second today, which leads to autopilot of cars...in traffic. (I work with these things).

I see AI improving vastly like every few months, because you add more and more "sub-rutines", which makes AI react to more things then we can hold in our heads . Like chess software today, they "know" every possible move, and the effect of every move.

This is where I see Civ AI moving.

(Untill the day Skynet gets selfaware)

 

[gunnergoz]

FWIW I just saw the film Alien Covenant and was impressed at how it set out a battle of AI vs AI - one human-like (self-aware, creative and psychologically/murderously flawed) and one less human (more drone-like, obedient, aware, not creative, but perfectly able to execute its duties while not threatening humans.) It was a scary scenario to contemplate. Perhaps I don't want a civ game that will beat me at the game, then beat me with the keyboard afterwards.

 

[miaasma]

As someone who avidly plays dota 2, Dendi is not a top player. Him and his team did not make it to the group stage of the biggest dota tournament in the world (TI) which ended on saturday. He was previously a top player, and is still the biggest household dota player name because of his personality and performance in the 1st, 2nd, and 3rd TI, but this was the 7th TI and he is nowhere near the top of the field. Miracle- for example would be considered one of the best players in the world right now, but there is also every other mid player that made it to group stages.

this bot also beat arteezy and sumail

 

[Ferocitus]

I see AI improving vastly like every few months, because you add more and more "sub-rutines", which makes AI react to more things then we can hold in our heads . Like chess software today, they "know" every possible move, and the effect of every move.

This is where I see Civ AI moving.

You are getting carried away by hype and unsubstantiated nonsense, the same
ones in a similar environment that led to previous "AI Winters".

What you have described are nothing more than a few good statistical tools and a
collection of adaptive algorithms. Good for simple games, and many other routine
tasks, but hardly intelligent, and certainly not even close to what is expected of
Artificial General Intelligence machines/organisms.

I'm not saying we won't be able to have incredibly powerful AI one day, but forget
about silicon-based hardware as the way there. From an energy perspective it's
extremely inefficient and very expensive.

If you have the choice between a University or a caravansary the choice is
likely obvious though.

To a human.

What resources are you prepared to devote to getting that decision right every
time, given that it will take resources away from other branches of the AI,
and/or be unbearably slow?

The thread is more about what could be done not what Firaxis will do.

Interesting that you begin that process by throwing around incantations like:

"The principles of machine learning..."

"It would make the AI always pick the best options..."

From that last line, I'm guessing that you don't have any actual programming or
AI experience.

We all know what they'll do: the bare minimum.

Oh, you're one of those.
But Ok, I'll play: they are in the business of maximising profit for minimum
effort.

Thats not the way the decision was taken though. If I play random moves in chess
sometimes I will make a good move. I still think an algorithm that does that is
trash.

If a Civ algorithm chose a random move from all possible moves in that
situation, it would certainly be abysmal trash. But it didn't.

 

[ztrapon]

Isn't video-game AI actually three-fold:

1) Speed and Precision
The "button" factor -- fast and precise execution of commands in real-time. Completely irrelevant to Civilization, but a major factor in for example sports, racing, fighting and RTS games (and DOTA too?)
On this front humans can never outperform optimized AI, since a program can perform pixel-precision tasks with 100% accuracy.

2) Combinatorics
Volume of information to process, relevant when all information is available to both players. Again, humans don't stand a chance in games like chess. A factor in Civilization, especially combat tactics, but not huge.

3) Judgment
This is where things get more complicated. In Civilization, or poker, much of the information needed to make the optimal decision is hidden, so you're inevitably making decisions in the dark. The best move can become the worst, and vice versa. In Texas Hold' Em for example you may bluff, or get lucky. Or in Civilization, for example settling in what appears to be an inferior position, only to discover a vital resource there once you research Bronze Working.
Sometimes you can't even confirm whatever assumptions you made, like when folding in poker. So you're not even learning, only making more or less rational guesses.

What an AI could do here would be to empirically observe gross amount of human games, and imitate (just like the FreeCiv AI mentioned earlier). But even so, the skill and mistakes of the humans it observed, would be reflected in its play style. And the AI wouldn't strictly speaking be superior, only an average derivative of its best player input.
(Which I also suspect is the reason there's online poker and Hearthstone: if one could write an AI guaranteed to win in games like that, it wouldn't be viable to have them running.)

I'm not a computer scientist, or mathematician, so maybe I'm just talking out of my ass here. But what exactly is the inherent edge of being a computer, in a game like Civilization? It might actually be disadvantageous to act rational and be predictable all the time.

 

[Victoria]

Dota is really a team game but let's not ignore that AI's are now being used in earnest everywhere and big bucks are being spent. In my company I was shown an AI project that will make hundreds of people redundant just in our company and we were asked about strategies to implement it without getting mass demoralisation during transition.

An AI is not a human, it is alien, a different species. Human vs AI is a ridiculous comparison. The only valid dota comparison is human vs human or AI vs AI. They work in different ways. Just human lag alone is enough to ensure an AI will win eventually, I am not surprised it is now.

The money it takes means the real dota challenge was likely a marketing one. When game shall we choose? .... dota or civ..... Dota wins that competition. Firaxis would never employ this level of AI without having alterior motives and exterior interests.

 

[TheMeInTeam]

But Ok, I'll play: they are in the business of maximising profit for minimum
effort.

Effort is only part of a cost proposition, and I wouldn't want the confusion of making the two out as the exact same thing. The more general max profit for cost is probably a way to look at it, though one could make a case that they are not optimizing for it very effectively when it comes to long term margins.

 

[Ferocitus]

1) Speed and Precision
The "button" factor -- fast and precise execution of commands in real-time. Completely irrelevant to Civilization, but a major factor in for example sports, racing, fighting and RTS games (and DOTA too?)
On this front humans can never outperform optimized AI, since a program can perform pixel-precision tasks with 100% accuracy.

To counteract that advantage is actually very easy.

Sample the click rates and accuracy from a lot of players, and then construct
appropriate probability distributions. (There's definitely no shortage of data
for StarCraft for a huge range of player abilities and in-game situations.)

In short, the AI's click rates and accuracy can be forced to be, on average, the
same as the class of player(s) it is playing against. It is easy to add or
subtract small random time increments from that average click rate (with the
same variation around the average as the human player), and to make it more or
less precise.

There are many other effects that can be included: e.g. tiredness, hesitation,
time since last coffee, nervousness or calmness about the current position in a
tournament, etc, etc.

Many facets of Esports science have similarities and exact counterparts to those
in well-established physical sports sciences.

2) Combinatorics
Volume of information to process, relevant when all information is available to both players. Again, humans don't stand a chance in games like chess. A factor in Civilization, especially combat tactics, but not huge.

Not quite.

If I asked you to find the shortest path around 4 random cities on a map of the
USA without visiting any of them twice you coud probably do that very easily.
All information is known to you and an "AI".

If it was 512 cities (a standard travelling salesman problem benchmark), you
would struggle to beat the best distance found by some cheap hardware and freely
available algorithms.

However, there are also many similar situations where humans can outperform
computers...

3) Judgment
This is where things get more complicated.

So much so that we could spend years discussing it!

Yes, computers can out-grind the best players in the world in chess and some
simple board games. However, there are many situations, where a human can size
up a position on the board in a fraction of a second, whereas a computer might
never find the solution.

In Civ, you might want to stop another civ from knowing about a certain part of
the map. You know that you can just buy a tile and the AI won't get into that
region for a long time, if ever, without declaring war. Computer algorithms can't
do that.

We are able to imagine future situations. For example, suppose that there were
mirrors at the ends of a chess board, and that pieces could move off the end
of the board and land back where they were reflected in the mirror.
Humans can see possibilities and difficulties with such a rule change in a very
short time. Computers cannot.

Great questions, by the way!

 

[Ferocitus]

Effort is only part of a cost proposition, and I wouldn't want the confusion of making the two out as the exact same thing. The more general max profit for cost is probably a way to look at it, though one could make a case that they are not optimizing for it very effectively when it comes to long term margins.

Yes, in other words neither you nor I have any idea what their plans
are for Civ6, any future enhancements or releases, or how much
effort they have put into the AI, or will in the future.

And we certainly have no idea about their business model, their
profits, the corporate knowledge they have accumulated through
the development and monitoring of Civ6, and millions of other
facets, so judging whether they are effective in the long-term is
impossible. (I was just replying to Acken's tiresome line.)

 

[TheMeInTeam]

Yeah fair enough. That's also why I said a case could be made, but there's not easily available information to make it rock solid. I can make a solid case the UI is terrible by any reasonable objective measure, but not that such will necessarily destroy the game's long-term, similar with AI quality or quality of the mechanical interactions. Firaxis has been at least decent with that last one.

 

[ztrapon]

We are able to imagine future situations. For example, suppose that there were
mirrors at the ends of a chess board, and that pieces could move off the end
of the board and land back where they were reflected in the mirror.
Humans can see possibilities and difficulties with such a rule change in a very
short time. Computers cannot.

This is really fascinating, and I suspect why quantum computers are such a thing, no? I mean from the way it sounds they resemble the human brain a lot more.

 

[elitetroops]

3) Judgment
This is where things get more complicated. In Civilization, or poker, much of the information needed to make the optimal decision is hidden, so you're inevitably making decisions in the dark. The best move can become the worst, and vice versa. In Texas Hold' Em for example you may bluff, or get lucky. Or in Civilization, for example settling in what appears to be an inferior position, only to discover a vital resource there once you research Bronze Working.

When dealing with unknowns or random outcome, short term success is irrelevant to judging the best move. Bankroll considerations aside, in Texas Holdem if you face an all-in preflop and have AA as your hole cards, it is the best move to call. Always. If you then end up losing the hand, your decision to call did not get any worse at all. It was still the best move and the correct play, you just got unlucky that one time. Similarly, settling an inferior position is always a bad move, even if something unexpected would happen at a later time to make it a seemingly better decision.

The point is, even if there are unknowns involved, you can still determine the best decision based on expected outcome. Things might not always end up as expected, but in the long term the winner is the one who makes the best decisions according to expectations. The more randomness and luck is involved, the larger sample size you need to determine who is really the better player and making the better decisions. In poker, you'd need thousands of hands. In Civ, you don't need that many games, because each game on it own already includes a lot of different luck based factors, which tend to even out to some degree. But still you can't decide it based on only one game, much less on the outcome of some single decision.

 

[ztrapon]

When dealing with unknowns or random outcome, short term success is irrelevant to judging the best move. Bankroll considerations aside, in Texas Holdem if you face an all-in preflop and have AA as your hole cards, it is the best move to call. Always. If you then end up losing the hand, your decision to call did not get any worse at all. It was still the best move and the correct play, you just got unlucky that one time. Similarly, settling an inferior position is always a bad move, even if something unexpected would happen at a later time to make it a seemingly better decision..

Well that was the point in my post, that some actions are not as straightforward as "do X, get Y", so you will be using judgment and taking chances. And when you say "best move", that means most rational decision, not optimal outcome. The way I imagine, in either poker or Civilization, an AI consistently making rational, predictable moves, would be punished by a human player, who is more concerned with the best outcome, and thus may gladly play irrational based on his/her judgment.

Poker really isn't that simple. One of the best players ever is Daniel Negreanu, and he's known for wild and risk-taking play. Obviously somewhere along the line he's probably using some sort of judgment (as opposed to just playing willynilly), but how would one translate that to an AI? Give it a set chance to bluff? Make it play irrational a certain percetange of hands? One thing is sure, and that is he is not playing rational decisions for long-term, every hand he plays, because anyone could repeat that.

Similarly in Civ, how would you design an AI to understand things like being tricked, reading its opponents, or judging whether to take a risk or not? Maybe those things all boil down to numbers too, I really have no idea, but I perceive it as a kind of decision-making that's inherently non-computer-like. Everything contemporary programs do follows predictable patterns and pre-determined rules by their programmer.

 

[Acken]

To a human.
What resources are you prepared to devote to getting that decision right every
time, given that it will take resources away from other branches of the AI,
and/or be unbearably slow?

If it's really slower than say current implementation in Civ5 (I don't know Civ6 AI sorry). I'm really not so sure unless you really go heavy on it. See the current algorithm still processes a lot of information through for loops and doing weights calculations anyway.

Interesting that you begin that process by throwing around incantations like:
"The principles of machine learning..."
"It would make the AI always pick the best options..."
From that last line, I'm guessing that you don't have any actual programming or
AI experience.

What are you even talking about lol.
How do you go from "The thread is more about what could be done not what Firaxis will do." to "
From that last line, I'm guessing that you don't have any actual programming or
AI experience."
That famous line when arguing techniques. So classy. There must be the equivalent of the Goodwin Point for that too.

Also that second quote is a misunderstood hyperbole I think. My point was that it would simply converge to all AI doing more or less the same thing and picking the most valued ( what I meant by best) options (like universities over caravansaries for example. Or that aqueducts are a priority, never picking Piety etc). Not that it would always play like the best. And my point was that we may actually not want that as long as the game remains strongly imbalanced.

Oh, you're one of those.
But Ok, I'll play: they are in the business of maximising profit for minimum
effort.

Yes. That's basically what I said. Point ?

If a Civ algorithm chose a random move from all possible moves in that
situation, it would certainly be abysmal trash. But it didn't.

If it feels as if, the judgement of the whole system is similar. Ok to be fair it's not random... but it's certainly no good. One of the first reason being that the programmer is probably not a very good civ player to begin with. This is a serious issue when you try to make an AI through logic based rules.


Also I'm not sure if you think people here believe that it would necessarily become a top level player. People are just considering that it can improve it. To make it suck less. To make it clear what is your opinion on Civ6 (or Civ5) AI to begin with ?

 

[elitetroops]

One thing is sure, and that is he is not playing rational decisions for long-term, every hand he plays, because anyone could repeat that.

He most definitely is making rational decisions all the time, but the untrained eye cannot spot one when it sees one. If you studied poker a bit more, you would also be able to see the reasons behind what you right now might consider "crazy risk taking". There's for example nothing irrational about successful bluffing. You can construct game theory optimal rules for the exact frequency you should bluff in any given situation. If you were to follow these rules exactly over an infinite set of hands, then nobody can win any money from you, only lose money, unless they also play GTO, in which case in the long term you end up even. GTO is far from the most profitable though, because that doesn't take the opponent's tendencies into account. The pro poker player would of course also analyze the opponent and adjust accordingly. As would a good AI, because adjusting to your opponent and exploiting their mistakes is the rational thing to do.

Anyway, poker is quite a simple game for an AI. In Limit Hold'em computers reached the level of top pros a long time ago. No limit Hold'em is more complex, but earlier this year someone made an AI that beat several top pros at that over a few hundred thousand hands.

Many aspects of Civ are way more complex and don't reduce to numbers quite as easily. Still, I'm sure the AI could learn to recognize a lot of patterns in human behavior if fed with enough games to study, and through this learn when it is being tricked and how to deal with it. However, in the current state of the AI, there are way more pressing issues that would be so much easier to fix...

 

[Acken]

Many aspects of Civ are way more complex and don't reduce to numbers quite as easily. Still, I'm sure the AI could learn to recognize a lot of patterns in human behavior if fed with enough games to study, and through this learn when it is being tricked and how to deal with it. However, in the current state of the AI, there are way more pressing issues that would be so much easier to fix...

Wholeheartedly agree on both points.

 

[Ferocitus]

This is really fascinating, and I suspect why quantum computers are such a thing, no? I mean from the way it sounds they resemble the human brain a lot more.

No and no.
Brains don't think. Humans do.

Maybe those things all boil down to numbers too...

Yes, because that's all they can work with.
Computer programs are just very long (and in many cases, "illegal") numbers
or groups of numbers.

Similarly in Civ, how would you design an AI to understand things like being tricked, reading its opponents, or judging whether to take a risk or not?

I doubt that Civ developers would be thinking about anything like that
level of complexity for their AI. It would take huge amounts of research time
to even discover and categorise all of those human "tricks", let alone write
the program to work on the hardware most players are likely to have, and
be fast enough not to ruin the game.

Just to show you what kinds of issues are involved, here's a paper published
two days ago by combined research teams from DeepMind and Blizzard.
StarCraft II: A New Challenge for Reinforcement Learning
https://arxiv.org/pdf/1708.04782.pdf

Keen students like yourself will probably want to have a look at all the
papers in the bibliography too.
Havagoodweekend!

Ok to be fair it's not random... but it's certainly no good. One of the first reason being that the programmer is probably not a very good civ player to begin with. This is a serious issue when you try to make an AI through logic based rules.

So, "the programmer" also designed the whole AI system.
That explains a lot.

 

[Acken]

So, "the programmer" also designed the whole AI system.
That explains a lot.

... obviously it's not a single person... Does that have to be explicitly written ? I feel so sorry

That conversation is obviously not productive. Moving on.

 

[Ferocitus]

... obviously it's not a single person...

Shame. It explained why multi-player isn't working well.