Some statistics pertaining to relative civilization/victory type power in ~100 AI-only games

[L. Vern]

Hello all,
for the purposes of tuning some technology cost-scaling numbers I wrote a small framework to automate the process of running autoplay games and saving their logs, as well as some pandas code to aggregate the information contained therein. I figured it might also be interesting to collect victory related information to see relative power levels of certain victory types and civilizations, and here they are:

Victory types:

Spoiler victory types :

(Yes there were no domination victories)

Civilizations

Spoiler civilizations :

​	Civ	Games​	Victories​	Winrate​	Culture Victories​	Diplomatic Victories​	Science Victories​	Time Victories​	Average Victory Score​	Average Score Rank​
1​	Arabia	16​	7​	0.44​	7​	0​	0​	0​	1911.94​	4.88​
2​	China	15​	6​	0.4​	5​	1​	0​	0​	2589.53​	2.8​
3​	Siam	20​	7​	0.35​	1​	5​	1​	0​	2329.8​	4.25​
4​	Austria	16​	5​	0.31​	1​	3​	1​	0​	2499.44​	3.62​
5​	Poland	17​	5​	0.29​	5​	0​	0​	0​	2485.53​	4.06​
6​	The Ottomans	19​	5​	0.26​	3​	1​	1​	0​	2536.11​	3.32​
7​	Greece	18​	4​	0.22​	0​	3​	1​	0​	2419.83​	4.17​
8​	Korea	18​	4​	0.22​	4​	0​	0​	0​	2133.44​	4.67​
9​	Morocco	14​	3​	0.21​	1​	2​	0​	0​	2288.36​	4.21​
10​	Brazil	26​	5​	0.19​	4​	0​	1​	0​	1977.31​	5.04​
11​	The Maya	21​	4​	0.19​	4​	0​	0​	0​	2253.76​	4.19​
12​	The Netherlands	21​	4​	0.19​	0​	4​	0​	0​	2024.76​	4.86​
13​	Egypt	16​	3​	0.19​	3​	0​	0​	0​	1976.19​	5.19​
14​	Germany	12​	2​	0.17​	1​	1​	0​	0​	2074.67​	4.42​
15​	Babylon	19​	3​	0.16​	1​	0​	2​	0​	2247.74​	4.42​
16​	Ethiopia	21​	3​	0.14​	3​	0​	0​	0​	2366.76​	3.71​
17​	Assyria	22​	3​	0.14​	3​	0​	0​	0​	1984.05​	4.41​
18​	The Iroquois	15​	2​	0.13​	0​	2​	0​	0​	3157.07​	1.8​
19​	India	15​	2​	0.13​	2​	0​	0​	0​	2138.8​	4.73​
20​	Rome	16​	2​	0.12​	1​	1​	0​	0​	1705.25​	5.12​
21​	Polynesia	17​	2​	0.12​	1​	1​	0​	0​	2366.76​	4.06​
22​	Venice	17​	2​	0.12​	1​	1​	0​	0​	2343.59​	4.41​
23​	Carthage	20​	2​	0.1​	0​	2​	0​	0​	1536.4​	5.65​
24​	Japan	11​	1​	0.09​	1​	0​	0​	0​	2082.09​	4.64​
25​	The Celts	23​	2​	0.09​	1​	0​	1​	0​	2043.96​	5.17​
26​	Songhai	14​	1​	0.07​	0​	0​	0​	1​	2519.86​	4.07​
27​	Portugal	14​	1​	0.07​	1​	0​	0​	0​	2407.36​	4.36​
28​	England	16​	1​	0.06​	0​	0​	1​	0​	1513.5​	6.12​
29​	Spain	17​	1​	0.06​	0​	0​	0​	1​	1631​	5.47​
30​	The Inca	19​	0​	0​	0​	0​	0​	0​	2477.74​	3.42​
31​	The Shoshone	16​	0​	0​	0​	0​	0​	0​	2310.94​	4.75​
32​	Mongolia	19​	0​	0​	0​	0​	0​	0​	2207.47​	4.37​
33​	The Aztecs	16​	0​	0​	0​	0​	0​	0​	2089.5​	4.31​
34​	Persia	17​	0​	0​	0​	0​	0​	0​	2073​	4.53​
35​	Denmark	15​	0​	0​	0​	0​	0​	0​	1989.47​	4.47​
36​	Indonesia	14​	0​	0​	0​	0​	0​	0​	1935.43​	5.5​
37​	Russia	17​	0​	0​	0​	0​	0​	0​	1904.76​	5.18​
38​	Byzantium	19​	0​	0​	0​	0​	0​	0​	1881.58​	5.11​
39​	America	17​	0​	0​	0​	0​	0​	0​	1868.53​	4.82​
40​	The Zulus	15​	0​	0​	0​	0​	0​	0​	1864.27​	5​
41​	Sweden	19​	0​	0​	0​	0​	0​	0​	1768.89​	5.37​
42​	The Huns	22​	0​	0​	0​	0​	0​	0​	1647.82​	5.18​
43​	France	13​	0​	0​	0​	0​	0​	0​	1494.77​	6.08​

Settings used (everything below no trade events is unchecked):

Spoiler settings :

If there's any interest I can clean up and github the automation code, it's nothing special and pretty bare-bones but will still resume from autosaves on recoverable crashes and save minidump/save files on CTDs before starting the next one.

 

[ilteroi]

this is excellent data! do you have anything on lower difficulty levels as well? does that have an influence?

 

[balparmak]

I'm loving these, great work! Also pretty much confirms that pure domination civs are the worst for the AI. Their notoriety comes from being an annoyance to the human player, and that's pretty much the only thing they're good for. The AI needs to become a lot better at capital sniping

 

[bahamut19]

It is interesting that the Iroquois would have the highest average score rank, yet only have 2 victories via Diplomacy.

 

[pineappledan]

These games were played on standard size and speed, with the default number of 8 major civs and 16 city-states?
I'm assuming that's the case, but it's important info to say outright, because I don't have a basis to say what a proper baseline win rate is unless you tell me how many players there were.
eg. 8 players where it's anyone's game should mean that all civs would be at a 12.5% win rate, but that number goes up or down based on the number of civs in a game.

Wonderful chart; thanks for sharing!

 

[Stalker0]

Merry Xmas! Now that's what we have been missing. This is very solid!

Now we just need to run it longer. 80 games isn't bad, but a couple of hundred would be a lot more solid, especially with the civ stats. Only seeing civ stats over 16 games isn't too solid, but if we can double or triple that number it should give us a great baseline to look at balance.

We can also use to see what government types the AI is picking, as I know there is always concern that one type of policy is chosen too much, so we should be able to see if any one policy tree is being taken far more often than the others.

 

[L. Vern]

These games were played on standard size and speed, with the default number of 8 major civs and 16 city-states?
I'm assuming that's the case, but it's important info to say outright, because I don't have a basis to say what a proper baseline win rate is unless you tell me how many players there were.
eg. 8 players where it's anyone's game should mean that all civs would be at a 12.5% win rate, but that number goes up or down based on the number of civs in a game.

Wonderful chart; thanks for sharing!

Good point, I have attached the settings (everything below no trade events is unchecked)

Spoiler settings :

You are correct that the expected outcome is 12.5% winrate per civ

 

[andersw]

Time, is that top score at max turn limit?

 

[KlHannibal2]

Nice work! I ran a few AI only games before (with less automation) and I was able to see an AI get close to domination victory a single time.

Of course it was Songhai, on a huge Pangea map. The game crashed when he had about 70% of the map. He had an inaane amount of units that just steamrolled everything and he got big by taking over the first to AIs using his river bonus.

A very rare exception.

 

[L. Vern]

Time, is that top score at max turn limit?

The scores displayed are all recorded at the moment someone won that game

Nice work! I ran a few AI only games before (with less automation) and I was able to see an AI get close to domination victory a single time.

Of course it was Songhai, on a huge Pangea map. The game crashed when he had about 70% of the map. He had an inaane amount of units that just steamrolled everything and he got big by taking over the first to AIs using his river bonus.

A very rare exception.

Yea, I think a big reason for the lack of domination victories was the communitas maps has all the civs on 2+ landmasses, and the AI isn't always the best at amphibious attacks. I would expect on a Pangaea map to see at least a few domination wins.

 

[nekokon]

I think the main reason for lack of Domination Victory is most likely because the AI just go full genocide even after taking enemy's capital, thus racking up a lot more warmonger penalty than needed, thus to have a Domination Victory likely they would have to complete painted like >80% of the world to win (thus estimatedly perform 4 times better than the rest of the world combined in score), while other victory only requires you to be about x1.5 times or less score than the next runner up.

A better fix would be changing how Domination Victory is decided, like controlling more than half or 1/3 the world's cities/population after year XXXX instead. That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.

 

[4CV]

....That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.

And again, we'd see another attempt to force the player to play in a certain way. Why should the player not sit and wait for the rocket to start? Everone who wants to fight will do it anyway...

 

[pineappledan]

For reference, this is the results of 93 games.

Here is a list of my own observations from this data:

Spoiler :

• Militaristic civs are the ones that players have the hardest time dealing with and are effective in the player hands. These stats indicate to me that these AI-only games don't get us closer to leader balance, but they might still help for things like policy or victory balance.
• That having been said, if I were still to try to use this data to balance civs:
  • CV-oriented civs are top of the current meta, to no one's surprise.
  • Arabia in particular is exceptionally well-focused into a CV monster. He is not only winning 44% of his games, he appears to be winning so quickly and easily that he tanks his own victory scores by winning games too fast.
  • China doesn't have much overt CV-orientation, but she is the civ that is both consistently winning (2nd highest; 40%) and has consistently high scores (2nd highest; 2590), indicating that China is
  • On the other hand, France is the worst on every metric. No wins, lowest average rank (6th), lowest score; the only victory score that is below 1500, in fact. This aligns with my own personal experience of his kit being bad, and the AI being bad at using it.
  • While CV civs are top dog right now, it's specifically civs with tall bonuses, science bonuses that can be recruited for a CV, or civs with GW bonuses. wide CV civs that have culture on tile bonuses that can translate into tourism are being left behind. Polynesia -- the purest wide CV civ -- is tied for 21st, and France is 43rd. The exception is Brazil, but that's because people are playing him tall right now, his kit is ridiculously overtuned for GA/WLTKD bonuses, and so he doesn't even rely on his Brazilwood UI to win. This suggests to me that while CV needs to be made harder, its should be made harder in ways that won't also nerf things like the / tile converters.

It is interesting that the Iroquois would have the highest average score rank, yet only have 2 victories via Diplomacy.

Spoiler graph :

winning and score is only weakly related (r^2 = 0.16)
Iroquois have an average end score that is 22% higher than the next highest, China, but are tied for 18th place for win rate
Meanwhile, Arabia is 32nd overall for end score, but #1 for win rate

Average win score is 2118

A better fix would be changing how Domination Victory is decided, like controlling more than half or 1/3 the world's cities/population after year XXXX instead. That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.

This is similar to what I would want a religious victory. You have to convert X% of the world population to your religion to win, and that can include by the sword, but it's pop-based rather than city-based.

 

[nekokon]

And again, we'd see another attempt to force the player to play in a certain way. Why should the player not sit and wait for the rocket to start? Everone who wants to fight will do it anyway...

It's for balance reason, to give DV civ a better chance of actually winning. If you don't like a certain way the game "forces" you to play just change the setting and don't use DV at all, since it doesn't actually do anything right now regardless.

You complaining about being "forced" to play more aggressive to prevent DV would be just like someone else complaining about being "forced" to play peacefully with strong CV civ like the current meta, and everyone who want to play peacefully... wait that already 90% of the AIs and majority of players because that's exactly what a meta is.

 

[CppMaster]

winning and score is only weakly related (r^2 = 0.16)

Yeah, but that shouldn't matter, because the better you play, the earlier you can win, so you will acumulate less score, but on the other hand, you probably would get more score per turn. Too bad we don't have data on average game length.

 

[Legen]

CV-oriented civs are top of the current meta, to no one's surprise.

Diplomatic civs are also doing great, just check Siam and Austria position. Greece and Germany are also above average as well.

And curiously, Venice is in the middle of the pack, at 0.12 win rate and 22th position.

 

[4CV]

It's for balance reason, to give DV civ a better chance of actually winning. If you don't like a certain way the game "forces" you to play just change the setting and don't use DV at all, since it doesn't actually do anything right now regardless.

You complaining about being "forced" to play more aggressive to prevent DV would be just like someone else complaining about being "forced" to play peacefully with strong CV civ like the current meta, and everyone who want to play peacefully... wait that already 90% of the AIs and majority of players because that's exactly what a meta is.

Well, if you consider unwelcome questions being complaints we better not discuss them anymore. But that does not make the question being answered.

 

[nekokon]

It's already answered, to give DV civ a better chance of actually winning. Right now there's almost 0% of it happening.
And I considered that a complain and not a question because "force" is a pretty strong word.

 

[zeofig]

That's really cool. I would love to see MORE data!

 

[Skidizzle]

Is it possible to put a URL link to a surveymonkey in the one more turn window to get player data? Maybe with an auto generated .rtf that we could attach to the survey?