Some statistics pertaining to relative civilization/victory type power in ~100 AI-only games

L. Vern

Chieftain
Joined
Sep 5, 2022
Messages
29
Location
Ontario, Canada
Hello all,
for the purposes of tuning some technology cost-scaling numbers I wrote a small framework to automate the process of running autoplay games and saving their logs, as well as some pandas code to aggregate the information contained therein. I figured it might also be interesting to collect victory related information to see relative power levels of certain victory types and civilizations, and here they are:

Victory types:
Spoiler victory types :


Photoshop_3gBoG9aXnQ.png



(Yes there were no domination victories)

Civilizations
Spoiler civilizations :

Civ
Games​
Victories​
Winrate​
Culture Victories​
Diplomatic Victories​
Science Victories​
Time Victories​
Average Victory Score​
Average Score Rank​
1​
Arabia
16​
7​
0.44​
7​
0​
0​
0​
1911.94​
4.88​
2​
China
15​
6​
0.4​
5​
1​
0​
0​
2589.53​
2.8​
3​
Siam
20​
7​
0.35​
1​
5​
1​
0​
2329.8​
4.25​
4​
Austria
16​
5​
0.31​
1​
3​
1​
0​
2499.44​
3.62​
5​
Poland
17​
5​
0.29​
5​
0​
0​
0​
2485.53​
4.06​
6​
The Ottomans
19​
5​
0.26​
3​
1​
1​
0​
2536.11​
3.32​
7​
Greece
18​
4​
0.22​
0​
3​
1​
0​
2419.83​
4.17​
8​
Korea
18​
4​
0.22​
4​
0​
0​
0​
2133.44​
4.67​
9​
Morocco
14​
3​
0.21​
1​
2​
0​
0​
2288.36​
4.21​
10​
Brazil
26​
5​
0.19​
4​
0​
1​
0​
1977.31​
5.04​
11​
The Maya
21​
4​
0.19​
4​
0​
0​
0​
2253.76​
4.19​
12​
The Netherlands
21​
4​
0.19​
0​
4​
0​
0​
2024.76​
4.86​
13​
Egypt
16​
3​
0.19​
3​
0​
0​
0​
1976.19​
5.19​
14​
Germany
12​
2​
0.17​
1​
1​
0​
0​
2074.67​
4.42​
15​
Babylon
19​
3​
0.16​
1​
0​
2​
0​
2247.74​
4.42​
16​
Ethiopia
21​
3​
0.14​
3​
0​
0​
0​
2366.76​
3.71​
17​
Assyria
22​
3​
0.14​
3​
0​
0​
0​
1984.05​
4.41​
18​
The Iroquois
15​
2​
0.13​
0​
2​
0​
0​
3157.07​
1.8​
19​
India
15​
2​
0.13​
2​
0​
0​
0​
2138.8​
4.73​
20​
Rome
16​
2​
0.12​
1​
1​
0​
0​
1705.25​
5.12​
21​
Polynesia
17​
2​
0.12​
1​
1​
0​
0​
2366.76​
4.06​
22​
Venice
17​
2​
0.12​
1​
1​
0​
0​
2343.59​
4.41​
23​
Carthage
20​
2​
0.1​
0​
2​
0​
0​
1536.4​
5.65​
24​
Japan
11​
1​
0.09​
1​
0​
0​
0​
2082.09​
4.64​
25​
The Celts
23​
2​
0.09​
1​
0​
1​
0​
2043.96​
5.17​
26​
Songhai
14​
1​
0.07​
0​
0​
0​
1​
2519.86​
4.07​
27​
Portugal
14​
1​
0.07​
1​
0​
0​
0​
2407.36​
4.36​
28​
England
16​
1​
0.06​
0​
0​
1​
0​
1513.5​
6.12​
29​
Spain
17​
1​
0.06​
0​
0​
0​
1​
1631​
5.47​
30​
The Inca
19​
0​
0​
0​
0​
0​
0​
2477.74​
3.42​
31​
The Shoshone
16​
0​
0​
0​
0​
0​
0​
2310.94​
4.75​
32​
Mongolia
19​
0​
0​
0​
0​
0​
0​
2207.47​
4.37​
33​
The Aztecs
16​
0​
0​
0​
0​
0​
0​
2089.5​
4.31​
34​
Persia
17​
0​
0​
0​
0​
0​
0​
2073​
4.53​
35​
Denmark
15​
0​
0​
0​
0​
0​
0​
1989.47​
4.47​
36​
Indonesia
14​
0​
0​
0​
0​
0​
0​
1935.43​
5.5​
37​
Russia
17​
0​
0​
0​
0​
0​
0​
1904.76​
5.18​
38​
Byzantium
19​
0​
0​
0​
0​
0​
0​
1881.58​
5.11​
39​
America
17​
0​
0​
0​
0​
0​
0​
1868.53​
4.82​
40​
The Zulus
15​
0​
0​
0​
0​
0​
0​
1864.27​
5​
41​
Sweden
19​
0​
0​
0​
0​
0​
0​
1768.89​
5.37​
42​
The Huns
22​
0​
0​
0​
0​
0​
0​
1647.82​
5.18​
43​
France
13​
0​
0​
0​
0​
0​
0​
1494.77​
6.08​


Settings used (everything below no trade events is unchecked):
Spoiler settings :
1662581370039.png


If there's any interest I can clean up and github the automation code, it's nothing special and pretty bare-bones but will still resume from autosaves on recoverable crashes and save minidump/save files on CTDs before starting the next one.
 
Last edited:

balparmak

Prince
Joined
Sep 20, 2015
Messages
541
I'm loving these, great work! Also pretty much confirms that pure domination civs are the worst for the AI. Their notoriety comes from being an annoyance to the human player, and that's pretty much the only thing they're good for. The AI needs to become a lot better at capital sniping
 
Last edited:

bahamut19

Chieftain
Joined
Jun 15, 2020
Messages
75
It is interesting that the Iroquois would have the highest average score rank, yet only have 2 victories via Diplomacy.
 

pineappledan

Deity
Joined
Aug 9, 2017
Messages
7,829
Location
Alberta, Canada
These games were played on standard size and speed, with the default number of 8 major civs and 16 city-states?
I'm assuming that's the case, but it's important info to say outright, because I don't have a basis to say what a proper baseline win rate is unless you tell me how many players there were.
eg. 8 players where it's anyone's game should mean that all civs would be at a 12.5% win rate, but that number goes up or down based on the number of civs in a game.

Wonderful chart; thanks for sharing!
 
Last edited:

Stalker0

Baller Magnus
Joined
Dec 31, 2005
Messages
9,385
Merry Xmas! Now that's what we have been missing. This is very solid!

Now we just need to run it longer. 80 games isn't bad, but a couple of hundred would be a lot more solid, especially with the civ stats. Only seeing civ stats over 16 games isn't too solid, but if we can double or triple that number it should give us a great baseline to look at balance.

We can also use to see what government types the AI is picking, as I know there is always concern that one type of policy is chosen too much, so we should be able to see if any one policy tree is being taken far more often than the others.
 

L. Vern

Chieftain
Joined
Sep 5, 2022
Messages
29
Location
Ontario, Canada
These games were played on standard size and speed, with the default number of 8 major civs and 16 city-states?
I'm assuming that's the case, but it's important info to say outright, because I don't have a basis to say what a proper baseline win rate is unless you tell me how many players there were.
eg. 8 players where it's anyone's game should mean that all civs would be at a 12.5% win rate, but that number goes up or down based on the number of civs in a game.

Wonderful chart; thanks for sharing!
Good point, I have attached the settings (everything below no trade events is unchecked)
Spoiler settings :
1662581215913.png


You are correct that the expected outcome is 12.5% winrate per civ
 

KlHannibal2

Warlord
Joined
Aug 12, 2021
Messages
202
Nice work! I ran a few AI only games before (with less automation) and I was able to see an AI get close to domination victory a single time.

Of course it was Songhai, on a huge Pangea map. The game crashed when he had about 70% of the map. He had an inaane amount of units that just steamrolled everything and he got big by taking over the first to AIs using his river bonus.

A very rare exception.
 

L. Vern

Chieftain
Joined
Sep 5, 2022
Messages
29
Location
Ontario, Canada
Time, is that top score at max turn limit?
The scores displayed are all recorded at the moment someone won that game

Nice work! I ran a few AI only games before (with less automation) and I was able to see an AI get close to domination victory a single time.

Of course it was Songhai, on a huge Pangea map. The game crashed when he had about 70% of the map. He had an inaane amount of units that just steamrolled everything and he got big by taking over the first to AIs using his river bonus.

A very rare exception.
Yea, I think a big reason for the lack of domination victories was the communitas maps has all the civs on 2+ landmasses, and the AI isn't always the best at amphibious attacks. I would expect on a Pangaea map to see at least a few domination wins.
 

nekokon

Prince
Joined
Aug 9, 2017
Messages
399
I think the main reason for lack of Domination Victory is most likely because the AI just go full genocide even after taking enemy's capital, thus racking up a lot more warmonger penalty than needed, thus to have a Domination Victory likely they would have to complete painted like >80% of the world to win (thus estimatedly perform 4 times better than the rest of the world combined in score), while other victory only requires you to be about x1.5 times or less score than the next runner up.

A better fix would be changing how Domination Victory is decided, like controlling more than half or 1/3 the world's cities/population after year XXXX instead. That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.
 

4CV

Prince
Joined
Jan 9, 2017
Messages
303
Location
Germany
....That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.
And again, we'd see another attempt to force the player to play in a certain way. Why should the player not sit and wait for the rocket to start? Everone who wants to fight will do it anyway...
 

pineappledan

Deity
Joined
Aug 9, 2017
Messages
7,829
Location
Alberta, Canada
For reference, this is the results of 93 games.

Here is a list of my own observations from this data:
Spoiler :

  • Militaristic civs are the ones that players have the hardest time dealing with and are effective in the player hands. These stats indicate to me that these AI-only games don't get us closer to leader balance, but they might still help for things like policy or victory balance.
  • That having been said, if I were still to try to use this data to balance civs:
    • CV-oriented civs are top of the current meta, to no one's surprise.
    • Arabia in particular is exceptionally well-focused into a CV monster. He is not only winning 44% of his games, he appears to be winning so quickly and easily that he tanks his own victory scores by winning games too fast.
    • China doesn't have much overt CV-orientation, but she is the civ that is both consistently winning (2nd highest; 40%) and has consistently high scores (2nd highest; 2590), indicating that China is
    • On the other hand, France is the worst on every metric. No wins, lowest average rank (6th), lowest score; the only victory score that is below 1500, in fact. This aligns with my own personal experience of his kit being bad, and the AI being bad at using it.
    • While CV civs are top dog right now, it's specifically civs with tall bonuses, science bonuses that can be recruited for a CV, or civs with GW bonuses. wide CV civs that have :c5culture:culture on tile bonuses that can translate into :tourism: tourism are being left behind. Polynesia -- the purest wide CV civ -- is tied for 21st, and France is 43rd. The exception is Brazil, but that's because people are playing him tall right now, his kit is ridiculously overtuned for GA/WLTKD bonuses, and so he doesn't even rely on his Brazilwood UI to win. This suggests to me that while CV needs to be made harder, its should be made harder in ways that won't also nerf things like the :c5culture:/:tourism: tile converters.

It is interesting that the Iroquois would have the highest average score rank, yet only have 2 victories via Diplomacy.
Spoiler graph :
1662602058096.png

winning and score is only weakly related (r^2 = 0.16)
Iroquois have an average end score that is 22% higher than the next highest, China, but are tied for 18th place for win rate
Meanwhile, Arabia is 32nd overall for end score, but #1 for win rate

Average win score is 2118
A better fix would be changing how Domination Victory is decided, like controlling more than half or 1/3 the world's cities/population after year XXXX instead. That way AI would have an easier time to win through warfare, and for player who want to stop a Domination Victory they would have to be a lot more active in weakening the top contestant, instead of just ignore them to focus on defense and stall for time for other passive victory type like culture or science.
This is similar to what I would want a religious victory. You have to convert X% of the world population to your religion to win, and that can include by the sword, but it's pop-based rather than city-based.
 

nekokon

Prince
Joined
Aug 9, 2017
Messages
399
And again, we'd see another attempt to force the player to play in a certain way. Why should the player not sit and wait for the rocket to start? Everone who wants to fight will do it anyway...
It's for balance reason, to give DV civ a better chance of actually winning. If you don't like a certain way the game "forces" you to play just change the setting and don't use DV at all, since it doesn't actually do anything right now regardless.

You complaining about being "forced" to play more aggressive to prevent DV would be just like someone else complaining about being "forced" to play peacefully with strong CV civ like the current meta, and everyone who want to play peacefully... wait that already 90% of the AIs and majority of players because that's exactly what a meta is.
 

CppMaster

Deity
Joined
Feb 13, 2018
Messages
2,031
Location
Poland
winning and score is only weakly related (r^2 = 0.16)
Yeah, but that shouldn't matter, because the better you play, the earlier you can win, so you will acumulate less score, but on the other hand, you probably would get more score per turn. Too bad we don't have data on average game length.
 

Legen

Emperor
Joined
Sep 13, 2015
Messages
1,014
CV-oriented civs are top of the current meta, to no one's surprise.
Diplomatic civs are also doing great, just check Siam and Austria position. Greece and Germany are also above average as well.

And curiously, Venice is in the middle of the pack, at 0.12 win rate and 22th position.
 

4CV

Prince
Joined
Jan 9, 2017
Messages
303
Location
Germany
It's for balance reason, to give DV civ a better chance of actually winning. If you don't like a certain way the game "forces" you to play just change the setting and don't use DV at all, since it doesn't actually do anything right now regardless.

You complaining about being "forced" to play more aggressive to prevent DV would be just like someone else complaining about being "forced" to play peacefully with strong CV civ like the current meta, and everyone who want to play peacefully... wait that already 90% of the AIs and majority of players because that's exactly what a meta is.
Well, if you consider unwelcome questions being complaints we better not discuss them anymore. But that does not make the question being answered.
 

nekokon

Prince
Joined
Aug 9, 2017
Messages
399
It's already answered, to give DV civ a better chance of actually winning. Right now there's almost 0% of it happening.
And I considered that a complain and not a question because "force" is a pretty strong word.
 
Top Bottom