Ranking AI civ4

[Keler]

Hello everyone,

I'm quite excited about finding out a forum about Civilization IV exist
I have no idea about forum rules so feel free to correct me about anything that I'm wrong. Thanks

I know there is a thread about this but I found that is 5 years old and somebody posted something about time to post in that thread.

The time to post in this thread was a few months ago guys...

So again, feel free to fix any error you see in my post.

Now to the topic, I recently wanted to rank 52 AI from the best to the worst and I want to discuss what do you think about my method of ranking and how accurate do you think the results are and again, just like 5 years ago, who do you think is the best/worst. Feel free to make your own rank, give advice about how to rank AIs more accuratelty etc. I love advices and any kind of comments.

I chose
-small donut map, standard center and toroidal
-deity difficulty
-aggressive AI
-no technology trading
-no barbarians

And an island I made in the middle of the sea to enable terra map ability.

victory conditions:
-domination
-space race
-culture

no diplomacy,no AP



this is full of extreme amount of resources I know, which I decided that way due to small size of the island to prevent galleys to arrive. In proportion, it can't be similar to the one in terra maps.

I decided to put 13 different leaders, which I made 4 games to put every leaders in the game randomly.
I repeated this 30 times so far, ending up making 120 games in this map. And I removed myself to an isolated island. I made +100 in diplomacy towards every other leaders so they never wanted to declare war on me. I removed all resources in 9 tiles around my initial spawn tile and made it covered with ice or sometimes desert.


(orange is Darius, Sumerian blue at north west is Elizabeth)

Then I just passed turns and made things happen. I wanted to formulate this which wasn't well accurate but anyways. I noted turn numbers of civilizations to get destroyed and the final points of survivors.
To give example:

Space Race victory-302turn

Joao II (5316pt)
Charlemagne (3376pt)
Cyrus (2969pt)
Ragnar (1615pt)
Churchill (1612pt)
Bismarck (1537pt)
Darius I (241turn)
Tokugawa (214turn)
Brennus (189turn)
Alexander (179turn)
Saladin (172turn)
Elizabeth (132turn)
Napoleon (100turn)

There I gave destroyed leaders %1 point of the turns they survived
2.41 points to Darius
1 point to Napoleon etc.

For survivors, I divided their final points to game lenght so
for Joao II, 5316points/302turn=17,60 points.

so in this formula, a survivor with final point less than 3.02x302=912pt may get less point than somebody got destroyed late which rarely happened.

so this is my method and now the results of 120 games I played are:

1-Bismarck---------219,35
2-Pacal II-----------213,08
3-Qin Shi Huang----210,46
4-Boudica-----------210,15
5-Joao II------------201,64
6-Zara Yaqob-------196,88
7-Suleiman---------185,68
8-Justinian I--------182,55
9-Asoka-------------180,56
10-Saladin------------180,3
11-Charlemagne------179,97
12-Kublai Khan -------172,09
13-Cyrus--------------169,27
14-Catherina----------162,7
15-Hannibal-----------161,3
16-Gilgamesh---------156,55
17-Julius Caesar------156,37
18-Churchill-----------155,05
19-Mao Zedong-------153,76
20-Ramesses II-------153,73
21-Brennus------------152,22
22-Huayna Capac-----149,12
23-Ragnar-------------149,04
24-Willem van Oranj-138,25
25-Suryavarman II---135,95
26-Augustus Caesar---133,3
27-Alexander----------123,48
28-Mansa Musa--------123,31
29-Roosevelt-----------122,4
30-Wang Kon-----------121,01
31-Louis XIV------------117,14
32-Washington----------114,65
33-Isabella--------------111,77
34-Mehmed II-----------111,48
35-De Gaulle------------110,68
36-Hatshepsut-----------110,06
37-Stalin-----------------109,99
38-Tokugawa------------109,26
39-Sitting Bull-----------108,66
40-Peter------------------106,94
41-Pericles---------------106,9
42-Victoria---------------102,68
43-Darius I--------------101,88
44-Genghis Khan--------98,97
45-Shaka----------------98,87
46-Frederick-------------97,97
47-Gandhi----------------95,48
48-Hammurabi----------93,41
49-Montezuma-----------92,2
50-Lincoln----------------81,92
51-Elizabeth--------------76,31
52-Napoleon--------------67,65

Now, this is my ranking. And sometimes in some games, not the leader with the highest score wins. For that reason, I have one more data for you

Number of wins

Pacal II 7
Asoka 6
Bismarck 5
Mansa Musa 5
Willem van Oranje 5
Zara Yaqob 5
Boudica 4
Brennus 4
Hatshepsut 4
Joao II 4
Justinian I 4
Kublai Khan 4
Qin Shi Huang 4
Ragnar 4
Cyrus 3
Gilgamesh 3
Hannibal 3
Huayna Capac 3
Pericles 3
Ramesses II 3
Saladin 3
Suryavarman II 3
Wang Kon 3
Alexander 2
Churchill 2
Isabella 2
Louis XIV 2
Mao Zedong 2
Suleiman 2
Washington 2
Augustus Caesar 1
Catherina 1
Charlemagne 1
Elizabeth 1
Frederick 1
Genghis Khan 1
Julius Caesar 1
Montezuma 1
Peter 1
Shaka 1
Sitting Bull 1
Stalin 1
Tokugawa 1
Victoria 1
Darius I 0
De Gaulle 0
Gandhi 0
Hammurabi 0
Lincoln 0
Mehmed II 0
Napoleon 0
Roosevelt 0
+_____
120

Pacal II wins 7 gams out of 30 while there are still some leaders who failed to win at least one.
So I hope this is something interesting for some poeple and this post is appropriate for this forum.
I have an excel file for showing final standings for every game which I couldn't attach here.

Regards
Thanks for reading

 

[Lennier]

Interesting way of testing. One thing: Surrounding your "isolated island" with mountains may be a more effective way of isolating yourself as the AIs battle each other.

 

[Kid R]

Hey - nice post!

The method of running many iterations to average out the RNG is definitely the way to go if you've found a way to do that quickly and easily. And I think your method to allocate a score at the end of games is reasonable.

But to get the "true" answer it will have to be on a realistic map or maps, and mixing up the opponents each time.

Using the same map each time would be ok I guess but it would have to be a common one like a fractal or something, and even then the results would have to be qualified by the fact that different AIs have advantages on different map types (eg financials on very watery maps). If not a random map for each and every game then maybe a small roster of a carefully chosen "typical" maps, 1 from the commonly-played 4 or 5 types.

Re. Starting locations clearly if Gandhi is next to Shaka every time he will not get very far! But previous attempts to solve this puzzle have shown that Gandhi if allowed to live by the other AIs can and does regularly top the leaderboard and has no worries about nuking anyone if it comes to it I've considered trying something like you did but never had the energy (or probably SDK knowledge ) to write a tournament "driver" shell which would rotate starts in a fair way, but I do think that's what would be needed.

Deity difficulty, well I guess it shouldn't make a difference but it might depending on the different AI preferences and the way they have scaled various costs etc. for difficulty. Also higher difficulties perhaps favour warmongers simply because the land fills faster, generating conflict. Using standard speed I guess has to be right, but as is well known, slower speeds favour war (hence warmonger AIs) both for tempo reasons (Units obsoleting, although that's less of a problem for AI than for human, and map movement) but more specifically for some glitches in the way the game scales costs for speed.

Personally I would add a bonus to the ranking for AIs who actually win games, to reflect your last list, but then that'd be another subjective opinion - I.e some would feel an AI who never wins but comes second a lot to be "better" than one who occasionally wins but otherwise gets knocked out early (bit like the way I play )

Anyway hope this wasn't too critical a reply! Your method has definitely picked out many of the leaders that other experiments have done, and that our own experience of playing against them would suggest. If put on the spot I would agree that Pacal may well be the strongest. 2 great traits and seems to have good adaptable personality settings. Of your top 20 Mao and Bismarck are the only ones I am raising eyebrows at...!

 

[Seraiel]

Bismarck has neutral peaceweight, so he doesn't get hated easily, and he builds a lot of military because of his flavour. Mao I don't understand either, but every statistical test has a glitch.

 

[Mizar]

Doesn't reflect what I see in my games, for example Willem tends to do much better then many that are listed above him.

As general rule in the unmodded game, leader with passive bonues for quicker tech (financial, organized) and leaders with good expansion traits (expansive, creative, imp) tend to do better then the rest.
Another issue is the AI-AI attitude pre-modifier, some just attract so many foes they usually get crippled.

 

[Kjotleik]

Hello, Keler!

Very interesting test you made there...

Thanks for posting your results!


Yours Sincerely
Kjotleik of Norway

 

[Keler]

Hello again,

Thanks for replies

I actually surrounded my isolated island with ice, so I think paratroopers can't land on a warrior And my civilization occupies less land percentage I guess? I'm not sure if peaks have same properties as ice.

I am also considering trying a different method. Originally I wanted to make 52x4=208 games.
120 so far What kind of a map should I try? I am not a fan of fractal, I like terra maps because I subjectively think galleons must be used in the game. But again, the spawn distribution is quite unfair, and there is always someone landlocked. In any way, there will always be glitch, so perhaps different methods may show coinciding results.

I generally play in Noble, I think why you find Bismarck a bit unrealistic or surprising is that when technology trading is enabled he is somewhat less advanced, just like Tokugawa or Sitting Bull or... Mehmed! He can get along with peaceful leaders and if he can find someone to kick, he expands crazily. I was surprised too. He generally fails if he gets absorbed by creative&cultural leaders. But I don't have well information about leader traits but just some personal observations.

Forgot to mention game speed which is standard.

I chose deity diffculty to games can end faster. One game with 13 AIs took around 25-30 min for me, so not that time consuming. But now on, I can delete all the bonus units and put one settler/one warrior or scout in their spawning spot, in deity difficulty

About Gandhi, he expands well, gets high cultured cities which should boost his production speed but has little concern but I witnessed him finishing on top too.
And about Willem van Oranje, I feel like he has similar problems with Mansa Musa but he is more successful at expanding. I have no real information but again experience on AIs attraction to each other. Willem tends to hate Lincoln a lot weirdly...?
And all of a sudden lots of other AIs declare war on him just like they do to Mansa Musa so he ends up getting wiped out. They both have 5 victories despite finishing low on scoreboard is because they won culture victories mostly so far.

Thanks for reading.
Any advice or correction is welcomed.

Regards

 

[elitetroops]

Interesting results... Most surprised to see Napoleon at the bottom, he's usually quite strong.

If you're gonna do more of these, I believe there is some way to have the game play out automatically, without having to sit there and press end turn. Not sure how it's done, but I think Kjotleik did it in his AI AutoPlay series, which you can find in strategy&tips.

About the settings, why no tech trading? This makes it somewhat artificial and changes the dynamics a lot. And it means no bribing into war, which probably favors the more peaceful leaders. Also, why no diplo/religious victories?

 

[drewisfat]

I also was confused about the Orange, he also ranks at the top for me for AIs. When he's in the game he's often the boss fight for me. Gandhi also is a known threat for culture, and I'm confused about Pacal being 2nd.

 

[Pangaea]

Interesting results, and they conform to similar tests by other people. Not too surprised Bismarck does well; if he is given time to expand and build a military, he becomes strong. Gandhi has done very poor here, but he has done well in other tests, so that's a little odd.

Zara almost always does well in my games, so not surprised to see him up there.

Should be noted, however, that the strength of the AIs can be very different in human hands, where typically good early UUs is important. Elizabeth and Darius for instance are strong in human hands, yet here they rank near the bottom.

 

[Keler]

Forgot to mention some parts of settings.
-no vassal state
-no random event

I didn't want some other civilizations to hide in to other.But I'm not sure if I should keep it disabled.
No Diplomatic victory because I love Justinian
no seriously, I find diplomacy nothing but luck for some reason. It is not fair when AP picks on some leaders and destroy them. But I may decide to enable that too. Should I?
I turned off technology trading because I wanted to see how much research leaders focus on. I might as well enable this too.

Also for maps again, I'm considering shuffle cylindrical maps. Small or standard size with 13 AIs again. Any suggestion with map option?

 

[Kjotleik]

If you're gonna do more of these, I believe there is some way to have the game play out automatically, without having to sit there and press end turn. Not sure how it's done, but I think Kjotleik did it in his AI AutoPlay series, which you can find in strategy&tips.

Yes, I am using the Better BUG AI mod, because that includes the AI Auto-Play function. It is just to set up the game like you would like it to be, and then press [CTRL + SHIFT + X] and set the number of turns you would like the game to play on its own.

After the game is done, you can load any auto-saves that you made (you DO have the game set to auto-save EVERY turn, right?). Off course (unfortunately), when the "player" civilization is defeated, loading any auto-saves later than your demise, will just continue the game unless you choose to change civ by pressing [ALT + Z] upon loading it... a minor inconvenience...

By the way, my AI Auto-Play series is not done. It's just that I have realized I'm not able to do a detailed write-up about every game any longer. Also I am not able to continue the C-part of the competition.

But it will return in September, for the remainder of season one. I guess I will just record the game, and upload the "World History" part at the end to YouTube, so that any interested parties can see for themselves how "their" chosen winner got killed off


Yours Sincerely
Kjotleik of Norway

 

[Lone Wolf]

The largest surprise is Napoleon, in my experience he is definitely not the worst.

Balanced AI's like Bismark, Justinian or Zara being near the top is no surprise.

 

[Keler]

Gandhi has done very poor here, but he has done well in other tests, so that's a little odd.

Which other tests? I can't find any other attempts about ranking AI. I'd really like to check them

In my balanced donut maps, Gandhi time to time performed well but he lacked millitary a lot and ended up feeding enemy even after turn 250.

Currently I just enabled technology trading in my settings and I will keep doing some more until I reach making 52x4=208 games for now.

 

[Pangaea]

Don't have the links, but Kjotleik have run similar games to this, and I think there have been other(s) too. Seem to recall Gandhi doing surprisingly well, winning Culture for instance.

 

[Kjotleik]

Which other tests? I can't find any other attempts about ranking AI. I'd really like to check them

If you want to find my series (AI AutoPlay) you can go into the Strategy & Tips sub-forum and:

Set -
Sort by: Thread Title
Sort order: Ascending
From the: Last Year
Prefix: [BTS]

Click the "Show Threads" button, and my threads should be on page one

I'm only about halfway through my 16-game series, though. I'll be back with the next game in September.


EDIT:
Also, after some searching, I found Kossin's attempt:
AI tournament, Season Three by Kossin

EDIT II:
And who am I to forget Sullla's masterpieces... shame on me...
Sullla's Civilization IV AI Survivor Games



Yours Sincerely
Kjotleik of Norway

 

[elitetroops]

If you want to find my series (AI AutoPlay) you can go into the Strategy & Tips sub-forum and:

...

or click Kjotleik's name on the post above and select "find all threads started by".

 

[Keler]

This picture is about how many victories leaders earned so far and which type it is.

Total Domination victories: 59
Total Space Race victories: 52
Total Culture victories: 33

Final results of 36x4 games. I think I'm not going to do more of 52 civilization 4 IV AI Ranking League. I really enjoyed Kjotleik's settings and tournement style. And Garath Sulla's AI Survivor too. (apart from groups of 6-7 and pangea map which I believe it needs some oversea islands for astro lovers.
Anyways I'll tell about what I'll try as a tournement later.

Pacal II 263,04
Boudica 262,77
Justinian I 259,95
Bismarck 247,78
Joao II 246,64
Qin Shi Huang 235,63
Catherina 232,93
Zara Yaqob 230,33
Asoka 219,84
Saladin 216,86
Suleiman 215,73
Charlemagne 210,38
Cyrus 209,21
Kublai Khan 208,09
Brennus 201,18
Hannibal 195,44
Mao Zedong 193,83
Churchill 190,5
Julius Caesar 189,83
Peter 184,89
Huayna Capac 180,12
Willem van Oranje 175,02
Gilgamesh 171,21
Ramesses II 168,41
Ragnar 166,46
Louis XIV 160,82
Augustus Caesar 157,11
Suryavarman II 155,45
Roosevelt 149,9
Wang Kon 146,47
Washington 144,97
Alexander 144,77
Mansa Musa 143,96
Victoria 141,36
De Gaulle 140,06
Darius I 137,14
Mehmed II 134,92
Pericles 134,21
Isabella 133,86
Hatshepsut 129,18
Sitting Bull 127,58
Shaka 123,35
Tokugawa 120,56
Stalin 120,22
Frederick 112,12
Genghis Khan 111,94
Hammurabi 110,52
Montezuma 109,49
Gandhi 108,87
Napoleon 106,19
Lincoln 103,15
Elizabeth 98,65


Once again, number of victories out of 36

Pacal II 8
Justinian I 7
Willem van Oranje 7
Asoka 6
Zara Yaqob 6
Brennus 6
Bismarck 5
Mansa Musa 5
Boudica 5
Joao II 5
Hatshepsut 4
Kublai Khan 4
Qin Shi Huang 4
Ragnar 4
Pericles 4
Saladin 4
Peter 4
Cyrus 3
Gilgamesh 3
Hannibal 3
Huayna Capac 3
Ramesses II 3
Suryavarman II 3
Wang Kon 3
Louis XIV 3
Catherina 3
Victoria 3
Alexander 2
Churchill 2
Isabella 2
Mao Zedong 2
Suleiman 2
Washington 2
Augustus Caesar 1
Charlemagne 1
Elizabeth 1
Frederick 1
Genghis Khan 1
Julius Caesar 1
Montezuma 1
Shaka 1
Sitting Bull 1
Stalin 1
Tokugawa 1
De Gaulle 1
Darius I 1
Napoleon 1
Gandhi 0
Hammurabi 0
Lincoln 0
Mehmed II 0
Roosevelt 0



Worst perfomance : Lincoln, 0.73 point

Space Race victory-326turn

Pacal II (6127pt)
Mao Zedong (4999pt)
Boudica (3943pt)
Alexander (298turn)
Suleiman (256turn)
Shaka (233turn)
Montezuma (229turn)
Zara Yaqop (223turn)
Mansa Musa (207turn)
Tokugawa (195turn)
De Gaulle (172turn)
Brennus (159turn)
Lincoln (73turn)

Best performance:Catherina, 25.01 point
Also shortest game:271turn

Domination victory-271turn

Catherina (6779pt)
Peter (4200pt)
Louis XIV (2188pt)
Huayna Capac (266turn)
Gilgamesh (264turn)
Qin Shi Huang (251turn)
Joao II (229turn)
Willem van Oranje (212turn)
Tokugawa (209turn)
Mansa Musa (189turn)
Genghis Khan (174turn)
Isabella (164turn)
Ramesses II (136turn)

Longest game:405turn

Culture victory-405turn

Alexander (5668pt)
Kublai Khan (5278pt)
Peter (392turn)
Darius I (358turn)
Hammurabi (314turn)
Sitting Bull (307turn)
Pericles (258turn)
Willem van Oranje (206turn)
Mehmed II (180turn)
Montezuma (165turn)
Elizabeth (162turn)
Roosevelt (155turn)
Napoleon (154turn)

yes, creative Kublai didn't get the culture victory.

Victory with lowest score : Mansa Musa

Culture victory-328turn

Mansa Musa (1896pt)
Justinian I (4477pt)
Roosevelt (3034pt)
Frederick (2828pt)
Lincoln (2767pt)
Saladin (308turn)
Ramesses II (306turn)
Augustus Caesar (273turn)
Sitting Bull (248turn)
Pericles (231turn)
Napoleon (164turn)
Darius I (155turn)
Kublai Khan (142turn)

Victory with highest score: Zara Yaqob

Space Race victory-372turn

Zara Yaqop (7342pt)
Charlemagne (2702pt)
Catherina (2404pt)
Justinian I (2345pt)
Isabella (329turn)
Shaka (329turn)
Huayna Capac (298turn)
Napoleon (267turn)
Lincoln (212turn)
Stalin (196turn)
Ragnar (167turn)
Elizabeth (144turn)
Victoria (139turn)

It should get more accurate when I make more of these but averagely a victory brings 15-20 points which keeps pushing middle leaders to the 5-6 row higher. So I guess the point difference shouldn't matter so much. For example, Mansa Musa can be considered better than Roosevelt.

I didn't want to post every game which there are 144 so far. I don't think it'd be interesting so, now on, I'll decide on some different settings and make tournement like ranking. I'll post about it later to this thread. Still, if you want to advice about any setting for unmodded AI games feel free to say. I'm not really aiming for story telling but still, I think I should post about every game with pictures showing starting places and how it ended.

 

[Folket]

I find this interesting. But I would prefer to see games with tech trading and without aggressive AI. Since that is how most people play. Also a lot of personality difference between leaders are removed with these settings.

If you do with tech trading you should probably put your settler on a 1 tile island isolated by ice 3 tiles wide so that you do not influence the AIs tech trades. Or perhaps just play with the AIplay mod.

 

[Keler]

So I came up with this way of ranking, which will let me sort the leaders that went to top 8 and top 20.

I decided on these

Settings
Map: Hemispheres
Size: Small
Climate: Temperate
Sealevel: Medium
Era: Ancient
Speed: Normal
Continents size: Massive
Island size: Islands?? (probably mediumif the other one is tiny)
Number of continents: 2

Options

[x]No Barbarians
[x]Aggressive AI
[x]No Vassal States
[x]No Tribal Villages
[x]No Random Events

means yes technology trading!

Victories

All of them,but 4 types actually

conquest will be useless
since it is deity AI difficulty time will be useless
so diplomacy,space race,cultural and domination will matter


please note that my civ mide is 301?
and my build version is 3.0.0.93 hmmpf, was built in sometime in 2007 and I need 3.19 update for AI autoplay?
In the first free time I will check them. Can appreciate something like step by step installation guide if I can't find my way.

So, my previous league method results were in that build.

And to prevent "unfairness", I delete some tiles when an AI start in a peninsula or all tiles if it is a seperated island



And (not in the photo) I make 4 oversea islands with around 10-15 tiles. North and south of both oceans.

I gave it a try, and here is one single tournement result for you

Spoiler :

let me know what'd be interesting and more accurate before going this way, and then I can arrange my settings and hopefully figure that patch and then autoplay and start putting pictures of the games as well

Regards,

Keler


EDIT: If you are interested with this you may as well check my this thread