The AI Horatius League

[Thrasybulos]

Spoiler Current Leaderboard :

After four tournaments:

Welcome to a new Civ 4 AI bloodfest : the Horatius League !

As you may or may not be aware, I've endeavoured to try and find a way to rank the Civ 4 AIs in an AI Survivor-like context, using the Elo rating method.
And one major issue is that the Elo rating is designed for 1 vs 1 games, which isn't the case in AI Survivor.
Sure, I could just run 1 vs 1 games on a tiny map, but where would be the fun in that?
But... it recently came to my notice (after all these years, there are still things to discover about the game) that contrary to what I'd always simply assumed, there is nothing preventing running a game with duplicate leaders!

Pitting 3 instances of one leader versus 3 instances of another leader, I could have my cake and eat it: 6-player games which are also 1 vs 1 for ranking purposes!

Now of course, this is a tad goofy and won't reflect actual AI Survivor performance (if only for peaceweight distribution reasons).
But this is fun-goofy, so I went ahead!

Game Settings

I'll be using my now customary settings:

• AI Survivor S5+ rules: Deity level, no bonus techs, no AP, no huts, no events, no tech trading, no vassals.
• Adding to that: no UN, no barbs.

Victory conditions are thus limited to Spaceship, Domination, Culture (and Time in theory).
And there's going to be a new Conquest victory type: when all three opposing leaders are eliminated. In a game pitting 3 Shakas vs 3 Gandhis, if all 3 Gandhis are dead, I don't care which particular Shaka wins: it's a Shaka win.

Note: These are not team games. Fratricide wars are a very real possibility.


Tournament Format

Spoiler :

• The first phase of the tournament will be a Pool phase : 8 pools of 6 or 7 leaders.
  • Each pool will be run in a round-robin format: so 15 match-ups for the 6-player pools, and 21 match-ups for the 7-player pools.
  • Players will be ranked by total wins, then wins vs tied players, then by average win date.
  • The top 2 of each pool will move to the next phase.
  • For the 1st tournament, the leaders will be assigned completely randomly to a pool. For subsequent tournaments, past performance will allow pool seeding.
• Then we'll proceed to a 4-round knockout phase which will determine the winner.
  • In that phase, the tie-breaker, if needed, will be one extra game played on the AI Survivor S4-S7 mirrored map.
• Maps and "sets":
  • Each pool and each knockout matchup will be played on a different AI Survivor map.
  • The games will be played in "sets" of two games: a random starting position permutation will be drawn for each set, and the games will be mirrored accordingly.
    For instance, if Player1 gets assigned the 1,5, and 6 starting positions (and thus Player2 gets 2,3,4), the second game of the set will have Player1 starting from positions 2,3, and 4.
  • In the pool phase, each matchup will be played over a single set (2 games). That number will increase in the knockout phase, reaching 4 sets (8 games) for the Finals.

Note:
This isn't my "AI League 2.0": it's a different, side project to keep busy and which allows me to start working on the tools I'll need (all games results are saved to a database now for instance).

I'm actually done running the first tournament... so let's see how that went.

 

[Thrasybulos]

Tournament 1, Pool Phase

Spoiler Pool 1 :

First surprise: HC barely makes it!
Now, it might be a fluke, as he got hammered by Monty who, in turn, got hammered by just everyone else.

Spoiler Pool 2 :

A genuise surprise here: Gilgamesh doesn't make it!
Lincoln and Roosevelt, on the other hand, end up where we'd expect them: at the bottom.

Spoiler Pool 3 :

Ramesses:
I guess we have a tournament favourite now!

And Hatty beats Qin to the second place, making it an Egypt clean sweep.
Saladin disappoints.

Spoiler Pool 4 :

Freddy:

Spoiler Pool 5 :

I thought this setup would be the occasion for the economy-focused high peaceweights to shine.
Well, Viccy disappoints in a major way here, and while Gandhi does better, that's not good enough.

Spoiler Pool 6 :

Hannibal and Louis falter, but what could they do against top tier leaders like De Gaulle and Churchill?
Oh, wait...

Spoiler Pool 7 :

Justinian and Pacal do the job, with Willem right behind.

As for Darius...

Spoiler Pool 8 :

Mansa has to be a favourite here... but he got Shaka'd hard.

Suleiman and Cyrus doing so bad is a real shocker, though.

 

[Thrasybulos]

Tournament 1, Knockout phase round 1 (aka Pre-finals, aka Round of 16)

Spoiler Genghis vs Julius :

That one went to the tie-breaker.
Khan wins.

Spoiler Napoleon vs Churchill :

The crazies are having a field day it would seem!

Spoiler Ramesses vs Pacal :

And there goes the favourite out of the pools!
It was a close thing, though, even in the tie-breaker: Pacal's ship landed right before his civ got wiped out back on Earth.

Spoiler Zara vs Mansa :

That one wasn't close. Clinical Mansa.

Spoiler Kublai vs Huyna Capac :

HC getting his stride back?

Spoiler De Gaulle vs Pericles :

Ouchie. Toldya De Gaulle was a real star!

Spoiler Justinian vs Hatshepsut :

Sorry, Hatty.

Spoiler Shaka vs Ragnar :

Ragnar resorting to a Culture win was a bad a sign for him as you'd expect.
He fought well, but Shaka smash!

 

[Thrasybulos]

Tournament 1, Quarterfinals

Spoiler Genghis vs Napoleon :

Another shock of the crazies. In that situation, what khan Napoleon do?

Spoiler Pacal vs Mansa :

A close match, except for the tie-breaker when Mansa crushes Pacal.

Spoiler Huyna Capac vs De Gaulle :

Trust HC to put an end to a good joke.

Spoiler Justinian vs Shaka :

... or Justinian, for that matter.

 

[Thrasybulos]

Tournament 1, Semifinals and Finals

Spoiler Genghis vs Mansa :

Well, if you thought the Khan had simply got lucky, think again?
A very close call for Mansa, but again, the tiebreaker was a one-way affair.

Spoiler Huyna Capac vs Justinian :

That could have been the finals, and while Civ 4 is a game played with 52 players where the Incas usually win in the end... sometimes the Byzantines do, instead.

Spoiler Finals :

And if you thought from the start that Mansa was the heavy favourite... you were apparently correct.

Spoiler Final Bracket :

 

[Genghis the Barbarian]

Some of questions/comments

Spoiler :

1) I think the problem with this (very clever) format in general is it super nerfs culture leaders like Louis, Ram, and Gandhi, because they pretty much hurt each other in a not good way by taking religions away, stealing wonders, etc. This is especially so for high PW culturemongers, because they are very unlikely to conquer each other and thus not have enough religions and wonders for a viable culture play

2) Mansa is the exception to the above because he’s Mansa, and three Mansas is essentially an unstoppable T270 ticking time bomb. Also, Mansa’s biggest weakness IMHO is his culture, I truly believe he’d be much better if he just went pure econ and that this belief is confirmed by AHs. Other Mansas competing for religions actually works in his favor.

3) Conversely this format greatly favors crazy warmongers (hence GK, Shaka, Monty being pleasant surprises) because there are two scenarios:

A - 2-3 Shakas tag team an unlucky rival, grows big, boom conquest

B - Civil War - eventually there is one ginormous Shaka and the three little Freddles (this isn’t FNAF! XD) or Hatties don’t stand a chance

Both are good for the warmonger.

4) I think you should turn on the UN (to balance for high PWs a bit more) and turn on barbs (because being able to handle the barbs is still an important aspect of playing civ 4 and is thus needed to judge how good an AI is)

 

[Thrasybulos]

Thanks for the comments.

As far as what type of AI personnality this format favours... I'll reserve an opinion until more games have been played: I've started the second tournament, and it has already yielded some major surprises.

Keep in mind that this isn't an AH-like format: although many games are played, the "outlier" factor is very much in play. After all, in the Pool phases, each leader only has two games vs each other leader.
And beyond that, there's also the "luck of the draw" aspect for starting positions in the pool phase.
If a leader draws a 3 crappy spots vs 3 good spots game, that's gonna be a 1 - 1 result, while a competitor might have drawn a more balanced repartition and thus a chance at a 2 - 0 result.
(That's lead me to briefly consider using the same start repartition for all games in the same pool, but I ended up rejecting the notion).

I also thought that the culture-focused leaders would be negatively impacted, but that doesn't really seem to be the case: Gandhi didn't fare that well, but Hatty and Ramesses for instance did fine. So...

You may very well be correct regarding the warmongers, but again let's not draw hasty conclusions from a single sample set.
That said, if you are correct, then I wouldn't mind having the crazies shine for once, and it would also mean that if any AI Survivor game happened to be warmonger-heavy, they should be seriously considered for the win.
Doesn't happen too often, though: on top of my mind, I can only think of S4 G5, and the AH had the least warmonger (Kublai) as the favourite (although my jumbled AH experiment tended to show this was a fluke of the start draw: the favourite in this setup was whoever happened to be bordering both Bismark and Zara, and that was Kublai there).

As for enabling the UN and barbs: nah, they just add random noise to the results.
It's fun noise when watching AI Survivor, so they certainly have their place there, though.

 

[Thrasybulos]

Tournament 2, Pool Phase

Here we go again.
Thanks to the results of the first tournament, seeding was available, which yielded the following pool composition (first two leaders in each pool are seeded, the rest are randomly drawn):

Spoiler Pool 1 :

And this tournament gets started with a thunderous bang: our champion is out!
The other seeded leader, Pericles, does even worse and finishes last.
Hannibal and Saladin are two strong AI Survivor leaders, so their making it doesn't feel out of place, though, especially since they're the Mansa murderers here.

Spoiler Pool 2 :

The two seeded leaders make it here, but things still didn't quite go according to expectations: Ramesses barely makes it when he crushed the competition in the first tournament's pool phase. Instead, De Gaulle (!!) has a very strong run.
Gilgamesh, one the strongest AI Survivor leaders, fails again just short of the target.
Cyrus, another otherwise rather successful leader, confirms that he's just bad at this?

Spoiler Pool 3 :

And we lose another major favourite: Pacal !
Hatty feels untouchable, while Peter (!!) gets the second place.

Spoiler Pool 4 :

And... another one bites the dust: mighty HC is out! (that sound you're hearing is @Fippy laughing).
Kublai, the other seeded leader, is out too.
This felt like a very tough pool, though, as the two who do make it, Louis and Willem, don't seem out of place. And Gandhi was in there, too..

Spoiler Pool 5 :

And... Freddy is the first leader to draw a blank. Not unexpected ?
Justinian has a close call: Suleiman manages to snatch defeat in the jaws of victory in his first game against Boudica. Had he won that game, he would have taken the second place instead of the Byzantines.

Spoiler Pool 6 :

Somehow, the draw has bunched the crazy warmongers together in the last pools, so this was their first appearance on the scene.
... and this time, they failed to make an impression: Ragnar and Shaka are out (and so is Monty, but his first tournament wasn't a successful one either).
Qin had a weird set of results: he crushed (Ragnar, Joao, Monty, Shaka) or got crushed (Viccy, Mao).
That last set, against Mao, felt weird. So I replayed the last game, to make sure: and indeed, in that replay, Mao got wiped out.
Goes to prove that individually, those results are to be taken with a major grain of salt, as are AI Survivor's.

Spoiler Pool 7 :

Napoleon redeeems the warmongers by slaughtering everyone.
Lincoln, thanks to his Culture plan B, sneaks into second place in an otherwise weak field.

Spoiler Pool 8 :

In this tournament, Napoleon proves the exception: the warmongers fare rather poorly here as well, as we lose another semifinalist of the previous tournament when Genghis fails to make it.
Sitting Bull (!!!) and Churchill as the top two feels all kinds of wrong.

 

[Thrasybulos]

Tournament 2, Knockout phase round 1

So, only 16 remain now, and this might be the most dangerous round: with only 4 games, recovering from an unexpected loss is pretty hard.
Those matchups feel a bit like tennis games: playing from the strongest starts is like having the serve advantage, and winning from the weakest starts is managing to "break" the opponent.

(Note: no longer putting the leaders in the spoilers titles as that's actually... spoilerish. )

Spoiler Prefinals 1 :

This one felt like it could have been the finals, as Hannibal and Justinian are two very strong leaders indeed.

And it went the Byzantine way.
It was actually pretty close: in the last game, Hannibal got to the end of the tech tree first, by a significant margin.
But Justinian's spies made sure the Carthaginian spaceship never got completed.

Spoiler Prefinals 2 :

Ramesses apparently set that trend in the previous tournament? Having an extremely strong run in the pools phase means... getting eliminated in the very next round?

Spoiler Prefinals 3 :

Hint: unless you're called Manda or Gandhi (and even then...), the peaceful approach is not the correct approach when dealing with Hatty.

Spoiler Prefinals 4 :

A turn 416 Cultural victory is not what you'd expect from Louis, but that got the job done.

Spoiler Prefinals 5 :

Close, except for the tie-breaker which was a one-way affair.

Spoiler Prefinals 6 :

I didn't really know what to expect there, as both are strong leaders in their own right.
It went Qin's way this time, and once again Ramesses fails in the first knockout round. Unlucky or telling?

Spoiler Prefinals 7 :

Can't say I'm unhappy to see Peter out.

Spoiler Prefinals 8 :

Thank you, Willem.
I must confess that seeing Sitting Bull there was kinda embarrassing.

 

[Thrasybulos]

Tournament 2, Quarterfinals

Spoiler Quarterfinal 1 :

I didn't really expect Viccy to stand a chance, but she went down fighting.

Spoiler Quarterfinal 2 :

The first game was a hilarious reversal of roles: a warmongering Hatty saw Louis pull off a Cultural win right before she wiped him out.

Spoiler Quarterfinal 3 :

Spoiler Quarterfinal 4 :

Pen mightier than the sword?

 

[Thrasybulos]

Tournament 2, Semifinals and Finals

Spoiler Semifinals 1 :

This is the second time these two meet in the knockouts... and it goes the same way.
Justinian seems too high a step for Hatty.
She had a good run, though, making the final four. And her defeat isn't humiliating.

Spoiler Semifinals 2 :

Willem finally lives up to expectations?
It should be noted that Qin can sometimes "out-Willem" in his Rifling-eschewing ways.

So... who's gonna be the next Champion ?

Spoiler Finals :

It has to be Justinian's turn, right?

Wow. I certainly wasn't expecting this!
And it wasn't even close.

Habemus Campionem.

Spoiler Final Bracket :

 

[Thrasybulos]

Spoiler Leader ranking :

Seeded leaders:

Rank-and-file:

Performance in the tournaments is measured with the following scoring system:

• Reaching the knockout phase (ie finishing 1st or 2nd of a pool) : 1 point
• Reaching the quarterfinals : +2 points
• Reaching the semifinals: +4 points
• Reaching the finals: +8 points
• Winning the tournament: +16 points

Those scores are additive, so losing in the semis, for instance, means scoring 1 + 2 + 4 = 7 points.

So winning any stage is worth reaching it twice, plus 1 point.

Spoiler :

For instance, Justinian, who lost twice in the Finals, gets 30 points, while Mansa and Willem get 30 + 1 = 31 points.

The tiebreakers are the total career wins, then the average victory turn (lower is better).

 

[Thrasybulos]

Elo Ratings

So, getting there at last, as that was - almost - the whole point.

With the data from those first two tournaments (737 games), we get these ratings:

Spoiler :

Better than average:

Worse than average:

I fully expect those values to change as more results come in. In particular, the range of values should narrow quite a bit.

Spoiler Calculation method and explanations :

I've in fact decided to go for for a Bradley-Terry model instead of an Elo model.
Now, both are based on the same principle: if two players, P1 and P2 have respective skills valued S1 and S2, the odds of P1 winning are S1 / (S1 + S2).

Where they differ is in how the skill values ("ratings") are calculated.
In the Elo system, an expected outcome for a game is determined from the formula, and ratings get adjusted by a small amount if the actual result of the game differs from that expected outcome. So the idea is that after a certain number of games (and adjustments), the ratings will converge to their "true" values.
In the BT model, all the game results are considered at once, and the rating values (called "parameters" in that model) which best fit those results are computed.

The Elo system is a better fit for players whose skill evolves with time, as the order in which the games are played counts, and the most recent games are made more relevant.
But here... the Civ4 "AIs" are in fact just a bunch of static parameters. They don't "learn".
So a system where every game counts the same is actually a better fit, hence my choice.

In either system, the absolute values have no importance: what counts are their relative values. Ratings are always compared to other ratings.
So I've converted the BT values to an Elo scale, which is why I'm still calling them "Elo ratings".

The parameters I've used are:

• 1600 is the base (and thus average) rating.
• I've used a factor (scale) of 800 instead of the usual 400.
  That means a 800-point difference designates 10:1 odds here. See here for instance for an odds table (just double the indicated rating difference).
  I made that choice because I don't expect the range of "skill" between the AIs to be that big (no grandmaster vs amateur scale for sure), so I wanted to spread out the values a bit.

 

[Thrasybulos]

The draw for the next tournament has yielded the following pools:

Spoiler Tournament 3 Pools :

I've only just started running, so the results won't be in for some time (especially since I'll be away for a few days next week).

Any guesses how it'll turn out?

 

[Genghis the Barbarian]

I'll do some predictions (request, can you post one round at a time so we can keep doing predictions?)

Pool 1:
Some tough choices but I'll go with Hannibal and Cathy, Willem barely missing out because he Willems some games away.

Pool 2:
I'm on team French here. I think the French leaders in general work pretty well together. Nappy has a lot of good targets, and DG might actually be the best techer in this pool.

Pool 3:
I'd say Louis and Julius Caesar. This is not an inspiring group but those two are the only ones with clear win conditions.

Pool 4:
Shaka and Saladin. This is not Hatty's field, lots of military leaders who will punk her.

Pool 5:
Mansa and Rammy. I don't think this needs explanation.

Pool 6:
Zara and GK. I like warmongers in this format, and I think three Zaras are a pretty scary opponent.

EDIT: Zara and HC, since GK was an imposter here lol

Pool 7:
(Extra GK?)
Pacal and Gilgamesh, less so due to the two themselves but more so due to their competition

Pool 8:
I'm actually going to put my hot take here and say that Lizzy and Gandhi make it out here, with the belief that it may not be possible to stop three Lizzies or three Gandhis, even if three Justins have Cataphracts.

 

[Thrasybulos]

(Extra GK?)

Good catch.
Got me worried for a sec that I had an error in the script responsible for the draw, but luckily it was just a dumb error in the Excel file used for reporting, and limited to Pool 6.
I've edited the post with the correct version.
That "extra GK" is actually an HC.

 

[a pen-dragon]

My takes so far:

• This format disfavours culturemongers, as they will compete against each other for religions and wonders, slowing them down. Still culture is so strong that it will be able to compete with science, but it will not be that much faster than science. The strong econ culture players (Mansa, Willem, (Hatty)) will be less affected than pure religion grabbers like Gandhi, Asoka.
• Similarly wonderbuilders will have a disadvantage, since only one out of three can get them and the others will get failgold if they try for them. This mostly applies to the early game.
• Some AI weaknesses get counteracted, as for example it is less likely that all three Willems will skip Rifling, increasing their chances of winning. Another example is an Ai going wonder-crazy from stone+marble.
• In the pool stage it is more important to win consistently against the weaker leaders than to crush the direct competition for the top-two. I think this mostly favours overall solid but not shiny leaders, like Justi or Saladin. As an example of a leader unfit for this, consider Gandhi: He always has a chance to win, but also always a chance to loose, and will probably pull out a few really flashy result, but not get far overall, as he will loose games against "weaker" leaders, reducing his chances of getting out of the pools.
• Econ builders will be better because they do not compete as much as others and even one surviving will put a timer on the game.

thus:
1: Hanni - Willem
2: DeG - Nappy - Pericles
3: Qin - JC - Louis
4: KK - Saladin - Hatty
5: Rammy - Mansa
6: HC - Zara
7: Gilga - Pacal
8: Justi - Liz

 

[Thrasybulos]

Tournament 3, Pool Phase

Although I've run those games over the past few days, I've already forgotten some of the things of note I'd meant to mention.
Guess I should take notes, as memory apparently isn't what it used to be!

Spoiler Pool 1 :

I thought Willem and Hannibal would have an easy run... but Isabella crashed the party.
If these were human players, you'd view Willem's losses to Izzy as suspicious, since that's how Hannibal got eliminated.
Churchill, who had done well in the previous tournament, performs horribly here.

Spoiler Pool 2 :

Sury doing well for once was surely unexpected.
Which of the two French leaders made it hinged on Napoleon's first game vs Sury: it was a game were one Napoleon was ultra-dominant. There were only one Sury and one other Napoleon left, both much weaker. And Big Nappy chose the wrong target: he killed off tiny Nappy before turning on Sury... except that Sury had for once gone for a competent Cultural attempt, and Big Nappy was too late.

Spoiler Pool 3 :

There was a similar story regarding Julius vs Augustus... but I've forgotten the details.
The real shocker to me here was Qin doing so badly after reaching the semis in the previous tournament.
There were games were the Qins just played badly, but one issue for him (them ) was that in his good games, two Qins would do well instead of one. His loss to Joao for instance was a typical example: there were two big Qins, and a tiny Joao. But the Qins fought one another bitterly, with none getting the upper hand, and that allowed the Portuguese to backdoor a late Space win.

Spoiler Pool 4 :

Hatty does the job while Saladin pulls a Qin and has an abysmal performance.
Shaka disappoints too.
Kublai crushes everyone.

Spoiler Pool 5 :

That one went as expected, with Mansa and Ramesses almost perfectly tied.
Cyrus fared better than in the first two tournaments.
Sitting Bull was apparently exhausted by his unexpected decent showing in the previous tournament and became Sleeping Bull in this one. Even Freddy (Freddy !!) bested him.

Spoiler Pool 6 :

The random draw lead to this funny pool with all three American leaders.
Washington is widely considered the best (least bad ?) of the three... and yet here he has the worse performance out of the three, while Lincoln repeats his feat of the previous tournament!
Second place was a match between HC and Zara, and that happened to be their last match-up. Zara needed to win both games vs HC to make it. He won the first... but HC saved his skin in the second game.

Spoiler Pool 7 :

Pacal, Suleiman, Gilgamesh, Genghis: the four of them had a decent shot.
Instead, we got... Darius.
For once, he failed to disappoint.

Spoiler Pool 8 :

The results for this pool are unfortunately pretty much meaningless.
The games here were played on the Season 1, Game 7 map... which happens to be one of the most unbalanced maps to ever feature in AI Survivor.
Only two games out of the 30 played here failed to see the AI in Cathy's start win!
They both involved Toku, with Ragnar and Justinian respectively taking advantage.

Spoiler Current Bracket :

I've already played the next round (the knockout rounds are played much faster), but as requested, I'll wait a bit before posting the results.

 

[Genghis the Barbarian]

Game 1: Mansa. I think these settings are good for Izzy, but three Mansas are a LOT of Mansas.

Game 2: Sury, actually. I think Sury is uniquely equipped to stop HC - very aggressive, will take religions away from HC, expands a lot. I could see aome infighting between the HCs too.

Game 3: Louis. I think the Pacals will all hide in their corners while at least one Louis just snowballs

Game 4: Justin, Cataphracts vs Keshiks is not even a question.

Game 5: Ram, actually, he’s a bad matchup for Willem (better at the culture thing, better start, religion econ boost)

Game 6: DG, only because of his opponent

Game 7: Darius, as much as I hate to say it, since these two will not fight very often I’m going to go with the better econ leader.

Game 8: Probably the toughest and most interesting matchup. I think I’m going to go with Ragnar though. The Hatties will just take culture stuff from each other, while Ragnar can either pick off one Hat at a time or mould into one gigantic Ragnar who godzillas Egypt out of existence.

 

[Thrasybulos]

Tournament 2, Knockout phase round 1

Spoiler Prefinals 1 :

Mansa's message to Izzy was loud and clear: "You do not belong here."

Spoiler Prefinals 2 :

HC had a tough time but prevailed in the end.
He never went for culture.

Spoiler Prefinals 3 :

Pacal waited until his elimination was secured to start playing.

Spoiler Prefinals 4 :

Alright, that one was a shocker.

Spoiler Prefinals 5 :

It would seem the Ramesses curse is real?

Spoiler Prefinals 6 :

Wasn't expecting Lincoln to put up such a fight. Is he actually quite good at this?

Spoiler Prefinals 7 :

Could this be Darius's time?
Well, let's not get ahead of ourselves: it was indeed a very good matchup for him.

Spoiler Prefinals 8 :

Tough fight, and note that Hatty's victories were conquest victories: she laid some smack too!

Spoiler Current Bracket :