The way I see it, an AI's performance in a given game depends on 3 factors:
- The game composition. And while there's a bit more to it, it can be handily approximated as the peaceweight repartition. Eg, we know Mansa is good, but Mansa in a game where all the other leaders are low peaceweights is just gonna die.
- The starting position. Sullla puts a lot of emphasis on whether the initial available ressources match that civ's starting techs or not. And that sure has its importance. A few startling fast eliminations have drawn the community's attention to the metal situation: a civ whose only metal ressource is a 2nd or 3rd ring Iron, while its neighbour has BFC Copper, is indeed in grave danger. But the more I play those games, the more it seems the most common overriding factors there are : the quantity of available land, and the neighbouring starting positions. In a nutshell, a leader who can peacefully expand to 10 cities, while bordering a boxed-in civ... will tend to do well.
- The AI parameters. That's what sets the different AIs apart and what we'd like to actually measure.
Now, play enough games, and the first two factors should be cancelled. Trouble is, "enough" is probably quite large.
One other caveat: there are more low peaceweight AIs in the game than high peaceweights (feature, not a bug: high peaceweights are the natural allies of the human player, the game is balanced around having a human player). So AI Survivor tournament rules will naturally favour low peaceweight AIs.
Alternate histories eliminate the 4th, unlisted factor: dumb luck. But factors 1&2 remained unchanged.
So that's why I decided to try and take a shortcut: I replay each game, but shuffling the AIs around each time. Doesn't help with factor 1, but essentially eliminates factor 2, while also providing information about the relative strengths of the starting positions on the maps (which we could guess at, but we'll have hard data to complement the guesses).
The playoffs games will use the same maps, but with different AIs (those who performed best in the Opening Round games - some are the same as what happened in the live competition, but only some).
And I've dropped the Wilcard game, replacing it with an 8-game "Wildcard League" (keeps the player count constant, and allows the eliminated civs to figure in more games).
Now, owing to the afore-mentionned imbalance between low and high peaceweight AIs, the Opening Round games favour low peaceweights, and that translates into the playoffs.
But an interesting thing about my "Wildcard League" is that it reverses that imbalance: high peaceweights outnumber low peaceweights there. I'm not done with it yet, but I've played two games (with 20 runs each) so far with a 4 vs 2 advantage for the high peaceweights... and let me tell you it was ugly for the low peaceweights.
Now, let's have a look at Season 4, game 8.
That game had a 4v3 advantage for the high peaceweights, with Hammurabi and Ramesses in their ranks.
Sullla hasn't done Alternate Histories for that one, but I have.
So Hannibal performed the best there, with Stalin, Pericles, and Ramesses not far behind as far as winning was concerned.
Now, my replays of that game, shuffle edition, have show that the best starting positions on that map were Hannibal's, Stalin's, and Pericles'. And all three performed accordingly in the AH. Ramesses still got 4 wins in spite of not being favoured by the map.
Now, here are the results where the positions are shuffled:
With 9 wins, Ramesses crushed the competition (spoiler : so far, I haven't had a game where a leader got more than 9 wins).
And you'll note that he's seriously outperformed Hammurabi.
So that's why I think that Ramesses is the best leader in that field.
But out of those 9 wins, only one was achieved with a limited core. He needs to expand, and that means attacking, not merely defending well. And the lack of Mounted units means a serious lack of offensive punch.
So I think that Ramesses is notch or two above the field... but is that enough to pull a win with essentially one arm tied behind his back?