Thrasybulos
Prince
- Joined
- May 4, 2023
- Messages
- 404
In parallel to my AI League endeavour, I'm going to be running a set of "alternate AH" for Sullla's AI Survivor games, with the objective of ranking the AIs within the framework of AI Survivor.
Saxo Grammaticus mentionned in Eaupx's Season 1 AH thread he was using the AH results to rank the AIs.
That spurred my interest as while that seemed a good idea (far better sample set than the live games), I could see a few issues with that:
The other reason for doing this is that September/October tend to be busy months for me, so I don't have as much time available for running games manually as I've been doing for the League. The temptation was strong to start autoplaying them to keep the pace going, but that would mean losing information (not watching the games, some data becoming unavailable), so I was reluctant to proceed.
Now the League will continue at its own pace, while I can take advantage of autoplay for this project instead.
Here's how it'll work:
I could have used the existing AH results to save time, and just run new AH when they don't exist, and for the new wilcard/playoffs/championship game compositions.
Running new sets of AH has two advantages:
All these games will be run with the console command game.aiplay without me watching (if I were available to watch them, I'd be running League games instead).
The games will be run under my usual ruleset/procedure: S5 rules (no Deity bonus starting techs, no AP), every game run as a scenario (randomized peaceweights every time), no UN, no great spy inflitrations in the AI capitals by the observer civ.
This is going to be a pure data collection exercise: I'm not going to provide write-ups about the games.
Also, I'm not going to collect all the data which is usually collected for the AH: I'm not interested in the number of wars, so I'm not going to bother with that. And I'm only going to keep track of total kills, not the kills per game detail.
It also means that two elements of Sullla's scoring system are going to need revising:
Kill Credit
The game summary tells which civ has captured a city, not which leader. So in games where two leaders from the same civ are present, if both are at war with a leader who gets eliminated, it can be hard or even impossible to tell who got the killing blow.
Should that happen, I'll apply the following rule: the kill credit goes to the first leader who declared war on the eliminated leader (while still at war, obviously).
So, for instance, if Shaka declares on Louis, then Napoleon attacks Shaka, and Shaka gets eliminated, the kill credit will go to Napoleon.
But if Louis had declared on Shaka, with Napoleon piling in, then Louis would have got the kill credit.
(edit: Acametis has suggested a better way around it, allowing to attribute the kill credit to the actual killer).
Runner-up points
Since I won't be watching the screen at the time the win notification appears, I won't know the scores of the surviving leaders to determine who comes second.
In some cases it can easily be told from the width of the score graphs (and even more easily if there's a sole survivor, duh), but in other cases it's just impossible to tell.
So rather than use an unreliable statistics, I'm going to change it.
When discussing something similar in the League topic, Keler has suggested using the autoplay mod instead: since it stops the game when an AI wins, it makes getting the endgame scoreboard possible.
But after checking it out, I'm not comfortable with the fact that the observer civ isn't passive. In particular, the fact it can end up at war through Defensive Pacts has the potential to really alter some game outcomes.
So for this project, I'll keep using the console command.
Instead of the runner-up scoring 2 points, every survivor is going to score 1 point.
I had initially decided to give 2 points to a surviving AI when it was the only survivor besides the winner, but then I noticed something: 1 point for surviving and 1 point for kills makes the total points for each game constant. Killing someone is just getting their survivor point.
There's an elegance to that system which I couldn't resist, so even if an AI is the clear runner-up because everyone else is dead, it still gets only one point.
OK, so this is going to yield each AI's "score" for a given game, over 20 iterations: 5 points for winning a game, 1 point for surviving a game, 1 point/kill.
Next, this is going to be translated into the "AH Power Rating".
The winner of an AH is going to be the AI who won the most game iterations (in case of a draw, the score will be used as a tie-breaker).
The runner-up is going to be the AI with the highest score (winner excluded).
Now, kill points were a bit tricky. I haven't run exact calculations, but I would say the average for AI survivor should be around 3 kill points per game. I couldn't see a way to translate that precisely over a set of AH games, so here's what I have settled for:
The AI with the most kills gets one point for the Golden Spear award. Now, that can't be shared: it two AIs are tied for the most kill, then no one gets a point.
And then: 1 kill point is awarded for every 20 kills (rounded down). After all, it makes sense: in AI survivor, you get one point for one kill, here we're running 20 games.
Looking at the existing AH data, it should mean a lot fewer kill points attributed than in the live games (few AH sets feature AIs with 20+ kills). But since kills feature in the score, and the score determines advancement (runner-up spot, wildcard), kills are somewhat more important than in AI Survivor. So fewer points, but increased impact: it sorta balances out. Sorta.
Edit: See this post for the new system.
Wilcard game
Each season will feature a number of wildcard spots, depending on the number of AIs who made it to the wildcard game in the live event.
So here, all AIs who didn't make it through a win or a runner-up spot in the opening rounds will be sorted by Score: the top ones will get into the wildcard game.
Alternate scoring
On top of the "AH Power Rating" which is an attempt at translating Sullla's system for the AH, I'll also be using for comparison a very simple system of "Championship points":
Elimination in the opening round: 0 pt.
Elimination in the wildcard game: 1 pt.
Elimination in the playoffs: 2 pts.
"Elimination" in the Championship game: 4 pts.
Champion: 8 pts.
Question for you guys
I'm pretty comfortable with what I've described, but I'd like your input about one thing: who should be the champion?
Should it be the AI with the most wins over the 20 Championship game iterations? Or the AI with the highest score?
If, for instance, we had Gandhi and Gilgamesh in the Championship.
Gandhi wins 6 games, gets eliminated 14 times (survival: 30%), and scores 5 kills.
Gilgamesh wins 5 games, gets eliminated only 3 times (survival: 85%), scores 18 kills.
Which of the two should be considered the winner of the Championship?
Most wins:
Eauxps
Highest Score:
Acametis, Keler (but without kills)
Spoiler Origin of the project :
Saxo Grammaticus mentionned in Eaupx's Season 1 AH thread he was using the AH results to rank the AIs.
That spurred my interest as while that seemed a good idea (far better sample set than the live games), I could see a few issues with that:
- In many cases, the AIs which make it to the playoffs and the championship are not the AIs which the AH indicate, so using the playoffs & championship AH is at odds with the objective.
- Some game setups resulted in very unbalanced situations: using 20 iterations of such setups for a ranking system is bound to have a distortion effect.
The other reason for doing this is that September/October tend to be busy months for me, so I don't have as much time available for running games manually as I've been doing for the League. The temptation was strong to start autoplaying them to keep the pace going, but that would mean losing information (not watching the games, some data becoming unavailable), so I was reluctant to proceed.
Now the League will continue at its own pace, while I can take advantage of autoplay for this project instead.
Here's how it'll work:
- I'll be running a new set of AH for each season.
- For the wildcard, playoffs, and championship games, the AIs will be selected according to the AH results, not according to the live game result.
- Each AH game result (that is, the result of the 20 iterations) will be translated on a 5/2/1* scale to attribute a Power Rating score to the AIs.
Spoiler Why run new AH ? :
I could have used the existing AH results to save time, and just run new AH when they don't exist, and for the new wilcard/playoffs/championship game compositions.
Running new sets of AH has two advantages:
- It ensures consistency. All the games will be run under the same ruleset and using the same procedure, which is not the case for the existing AH.
- It will provide a new result dataset. 40 is better than 20.
All these games will be run with the console command game.aiplay without me watching (if I were available to watch them, I'd be running League games instead).
The games will be run under my usual ruleset/procedure: S5 rules (no Deity bonus starting techs, no AP), every game run as a scenario (randomized peaceweights every time), no UN, no great spy inflitrations in the AI capitals by the observer civ.
This is going to be a pure data collection exercise: I'm not going to provide write-ups about the games.
Also, I'm not going to collect all the data which is usually collected for the AH: I'm not interested in the number of wars, so I'm not going to bother with that. And I'm only going to keep track of total kills, not the kills per game detail.
It also means that two elements of Sullla's scoring system are going to need revising:
Kill Credit
Should that happen, I'll apply the following rule: the kill credit goes to the first leader who declared war on the eliminated leader (while still at war, obviously).
So, for instance, if Shaka declares on Louis, then Napoleon attacks Shaka, and Shaka gets eliminated, the kill credit will go to Napoleon.
But if Louis had declared on Shaka, with Napoleon piling in, then Louis would have got the kill credit.
(edit: Acametis has suggested a better way around it, allowing to attribute the kill credit to the actual killer).
Runner-up points
Since I won't be watching the screen at the time the win notification appears, I won't know the scores of the surviving leaders to determine who comes second.
In some cases it can easily be told from the width of the score graphs (and even more easily if there's a sole survivor, duh), but in other cases it's just impossible to tell.
So rather than use an unreliable statistics, I'm going to change it.
Spoiler Why not using autoplay instead? :
When discussing something similar in the League topic, Keler has suggested using the autoplay mod instead: since it stops the game when an AI wins, it makes getting the endgame scoreboard possible.
But after checking it out, I'm not comfortable with the fact that the observer civ isn't passive. In particular, the fact it can end up at war through Defensive Pacts has the potential to really alter some game outcomes.
So for this project, I'll keep using the console command.
Instead of the runner-up scoring 2 points, every survivor is going to score 1 point.
I had initially decided to give 2 points to a surviving AI when it was the only survivor besides the winner, but then I noticed something: 1 point for surviving and 1 point for kills makes the total points for each game constant. Killing someone is just getting their survivor point.
There's an elegance to that system which I couldn't resist, so even if an AI is the clear runner-up because everyone else is dead, it still gets only one point.
OK, so this is going to yield each AI's "score" for a given game, over 20 iterations: 5 points for winning a game, 1 point for surviving a game, 1 point/kill.
Next, this is going to be translated into the "AH Power Rating".
The winner of an AH is going to be the AI who won the most game iterations (in case of a draw, the score will be used as a tie-breaker).
The runner-up is going to be the AI with the highest score (winner excluded).
Now, kill points were a bit tricky. I haven't run exact calculations, but I would say the average for AI survivor should be around 3 kill points per game. I couldn't see a way to translate that precisely over a set of AH games, so here's what I have settled for:
And then: 1 kill point is awarded for every 20 kills (rounded down). After all, it makes sense: in AI survivor, you get one point for one kill, here we're running 20 games.
Looking at the existing AH data, it should mean a lot fewer kill points attributed than in the live games (few AH sets feature AIs with 20+ kills). But since kills feature in the score, and the score determines advancement (runner-up spot, wildcard), kills are somewhat more important than in AI Survivor. So fewer points, but increased impact: it sorta balances out. Sorta.
Edit: See this post for the new system.
The number of kill points attributed for a game will depend on the total kills: total kills / 20, rounded to the nearest.
31-49 kills: 2 points
50-69 kills: 3 points
70-89 kills: 4 points
The attribution algorithmn will be as follows:
1- Sort by kills.
2- Give 1 point to top leader, decrease his total by 5.
3- If still points to attribute, go to 1.
Wilcard game
Each season will feature a number of wildcard spots, depending on the number of AIs who made it to the wildcard game in the live event.
So here, all AIs who didn't make it through a win or a runner-up spot in the opening rounds will be sorted by Score: the top ones will get into the wildcard game.
Alternate scoring
On top of the "AH Power Rating" which is an attempt at translating Sullla's system for the AH, I'll also be using for comparison a very simple system of "Championship points":
Elimination in the opening round: 0 pt.
Elimination in the wildcard game: 1 pt.
Elimination in the playoffs: 2 pts.
"Elimination" in the Championship game: 4 pts.
Champion: 8 pts.
Question for you guys
I'm pretty comfortable with what I've described, but I'd like your input about one thing: who should be the champion?
Should it be the AI with the most wins over the 20 Championship game iterations? Or the AI with the highest score?
If, for instance, we had Gandhi and Gilgamesh in the Championship.
Gandhi wins 6 games, gets eliminated 14 times (survival: 30%), and scores 5 kills.
Gilgamesh wins 5 games, gets eliminated only 3 times (survival: 85%), scores 18 kills.
Which of the two should be considered the winner of the Championship?
Most wins:
Eauxps
Highest Score:
Acametis, Keler (but without kills)
Last edited: