Study on Sabathiel vs Charadon

[Bad Player]

Introduction

It has been hypothesised on many occasions that some civilisations in Fall From Heaven 2 are much less likely to win than other civilisations. In this present study it was hypothesised that the Doviello, in particular Charadon as leader, are significantly less likely to win a game as compared to another rival. The Bannor are not commonly suggested as being overpowered as compared to other civilisations, therefore Sabathiel of the Bannor civilisation was taken as the single AI opponent of Charadon of the Doviello.

Thus, this study aimed to determine whether, under a specific game and map type, AI Charadon of the Doviello was not significantly less likely to win games when matched against AI Sabathiel of the Bannor.


Methods

Civilisation IV complete (being completely prepatched including Beyond the Sword patch v3.13) was installed on Windows Vista Business. The "mod" entitled "Fall From Heaven II" version 0.31 patch "d" was installed onto Civilisation 4 complete.

To enable this debugging, CivilizationIV.ini and CivilizationIV.ini.bak were edited and Cheatcode = 0 changed to Cheatcode = chitpotle.

The game was launched and minimised mostly whilst being run to minimise crashes. The settings were as shown below in the screenshot (altough the human player was later set to random civ). At the start of the game the human controlled units were deleted through WorldBuilder but later on in the study it was discovered that they are automatically deleted if nothing is built.


If a victory condition was fufilled then that civilisation would win and if no victory condition was fufilled then the team with the highest score at turn 500-501 would win the game. Therefore all games would result in a win or a loss for both teams respectively.

It was intended that 25 games would be performed and statistical analysis of the results (wins vs losses) but the Doviello under the control of AI Charadon were doing really badly so there aren't enough Charadon wins (and I didn't feel like dragging it on for another half a dozen games until they won more!).


Results

Sabathiel wins by fufilling a victory condition:









Results continue in the next post (too many IMG tags for one post).


Discussion

A limitation of this present study is that Charadon may gain a hefty advantage through the arrival of the horsemen of the Apocalyse. However it is likely that the AC will rise further when more civilisations are playing which is not the case in a 2 civilisation combat.

During game testing 5 games crashed and in at least 3 of the crashed games Charadon of the Doviello was leading in score. Thus potentially there is a bias in results, however if these instances are added to the win totals of Charadon of the Doviello to 8 (vs 25 to Sabathiel). All the 'wins' of Charadon were via highest score, not by completing a victory condition.

Care should be taken when attempting to extrapolate these results since terrain changes, free map space, single or multiple landmasses, number of barbarians, number of civilisations, and other adjustments are likely to affect the ability of Charadon or Sabathiel to win more often.

The null hypothesis, that AI Charadon of the Doviello was not significantly more likely to win or lose against AI Sabathiel of the Bannor, was rejected and this study concluded that, under these specific conditions, AI Charadon was more likely (I'd like to say statistically significantly because lets face it - it really would be!) to lose against AI Sabathiel of the Bannor.

 

[Bad Player]

Sabathiel wins by getting the highest score:




















N.B.: It's possible that I doubled up on some images... tell me if I did and I'll edit it.

 

[Bad Player]

Charadon wins by getting the highest score:

 

[Calbrenar]

Why these results are interesting I'm not quite sure what the goal is. Are we talking about the balance of the civs vs each other? Your hypothesis sounds like it means in general. If that is the case then obviously the AI is not going to be able to take advantage of how each civ is uniquely played and their own advantages and suggested play style based on these. The AI doesn't do that for any civilization really.

 

[JDexter]

Incredible that you pulled this off, seems to be publishable in Nature, almost.

How did you disable the human player input though (the press Enter dialog)?

 

[tiberion02]

I wonder how much of this relates to the Bannor being the most "vanilla-ish" of the FFH2 Civilizations.

Also, did you have aggresiveAI/AI no build requirement on or off?

 

[Zechnophobe]

An interesting observation, but considering how many variables involved it is hard to justify making any statements based on this.

How about trying again with Mahala? She isn't allied with the barbarians, right? Maybe the main difference is in XP routing.

 

[Bad Player]

Incredible that you pulled this off, seems to be publishable in Nature, almost.

How did you disable the human player input though (the press Enter dialog)?

I described this in methods - just edit (and make a backup copy!) of CivilizationIV.ini and Civilization.ini.bak to Cheatcode = 0 to Cheatcode = chitpotle. Then in game details (once you start FFH2) check the box "Allow debug tools". Then simply press tilde key (~) and type game.aiplay [integer] (which I made 500). Probably you want to press ctrl + z to show the whole map first though.

 

[Bad Player]

I wonder how much of this relates to the Bannor being the most "vanilla-ish" of the FFH2 Civilizations.

Also, did you have aggresiveAI/AI no build requirement on or off?

I posted a picture of the settings in the first post - no checked boxes in that section (so no aggressive AI etc). The only checked boxes were that all victory conditions were enabled. BTW Ignore the civs selected in that shot (I can't remember why they are there anyway).


Perhaps Sabathiel has very basic abilities which the computer can take advantage of rather than the exploitish things some other civs (e.g. Calabim) might have.

I think the overwhelming win:loss ratio for Sabathiel shows that either the AI for Charadon needs to be improved or (more likely IMO) the Doviello civlization needs to be improved.

 

[Bad Player]

An interesting observation, but considering how many variables involved it is hard to justify making any statements based on this.

You can make statements from this data but for the specific circumstances in which it was performed (ie balanced tiny map with only those 2 AI civs/leaders). The problem lies with extrapolating the data - but this same test can be performed under other conditions obviously (e.g. different map types etc). However doing this test with 2 civs removes a lot of variables and so I would recommend only 2 civs but you can easily change the other variables.

How about trying again with Mahala? She isn't allied with the barbarians, right? Maybe the main difference is in XP routing.

It took me many hours to run all those (28 in the end) games - if I do it again it would be great if other people could help out by doing the same things so each person only does a few games - then all the data could be put together!

Are there a few ppl willing to do this?

 

[Bad Player]

Why these results are interesting I'm not quite sure what the goal is. Are we talking about the balance of the civs vs each other? Your hypothesis sounds like it means in general. If that is the case then obviously the AI is not going to be able to take advantage of how each civ is uniquely played and their own advantages and suggested play style based on these. The AI doesn't do that for any civilization really.

Well the hypothesis (unfortunately) has to refer to that specific situation - balanced map, tiny, no modifiers (e.g. aggressive AI) on, etc. However I did choose settings which I hoped were fairly general (e.g. temperate climate, medium sea level, balanced, no modifiers, etc) so that some extrapolation to a generalised argument could be made. However I did note in the discussion that care needs to be taken in doing such an extrapolation.

Therefore I recognise and to some extent agree that some leaders will have advantages in some areas which others won't have. If we could get enough people to help by running more studies we could compare Sabathiel and Charadon under many different map conditions and then come up with a strong case for a more general argument about Charadon's weakness (or not as the case may turn out).

As I said in an above post, if there is a weakness to Charadon's AI (which my study seems to suggest), then it could be due to:
1) Charadon's AI doesn't play as well as Sabathiel's AI
or (and I think 2) is probably the case)
2) Charadon's civ is weaker than Sabathiel's civ (AI or human).


But testing with humans introduces biases so it's a lot less reliable than testing with AI. But of course humans would be more desirable if there was a way to remove biases.

 

[MagisterCultuum]

I've always considered Charadon to be the weakest leader in the game. He really only has one trait (aggressive), since the Barbarian trait's benefit can quickly disappear, it can reduce xp gain, and its penalties don't go away.

 

[Calbrenar]

My guess is that the Doviello can only really be played properly if they pretty much mass warriors and go all out on the offensive while rushing anything that allows them metal or upgraded troops since they can do that in the field.

The AI won't do that by default it'll play the same way they all play which heavily favors Bannors style and thus skews any AI testing results imo. I agree that this probably indicates that the Doviello AI prolly needs to be edited to set higher priorities for more combat oriented things but I disagree that its indicitive of the Doviello civilization being weak. Not saying they aren't just saying these tests don't really relate to that at all.

 

[rusty217]

As MagisterCultuum said the traits have an impact on the game, and with these settings the traits seem to make it biased to Sabathiel.

The first religion founded will likely be adopted by both civs, thus making war even more unlikely so Charadon's aggressive trait has little affect.

And the barbarian trait makes Charadon research at a slower rate than Sabathiel and alliance with the barbarians doesn't really help, as the AI under any leader can handle them, so they will be extra xp for Sabathiels troops but not for Charadon.

And Sabathiels trait charismatic allows his troops to level up faster, and since he is the only one getting barbarian xp it helps alot.... and it also adds 1 per city, helping Sabathiel even more.

So a likely solution would be to make Charadon more likely to go for early rushes and maybe an extra trait to make up for the Barbarian trait since all other leaders with the barbarian trait have a third trait aswell....

 

[Grey Fox]

To remove random factors as much as possible, try modding the events and set every event except the needed ones (armaggedon, orthus, etc etc, dunno exactly) to 0 at the tag which determines the chance for the event to show up in the game. (I'm too tired to check what it is atm)

Then play on a mirror map, without unique improvements.

And Charadon is a leader which is hard to play successfully as. Especially for the AI.
Balance sure is hard when civs play so different.

Or we could just use Capcom's Principle of Balance

Give every character something “so good that it’s broken.” Include so much variety that by the time anyone ever figures out which broken thing actually does ruin the game—the game will be dead by then anyway.

 

[sylvain5477]

I think the overwhelming win:loss ratio for Sabathiel shows that either the AI for Charadon needs to be improved or (more likely IMO) the Doviello civlization needs to be improved.

On the contrary I would suggest the IA of FFH 2 special civ have yet to be implemented (as planned by the team).
Played as Human, the doviello does not seems that bad, especially on rush time.

 

[Bad Player]

On the contrary I would suggest the IA of FFH 2 special civ have yet to be implemented (as planned by the team).
Played as Human, the doviello does not seems that bad, especially on rush time.

Perhaps you're right - I can't really judge whether it's the civ/leader itself which is too weak or the AI way of playing it. Perhaps if it was run against another civ which relies on human style strategies (e.g. Clan of Embers has a similar need to rush and also has the barbarian trait) that could be a better comparison. Maybe I should try that when I get time (or some ppl to help...).

But if I could use humans to play the civs so that the correct rush strategy was used then that would be very useful (but so many biases I don't think the study would be useful unfortunately).

 

[Bad Player]

To remove random factors as much as possible, try modding the events and set every event except the needed ones (armaggedon, orthus, etc etc, dunno exactly) to 0 at the tag which determines the chance for the event to show up in the game. (I'm too tired to check what it is atm)

Then play on a mirror map, without unique improvements.

I don't know how to mod stuff but regarding doing a study removing those variables; I am tending to think that some of those variables may have an important influence on the success of the AI and so should be included. To remove the influence of e.g. Sabathiel getting a really good starting location with a few special resources all you have to do is have enough games such that it creates sufficient randomisation to give Sabathiel and Charadon and equal number of really good (and really bad) starts. This also means that if e.g. pigs give Charadon a really good start then it will show up in the results and give a more realistic indication of how well Charadon would do in a game.

Therefore I suggest leaving special resources in but running lots of games to average everything out.

 

[xienwolf]

It could be a decent idea to run two copies of the same civ/leader against each other for a couple runs to ensure that a 50/50 split is actually possible to ever see. Then try same civ, different leaders and slowly branch out from there.

It could actually be interesting to see what kind of victories are common for 2 Mahala playing against each other.

 

[Bad Player]

It could be a decent idea to run two copies of the same civ/leader against each other for a couple runs to ensure that a 50/50 split is actually possible to ever see. Then try same civ, different leaders and slowly branch out from there.

Well you never actually get a 50/50 split for a good and bad starts - it's just a statistical probability. If you flip a coin 5 times you might get 4 heads and 1 tail which makes you think that coins are more likely to come heads up. But actually that's just sample size and if you flip a coin 100 times you are very unlikely to get 80 heads and 20 tails and if you flip a coin 1000 times you are extremely unlikely to get 800 heads and 200 tails.

EDIT: But the advantage of what you suggest is that it would give an indication of how many sample games you would need to get an approximately 50/50 split.