Cognitive error?

[Sweetchuck]

Ok, I want to test this theory.

Suppose I am attacking a gallion with a privateer.

The attacking privateer has an attack of 2 and defense of 1. The defending gallion has a defense of 2 and an attack of 1.

Statistically, in say 10 separate encounters, how many times would the privateer win and how many times would the gallion win?

My initial thought is that since the privateer has an attack of 2 and the gallion has a defense of 2 (assuming both have 4 hit points), they would split? I might be incorrect on this assumption since I do not have the battle sequence handy, so maybe someone can provide some insight here.

Once I have determined an accurate stastical measure of the probability of the privateer winning and the probability of the gallion winning, I will "sample" 10 or so rounds to see if it actually happens.

Can anyone provide me with an accurate account of the battle sequence?

 

[Tribute]

They should split the battle wins. The galleons should win a little bit more because of the coast/sea 10% defense. But if on ocean, it should be just about equal.

 

[Chamnix]

Ocean is +10% also. If they each start with 4 hp, then the defender should win about 55% of the time.

EDIT - "win" means opponent sinks (not just an individual battle).

 

[ecuwins]

The only cognitive error I see is the one where ten encounters are assumed to be statistically significant.

 

[Sweetchuck]

The only cognitive error I see is the one where ten encounters are assumed to be statistically significant.

Fair enough, I'll run a couple of sets.

 

[Padma]

Make that a couple hundred tests, and then we can discuss cognitive bias.

 

[Sweetchuck]

C'mon Padma - you're interfering with scientific research here.



Ok, first trial. Privateer vs. gallion, on secondary(?) coast (the medium blue between the very light/white blue immediately on the coast and the darker blue of deep sea - I was on a whales tail, but I don't think that matters). Both privateer and gallion had 4 hit points.

Privateer lost 6 matches and won 4 in a set of 10.


Second trial - same conditions.

Privateer lost 6 matches and won 4 in a set of 10.


So far, the cognitive bias theory is proving accurate.


Third trial - again, same conditions

Privateer lost 7 matches and won 3 in a set of 10.


So, based on my initial investigation of the theory of cognitive bias, in 30 matches total, the privateer lost 19 matches and won 11 giving it a 36.7% probability of winning in this scenario.


Interesting.


I will work on this further, all in the name of science.

 

[eldar]

Based on what we know (2 attacking 2.2) your results are about what I'd expect.

Oh and the different types of water tile are Coast, Sea, and Ocean. They all get +10% defense bonus in an un-modded game.

 

[Pyrrhos]

Err...

You're forgetting one factor - if you do well, the AI is programmed to "challenge you". This is very evident in your test runs as the galleon should only have won a total of 16½ encounters, not 19.

This is a statistically significant deviation.

 

[eldar]

if you do well, the AI is programmed to "challenge you".

No it's not.

 

[Sweetchuck]

Err...

You're forgetting one factor - if you do well, the AI is programmed to "challenge you". This is very evident in your test runs as the galleon should only have won a total of 16½ encounters, not 19.

This is a statistically significant deviation.

Ah-hah!

So, assuming that my mindset is based on what should happen in this encounter, statistically, cognitive error is NOT realized since the AI is "tweaking" the battle sequence in favor of the gallion considering I am doing very well in the game?

 

[Padma]

In any given game, on any given level, the AI does not modify the chance of winning/losing. Period.

Which is why I said run a couple hundred sets. In your statistically insignificant sample, all kinds of weird results can occur. Over thousands of runs, though, you will see things approach the results predicted statistically.

 

[Sweetchuck]

So, does the AI "tweak" the battle system considering how well a player might be doing in a game?

I am hearing conflicting suggestions.

As far as statistics goes, and considering what little I know of how the battle system actually works, what would the standard deviation be for the battle that I am using as a test?

From what I understand about stats, the standard deviation could be calculated based on the battle formula, which I know little about. Once the standard deviation is calculated, the ideal sample size could then be determined - and we could know for sure if that sample size is tens, or hundreds, or thousands - which are all suggested here.

I am guessing that something in the tens would be adequate, but again - I do not know the battle system formula. You generally do not need a huge sample size from a population based on a simplistic setting.

If some smart fella here could determine the standard deviation (and then the correct sample size), and this site is loaded with smart fellas - then we could adequately test this theory.

Science!

 

[eldar]

The AI does not modify the battle the outcome of a battle based on "how well" the human player is doing.

If I had the time I'd set up three scenarios:
1. Human 15 cities, AI 5 cities. Human is "doing well".
2. Human 10 cities, AI 10 cities. Human is "doing okay".
3. Human 5 cities, AI 15 cities. Human is "doing badly".

Then play the same battle 100 times in each scenario.

However it is a tedious exercise which I already know would come up with the same outcome in all three instances.

 

[eldar]

Tedious it may be, but I made a start nonetheless!

Attached are 2 files:
1. A save with 100 Privateers vs 100 Galleons, Human has 15 cities, AI has 5.
2. A save after those 100 battles with the results as follows:
- Human wins: 50/100 (incidentally 12/50 resulted in enslavements, ~25%, which is the expected rate).
- AI wins: 50/100

The upload system is down atm so I can't attach/post a link to the .biq. However it is very easy to set up.

However I believe that, given my 50/100 is "above average" already, the theory that the game favours the AI in battles when the human is "doing well" would appear not to hold water. Unlike the 38 Chinese Galleons that were sunk and not enslaved....

 

[Sweetchuck]

Excellent work, eldar.

This myth is officially busted then?

 

[miltonfriedman]

I don't get this at all-- what "cognitive error" is there?

Because we are not sure how the combat system operates, there is no way to know what results should we expect. There are 2 possible scenarios:

1. Because AI receives 10% defensive bonus, you should expect to lose many, many times because you are less likely to win each battle
2. Because AI receives 10% defensive bonus, you should win 4.5 times out of 10.

It seems that we are assuming the second scenario is more likely to occur. If that is so, there is no need to "randomly sample" 10 trials out of the 30 trials you have administered. In fact, there is no standard deviation involved. You won 11 out of 30 times, so use the binomial formula (30 chooses 11)(.45)^11(.55)^19

If the value surpasses the alpha, then it's significant.

 

[eldar]

Because we are not sure how the combat system operates

Except we know precisely how the combat system works and we can model it in its entirety. Thousands of games have provided empirical evidence that we've got it right. That, and the developers have come out and told us how it works anyway. There are even utility programs that will work out combat odds for you (and these all work by running n-thousand iterations of each battle).

Therefore we know that in each round of combat (i.e. for each HP won/lost) in a 2 vs 2.2 battle, the attacker has a 2 / (2 + 2.2) = 2 / 4.2 = 0.48 chance of winning. And the defender therefore has a 0.52 chance of winning. So overall you'd expect more than 50% of the defenders to win. I just got lucky Even so after 50 fights the score was 23-27 in favour of the defenders.

It gets a bit more complicated as you have to factor in multiple rounds of combat, and my example is not as accurate is it might have been because all the units are Regulars (3HPs). So there are 300-500 rounds of combat involved.

If I'd made them all Elites (5HPs), the minimum number of combat rounds would've been 500, and the max 900. However that's additional work involved to set 200 units' experience levels all to Elite!

 

[miltonfriedman]

Then the calculation is extremely easy once we know that the odds of winning is .48.

 

[Quintillus]

I downloaded eldar's first save and got these results:

After 50 battles:
23 wins, 27 losses (46% victories)
3 Enslavements (13% of victories)
6 Promotions (26% of victories)

After 100 battles:
50 wins, 50 losses (50% victories)
12 Enslavements (24% of victories)

At this point I decided it was extremely likely that eldar had the "Preserve Random Seed" option checked and that was why my results were exactly the same as his. It's possible pure chance resulted in that, but unlikely.

If my ships are in different positions, it's because of the order we attacked in. That may affect which ships one, but since it was 100 of the same battle the end results are the same for both of us.