Combat system - statistical stuff from >20 games

[Rhandom]

Dan, when a person approaches you with good statistical evidence that there is a problem with your game, the LAST thing you should do is blow them off with a "people are whiners" comment.

Here is a very, very simple proposal for a quick test. Make a small, all grasslands map (call it Kansas). Set plains at no defensive bonus. Now there are no hidden factors to to screw with the statistics. All calculations should be simple. If you want to be real cute, make all of the units have the same attack and defense factor. Make a data sheet showing Attacker experience, wounds at start of combat, the same for defender and whether it was fortified, and the number of hits recieved by each side. this should tell you with no wiggle room or assumptions what is going on with the combat calculations or the non-random number generator (my personal bet for the cause).

I don't want to do it myself, having more than enough to do on my own, but I think I might now.

 

[Zachriel]

Funny thing about luck. There is a bell-curve with most results in the middle, with the exceptions as rare events on either end. This bell-curve also applies to people. If you take a 100,000 players, some will be lucky, some won't. Most will be in the middle. That is a lesson from statistics and from life.

But take heart, just because you have been unlucky so far does not mean you will be unlucky in the future.

 

[DarkwingGT]

No offense Grasshopper but I completely understand luck (I'm usually one of the luckiest gamers out there) and Bell Curve and what not (I've taken Statistics and Probability classes at Georgia Tech).

But I will look on the bright side, when Civ 4 comes out I may never lose a battle and then I might complain it's too easy.

 

[Zachriel]

Originally posted by DarkwingGT
10 to 1, right, sure. So that indicates that regardless of technological superiority, I still need a 10 to 1 ratio. That's crazy.

Actually, you don't want to play fair. You want to bring as much force to bear as is practical. If you must take chances, and of course that will happen in any competitive game, then that is what the randomizer is all about.

I rush with Veteran Legions (3/2) against fortified Spearmen (1/2) all the time. They usually win as long as the town is not on a hill, or across a river (civulator shows it as 70% chance of victory). That shows that at least on my machine the randomizer is working correctly. Rome has been my latest fun, whenever I feel like a quick game of conquest.

 

[cephyn]

In my experience, the 'killer phalanx' problem occurs when a defensive unit is of a higher xp rank than the attacker AND has full HP. Full HP seems to have a huge effect.

What I'm saying is that it is so common for me, my strategy is to "break" the defensive unit by taking 1 HP away from it. I'll throw 7 or 8 attackers against a 'killer phalanxer' (not necessarily a spearman, of course) just to get his 1 hp away. Then, after that, the next guy can usually kill him.

Additionally, it seems that whoever loses the first hp in a fight will rout the opponent, rarely losing more than 1 hp (assuming both troops were full to begin with) -- I see this a lot when the defender has a lower xp level than the attacker -- it will almost always win when it gets the first hit in.

Something is goofy with combat. It just is. Maybe I'll do a statistical study, i dunno. But something IS goofy.

 

[Salvor]

Originally posted by Rhandom
when a person approaches you with good statistical evidence that there is a problem with your game, ...

What "good statistical evidence"? All I saw was someone who modded the heck out of the game, made some assumptions that he knew weren't correct about defense modifiers for plains and rivers and the effect of veteran units, admitted that his calculations were rough and simple, then posted that 389 of 528 battles went one way so therefore the calculations were flawed.

How do we know that 1:1 was the correct expected result?

How do we know that all situations were identical (how do you set up 528 identical situations without re-loading the same game)?

How do we know that the mods he made to the game didn't skew the expected results?

Sorry, but I too have studied probability and statistics, including modeling and sampling, and this is not good statistical evidence by any stretch of the imagination.

 

[muppet]

Wow. That's a pretty extreme result.

389 out of 528 on what is suppose to be a coin toss is (without a calculator handy) somewhere in the 5 standard deviation range. That's something like it shoud happen less than 1 in 1,000 times... I think (approximating) your result should be closer to 1 in 5,000.

528 is a large sample set. For a coin toss, you only need around 200 as a sample set for a high confidence level. I believe most statisticians would feel very safe with a 250 sample set.

That being the case, the evidence IMO, is sufficient on the basis of statistical theory. The only question really is the method. Is there absolute certainty that the test conducted is suppose to be a 50/50 result? Like there isn't something that one forgot to take into account?

However, that it is "possible -- however improbable" might suggest that you have really really extreme luck! If you like gambling, you should buy some lottery tickets real soon!

 

[Dan Magaha FIRAXIS]

Originally posted by Rhandom
Dan, when a person approaches you with good statistical evidence that there is a problem with your game, the LAST thing you should do is blow them off with a "people are whiners" comment.

Here is a very, very simple proposal for a quick test. Make a small, all grasslands map (call it Kansas). Set plains at no defensive bonus. Now there are no hidden factors to to screw with the statistics. All calculations should be simple. If you want to be real cute, make all of the units have the same attack and defense factor. Make a data sheet showing Attacker experience, wounds at start of combat, the same for defender and whether it was fortified, and the number of hits recieved by each side. this should tell you with no wiggle room or assumptions what is going on with the combat calculations or the non-random number generator (my personal bet for the cause).

I don't want to do it myself, having more than enough to do on my own, but I think I might now.

First of all, I didn't intend to blow anyone off, nor did I ever call anyone a "whiner". If I was in the business of blowing him off, I would not have even posted here; I am very much interested in his results, but also wanted to know if he had happened to do any reloading, as many players have no idea that saving and reloading will yield the exact same results.

There very well may be unintentional anomalies in the random number generator, and if there are, and they can be substantiated, we want to know about them, but unless all the programmers are lying to me, other members of Firaxis, and the public at large, there are no built-in combat "bonuses" for the AI.

My "conspiracy" comment was nothing more than a lighthearted jab, poking fun at the "the game is cheating against me" movement, but I suppose the subject of combat results is far too grave and serious a topic to have a little fun with

Now, on to the business at hand. What I'm going to try to do in the next week or so is use a super secret debug version of the game to put together a completely barren map with a single terrain type and populate it with an array of AI and player units for the sole purpose of determining combat results. If anyone else has any specific requests or thoughts on how I should lay this out, I'm all ears. This should allow anyone to test the combat system to their heart's content, and in a much more unbiased and efficient way.


Dan

 

[joespaniel]

Dan,

Aside from the unsinkable AI trireme in ocean squares *ahem*, I dont have any problem with the combat system.

In fact, aside from a very few, very rare occurances (we lost an infantry army to longbowmen in the Roman succession game) it seems to be pretty even.

Im very satisfied with the new game.

Now get to work on the editor/mp version!

-joespaniel

 

[Ohwell]

Originally posted by joespaniel
(we lost an infantry army to longbowmen in the Roman succession game).

That was my bad. I just have bad luck .

I also have no problems with the combat system, except with Naval combat, which is highly unsuccesful. Battleships can be killed by Galleons and Ironclads, even Transports!

Anyway, it is a good game overall, I love bombarding!

 

[cephyn]

Dan,

I would like to ask that in your test, try and model some instances where I've seen the most extreme "1hp stands" -- happens MOST often, id even say exclusively, in enemy capitals. Is there a bonus defenders get in a capital i dont know about? Ive seen defenders turn back HORDES of my troops suffering only a hit point of damage -- we're talking (more rarely) pikemen or riflemen stopping 30 some cavalry.

Happens when i attack an enemy capital, ESPECIALLY if its the last city they have.

 

[DarkwingGT]

Dan, I thank you for you comments and the fact that you even post here. Most companies don't even do that.

I'd appreciate being able to test that map on my system. And I'm not sure if this helps, but the situations I mostly run into about unusually lopsided combat is the "Miracle Comeback". As the name implies, it's where I have a superior unit such as a Knight or Cavalry attack a Spearman, knock them down to 1 HP from 4 or 5, not lose a single health and then proceed to be routed by the Spearman. This happens a lot in my fights. Note that I never use Regular troops, I only build troops once I have a Barracks in that city, so you can assume I only use Veteran or Elite units.

From me, it seems that the first 3 or 4 HP for enemy units act like they should, but that last HP takes an act of God and Congress to beat (whichever is slower, so that would be Congress).

 

[Dan Magaha FIRAXIS]

Just to clarify, I'll be creating a map with NO cities, no terrain features (it'll all be the same terrain), just units ready for you to run through attack (or defense) with. I can place units with the the debug version but not cities or anything like that, so this would look a lot like a chessboard, with a pantload of units but nothing else.

Obviously this will be a lot different than the actual game with terrain bonuses, cities, and so on, but it seems like what you guys want to test is controlled combat without all those factors. Is that a fair assessment?

Dan

 

[Loopy]

Originally posted by Shockwave
*still remember the time two of his marines were killed by. . .

I've said it before and I'll say it again, this is the main reason why people think combat is buggy. Those "unusual" outcomes are much more memorable than the expected outcome. How often do you remember a 1 HP defender dying after being attacked?

Just to say something new, I'll post my current theory for the second reason people think combat is buggy. Sometimes when you save and reload, you'll seemingly always lose the next couple of battles regardless of what unit attacks and in what order. Aside for causing you to remember all these "improbable" losses, you might think that the computer is cheating. My theory is that it's really just the random number generator. It's not saving "battle outcomes" per se, but a bunch of numbers. If the next 5 numbers it's saving are 0.95, 0.98, 0.97, 0.93, 0.93 and you need to roll a high random number in order to win, you'll lose no matter what you do. The AI didn't decide that it needed to cheat or to win a bunch of battles, you just saved a crappy seed number.

This topic has been hashed out over and over again. Still I have yet to see a (preferably un-modded) saved game that demonstrates a broken combat calculator. I would love to see a save game where that lone unfortified spearman with 1 HP left on plains kills 10 tanks in a row. Everyone and their cousin says it happens to them all the time, yet they all forget to offer a save game. Go figure.

 

[Akka]

Originally posted by Dan Magaha FIRAXIS
Just to clarify, I'll be creating a map with NO cities, no terrain features (it'll all be the same terrain), just units ready for you to run through attack (or defense) with. I can place units with the the debug version but not cities or anything like that, so this would look a lot like a chessboard, with a pantload of units but nothing else.

Obviously this will be a lot different than the actual game with terrain bonuses, cities, and so on, but it seems like what you guys want to test is controlled combat without all those factors. Is that a fair assessment?

Dan

Here what I would suggest to do for a good statistical test, Dan :
build a map with two grassland tiles aside. Set the defense bonus for grassland at 0 %. Put the HP for ALL experience level at 1. Now, put 500 or 1000 warrior on a tile, and the same number of warriors on the tile just aside, but belonging to another civ. Set them to be at war to each other. Now launch the game, and come back counting how many survivors there is at the end.
Each warrior is 1/1 in A/D, and has 1 HP, so each one as exactly 50 % chance to win and 50 % chances to lose. As all XP level giv 1 HP, there is no "wounded" unit, nor any promotion stuff, to alter the chances of each side.
In one turn, one side will be annihilated. Statistically, there should be one survivor. If there is more than 10, then it's probable that there is a trouble with the random generator.

 

[Hurricane]

Akka has a good suggestion, but I think the conspiracy theory believers won´t be happy with this, since it doesn´t test for "Miracle Comebacks", "Super Capital" or "Killer Phalanx"-effects.

But I´ll gladly test it, just to prove that the random number generator doesn´t generate any strange streaks.

 

[cephyn]

Originally posted by Hurricane
Akka has a good suggestion, but I think the conspiracy theory believers won´t be happy with this, since it doesn´t test for "Miracle Comebacks", "Super Capital" or "Killer Phalanx"-effects.

But I´ll gladly test it, just to prove that the random number generator doesn´t generate any strange streaks.

I have to agree here, not for conspiracy reasons.

The times i think the calculations are "off" is precisely when i see 1 hp stands and super capital or super last city problems. I see some anomolous results otherwise, but never on the scale of 1hp stands in capital cities. 8/

Ive never encountered an unexpected result on grassland. 8( I guess what im saying is that im not as worried about the combat probabilities being screwed up, its when odd things happen when there's obvious reasons for them to happen -- ie the computer does not want to lose its last city or its last unit in a city. Its like then a "dont be destroyed" flag goes up and it will beat back my troops.

Ive never saved/reloaded a game for combat purposes. I don't play that way.

As for the exceptional result == exceptional memory theory, I just don't buy it. I can't think of any times ive been surprised at winning a battle (while defending) when i know i shoulda lost. Ive lost some i know i shoulda won, but Ive never had riflemen repel cavalry at the same rate a computer capital city can. Ive also never been in such a dire position as that where I would hope the 'dont be destroyed' flag would go on.

Its not combat randomness im worried about, its more "hidden bonuses" that I would expect to find -- grassland fighting i dont have a problem with. the weirdness i see usually happens in cities.

 

[s3d]

Originally posted by Dan Magaha FIRAXIS
Just to clarify, I'll be creating a map with NO cities, no terrain features (it'll all be the same terrain), just units ready for you to run through attack (or defense) with. I can place units with the the debug version but not cities or anything like that, so this would look a lot like a chessboard, with a pantload of units but nothing else.

Obviously this will be a lot different than the actual game with terrain bonuses, cities, and so on, but it seems like what you guys want to test is controlled combat without all those factors. Is that a fair assessment?

Dan

Another request : Please test units with different number of hitpoints. I have suspicion that number of hitpoints don't affect result of combat.

 

[Lt. 'Killer' M.]

Originally posted by Salvor


What "good statistical evidence"? All I saw was someone who modded the heck out of the game, made some assumptions that he knew weren't correct about defense modifiers for plains and rivers and the effect of veteran units, admitted that his calculations were rough and simple, then posted that 389 of 528 battles went one way so therefore the calculations were flawed.

How do we know that 1:1 was the correct expected result?

How do we know that all situations were identical (how do you set up 528 identical situations without re-loading the same game)?

How do we know that the mods he made to the game didn't skew the expected results?

Sorry, but I too have studied probability and statistics, including modeling and sampling, and this is not good statistical evidence by any stretch of the imagination.

If you`d read my post thoroughly, you`d have seem that when I made assumptions or changes they were ALWAYS affecting the balance in MY favor so 1:1 is actually a BAD result for me. And I don`t even get 1:1.

The rest of what comes to my mind would have to be taken out of the post by the moderators so I`ll just save them the hassle....

btw: I have several courses in statistics under my belt as well and regularly use statistcs as tool for paleontological analyses. I just wanted a quick feel for the combat thing - otherwise I could have sat down with SPSS and spent A LOT of time - for not much since the graphical representation IS NOT ALWAYS a true indicator of the battles since m some of my units have more HP than can be displayed correctly..

 

[Lt. 'Killer' M.]

Originally posted by Loopy

If the next 5 numbers it's saving are 0.95, 0.98, 0.97, 0.93, 0.93 and you need to roll a high random number in order to win, you'll lose no matter what you do. The AI didn't decide that it needed to cheat or to win a bunch of battles, you just saved a crappy seed number.

YES YES YES!!!!!!

What i mean is that I wonder whether the number generator produces random number - or whether perhaps the numbers aren`t approaching randomness given enough goes at it!

rolling a dice will often give you :

6 6 4 6 6 5 1 6 6

but when it`s only 1 "1" in 200 rolls and then that happenes in 20 games..... that`s just not "random" as "random" for a game should be!

And also there`s the question whether the application of the random numbers to the battle results is without glitch.

I know there are a lot of psychological factor involved; that`s why I did a rough test and will now do an exact one!