Rating the traits: an analysis

oranje willem

Prince
Joined
Feb 6, 2009
Messages
323
What do you do when you are lying sick in bed with a laptop next to you? You try to kill time with silly things you might normally not do. I noted all the lists of traits people wrote in the thread and let the computer calculate some statistics. Some traits were ranked in a random order within a tier, in that case I gave them all the same average number, resulting in 29 ranks for each trait. The computer calculated the average and the standard deviation.

Here is the data if you are interested:
Spoiler :
Org: 3 1 6 3 1 4 1 1 2 2 2 1 1 5 1 4 3 1 1 6 2 3 2 2 2 2 1 4 7 -> 2.55 1.72
Fin: 2 3 3 4 4 7 3 2 1 2 3 3 3 1 3 7 1 5 2 2 1 6 2 2 1 1 3 3 2 -> 2.83 1.67
Spi: 10 4 1 1 5 1 2 7 8 5 4 6 4 5 5 3 7 4 5 9 5 4 5 2 4 7 6 1 1 -> 4.52 2.43
Cre: 9 2 11 5 2 9 4 8 4 5 5 2 5 1 9 1 8 2 3 1 4 2 2 5 3 3 6 8 4 -> 4.59 2.83
Phi: 1 5 7 2 11 3 7 5 3 2 1 8 7 5 4 2 4 9 11 5 3 5 5 5 11 5 4 2 9 -> 5.21 2.94
Cha: 5 6 2 7 7 10 5 6 5 5 6 4 6 5 8 5 2 8 6 3 6 8 5 5 9 6 2 6 3 -> 5.55 1.99
Exp: 4 7 4 6 3 8 8 4 6 5 8 5 2 9 6 6 6 3 10 4 7 7 5 5 5 4 6 7 6 -> 5.72 1.87
Ind: 8 9 8 9 6 5 6 3 9 5 8 9 8 5 2 8 5 6 7 8 8 1 5 9 6 10 6 5 8 -> 6.62 2.23
Imp: 6 8 10 8 8 6 9 9 7 5 8 7 10 9 7 10 9 7 4 7 9 9 10 9 10 9 9 9 5 -> 8.03 1.66
Agg: 7 10 5 10 9 11 10 10 10 10 8 10 9 11 9 10 11 8 10 10 10 10 9 8 8 10 10 11 11 -> 9.48 1.35
Pro: 11 11 9 11 10 2 11 11 11 10 11 11 11 9 11 11 11 10 9 10 11 11 10 9 7 11 11 11 10 -> 10.07 1.83


Conclusion for each trait with (average rating/standard deviation=SD) behind the name):

Organised (2.55/1.72): Considered the best trait averagely, with a relatively low SD which basically means there is not much contest in its position.

Financial (2.83/1.67): Very close to organised and these two traits together really form the 'top tier' traits in Civ IV according to the forum. Again not much contest in how powerful it is as the SD is also low on financial.

Spiritual (4.52/2.43): A big gap before the next trait shows up: spiritual, starting the 'mid tier' is 'the best of the rest', however it has a considerably larger SD than FIN/ORG. This tells us people value SPI very different from each other. Spiritual is a very playstyle-dependant trait.

Creative (4.59/2.83): Very close to SPI, Creative is ranked #4. Very high SD on CRE just like SPI which means people disagree about it's power a lot. I personally think it's a difficulty issue: the power of some traits is very difficulty dependant and creative is one of those. This list would be better if it had a certain level attached to it, but that would require a lot more data and work :crazyeye:.

Philosophical (5.21/2.94): A bit further away from SPI than CRE, but still pretty close. PHI has the largest SD of all and it's also the only trait that is ranked worst and best by multiple people (which also explains the high SD). Imo strong on every difficulty, but better on higher ones. I think it's mostly a 'how to use properly' issue: some people can exploit this trait so good it will give them a better midgame tech position than FIN or ORG would ever give them (mostly on immortal+).

Charismatic (5.55/1.99): Another 'mid tier' trait, however with a considerably lower SD than the previous ones. My conclusion is that the power of CHA is pretty difficulty and play style independent.

Expansive (5.72/1.87): Very comparable with CHA both in average rating as in low SD. Average trait, no more no less.

Industrious (6.62/2.23): Almost 1 point behind EXP in the average rating, IND is the last of the 'mid tier' traits. A much debated trait, some say its better on lower difficulty because there you can get a lot of wonders , some say its better on higher difficulty, because it gives you a shot at wonders (GLH/Pyramids) that would otherwise be much harder to get and the fail gold is not to be underestimated. I think it's also play style dependant, some say most wonders are not worth the hammers, others love them. We see this back in the SD and overall we think it's average at best.

Imperialistic (8.03/1.66): Imperialistic is far behind IND. A weak trait overall and not much contest about it (low SD). It's considered better than AGG/PRO, but clearly worse than the others.

Aggressive (9.48/1.35)/Protective (10.07/1.83) The forum agrees that PRO/AGG are the two weakest traits. I was wondering which one would be worse, because of the discussion going on the last few days (see Protective Underrated thread). Protective is clearly worse, it's a bit more contested though, but I think the higher SD is mostly because of one individual who ranked it the second best trait there is, without much help for the end result though: PRO is statistically the worst trait.

Conclusion short:
Top traits: ORG, FIN
Many 'average' traits: SPI, CRE, PHI, CHA, EXP, IND
Bad Trait: IMP
Worst traits: AGG/PRO

Not very surprising results (maybe a bit too wide average trait section). Any trait combo with one top trait and one average trait is a very strong combo in my book.
 
Pretty cool :), but this forum ain't exactly a democracy...
 
Makes sense.

Org
Fin
--gap--
Spi
Cre
--small gap--
Phi
Cha
Exp
--small gap--
Ind
--gap--
Imp
--gap--
Agg
Pro

With the possible exception of Imp being higher than Ind, that's about how I'd rank them too.
 
The freedom to express one's own opinion is free speech, not democracy (i.e., you might be able to say what you want, but this mightn't mean that anyone takes your opinion seriously). A democratic organization is one in which everyone's opinion matters when it comes to deciding what's good and what's not.

To be perfectly honest, If I were taking the statistics, I would weight your opinion, Mr. Fleme, about 10 times more than the opinion of someone who thought that imperialistic was a better trait than philosophical. The stats here are weighed down with the opinion of noobs.

Not that sometimes the more controversial opinions can't be thought provoking, though. The stats are interesting, however.
 
Top: Traits easily leveraged by both human and AI alike

Middle: Subtler traits requiring human intelligence and tailored play to leverage

Bottom: Traits most beneficial to dumb AI "spam" playstyles
 
Still, my opinion is just my opinion and everyone here, atleast presumably, expresses their own opinion rather than someone elses. In that context, whether a "vote" is taken seriously or not is no longer an issue since it's a reflection of their opinion.

But, let's not make this into a political debate and focus on the point here: You valuing my opinion 10 times more than someone elses. Surely that shows lack of judgement on your part! :lol:

I agree on the point about controversial opinions; One only needs to look at the recent thread about protective to see that it holds true. It's good for conversation but I still don't see how it'd do anything to convince someone that protective is actually useful over other traits that give tangible benefits. And while I'm on protective, let's cease on that topic right away because we don't want this thread to derail :D Flammable topic these days!
 
You valuing my opinion 10 times more than someone elses. Surely that shows lack of judgement on your part! :lol:

Well... if you say so... :lol:
 
I was actually thinking of doing something like this myself. Nice job by the OP.

And, amazingly, I think the results correspond very closely to reality. I disagree with the consensus on a few things: IND and EXP are too low, CRE and SPI are too high. But then again, I'm also of the opinion that after ORG and FIN at the top, there is a mass of almost equal traits that can only really be differentiated by personal taste and style of play...followed by IMP, then AGG, then PRO. The gaps in the rankings reflect these three tiers markedly.

I think we can all pat ourselves on the back for being so smart.
 
It is democratic, because I didn't weight the ratings. I did consider it, but it would make the statistics worthless, because my opinion of whose list is better would basically determine the outcome. By the way, I checked and no one (not counting random order as correct) had the same list as this average one ^^ funny detail, but there are 39916800 different lists possible so yeah what are the odds.
 
Garbage in, garbage out - not only because the voters are incompletely informed (and with different levels of understanding at that), but also because the question they were being asked was underspecified.

Shrug.

That said, this looks as though it's essentially equivalent to a preferred voting system using Borda count. So I wonder

(a) what kinds of results do we get if we apply other preferential voting systems to the ballots collected
(b) do we get answers that make more sense if we use a different voting vector?

If what we're trying to do is estimate the ranking, then borda is fine. But if we're also trying to get the right spacing (which is sort of implicit in the calculation of standard deviation), I'm not sure it's the right answer.

In other words, does 10 4th place votes really mean the same thing as 5 3rd place votes and 5 5th place votes if we are trying to judge which tier a result belongs in?
 
In other words, does 10 4th place votes really mean the same thing as 5 3rd place votes and 5 5th place votes if we are trying to judge which tier a result belongs in?

It doesn't and that's exactly what the SD is calculated for, to show how different the lists are from each other, you can draw your own conclusions from that. I agree though that many people here don't know the game as well as some others and thus rate the traits totally wrong, but there are so many things like difficulty level and play style that I figured the fairest thing to do was to just create a general idea of how traits are ranked by people on this forum. And to be honest the result is pretty much how I rated the traits, only switching positions with traits close to each other.
 
Thanks for the analysis Oranje William.:goodjob: It's a lot of work and makes interesting reading. I think it comes out quite well, allowing for the caveats you mentioned. I think the democratic method is fairest because this is a general survey and specific mileage may vary.

I'm only noble and don't vote in the polls, so I don't influence the results. I probably would have come out the same as your results, allowing for the fact that Organized is more important as you move up in difficulty.
 
It doesn't and that's exactly what the SD is calculated for, to show how different the lists are from each other, you can draw your own conclusions from that. I agree though that many people here don't know the game as well as some others and thus rate the traits totally wrong, but there are so many things like difficulty level and play style that I figured the fairest thing to do was to just create a general idea of how traits are ranked by people on this forum. And to be honest the result is pretty much how I rated the traits, only switching positions with traits close to each other.

Hmm - I don't think SD works that way: it's tells you more about smear than skew, doesn't it?

And I want to amend my earlier comment - given that it is a case of GIGO, there's really no reason to assume that doing the calculations in a different way would produce results that are "better" in any real sense.
 
SD tells you about the variation, but the way of collecting the data was invariably flawed. It treats every difference in rank the same (ignoring for the moment that some players are better than others). So, the difference between #2 and #3 will be the same as the difference between #7 and #8. A better method would have been to have people rate each trait out of 100, but of course the OP was only working with the information available.
 
SD tells you about the variation, but the way of collecting the data was invariably flawed. It treats every difference in rank the same (ignoring for the moment that some players are better than others). So, the difference between #2 and #3 will be the same as the difference between #7 and #8. A better method would have been to have people rate each trait out of 100, but of course the OP was only working with the information available.
I agree, maybe the next time a similar thread is opened people should rate out of 100, it would give a better picture of the gaps between the qualities of traits, aka the tiers.
 
Why even this^^ (though a good idea :)) might not work. Major sarcasm alert.

Spoiler :
Never mind... this was sorta mean... should play nice... pity you can't delete posts...
 
Do a likert scale! More and less useful for each trait! Haha!

Of course then there's sources of bias, like someone trying to prove their point and the fact that the survey is on this forum rather than elsewhere, and if not careful even in the instructions/phrasing of the question(s).

Some stratification based on comfortable difficulty would be useful also, as it is beyond doubt that some of the traits are actually more/less useful depending on difficulty.

Similar strata for speed might be necessary too. When units are cheaper/more expensive and tech gaps more or less significant relative to move speed, results may vary.

All that and we STILL get some noise potentially.
 
Do a likert scale! More and less useful for each trait! Haha!

Of course then there's sources of bias, like someone trying to prove their point and the fact that the survey is on this forum rather than elsewhere, and if not careful even in the instructions/phrasing of the question(s).

Some stratification based on comfortable difficulty would be useful also, as it is beyond doubt that some of the traits are actually more/less useful depending on difficulty.

Similar strata for speed might be necessary too. When units are cheaper/more expensive and tech gaps more or less significant relative to move speed, results may vary.

All that and we STILL get some noise potentially.

There are too many variables to make one list perfectly. You would have to make lists depending on game speed map size difficulty level, perhaps even which victory conditions are allowed, a special OCC list etc. There have been so many topics about rating traits and stuff, but for what purpose? I felt someone had to draw a conclusion out of it and we all know it ain't perfect :lol:
 
Top Bottom