Scoring system? - calling all mathematicians

[Denniz]

Has anyone noticed the huge gap in scores between the peaceful and war-like VCs? I am not sure how that is going to affect the medal competition and GPR for 4OTM. Based on some of the past Civ3 discussions, this may prove to be rather interesting and intense debate.

I think the basic problem is that war-like VCs lend themselves to both high base score and bigger date bonuses as they can finish quicker.

We need a better victory bonus system that values each victory condition relative to its difficulty. Maybe a decreasing % based on a "victory window" for each VC.

It probably wouldn't hurt to look at a QSC-like approach to base score. Score needs to take quality into consideration as well as quantity.

I am not a math-wiz but I know there are alot of them out there. I wouldn't be surprise to find that few of you have already been thinking about it. I might be better get this worked out before we score GOTM-01.

My $0.02.

 

[AlanH]

My first concern on seeing the scoring algorithm and when results started to roll in was how we are going to provide a normalised score for each game so that the global rankings can be meaningful.

We always knew that the Firaxis scoring system was likely to be flawed for our purposes, and that we'd have to revisit it. Until version 1,09 it didn't have a modifier for difficulty level. Now that it has we might have to reverse it for normalisation purposes. Sure there's no modifier for victory condition, but the new factors for culture and techs add into the scores for later vcs, and we need to see what effect this has. If we determine that they are not rewarded sufficiently to compensate for a later victory then we'll have to mod the factors.

The good warmongers are often the more experienced players, and the others who pursue military victories have fewer turns in which to play badly. So, as ever, we have the challenge of distinguishing whether low spaceship or diplo or culture scores resulted from faults in the scoring system or from sub-optimal play towards those vcs.

It took a committee of elite players and some time to develop the Jason system for Civ3. That was done after quite a few games had been played on different maps, and it still raises some players' blood pressure. So I don't think we are ready to jump the gun on this right now. And it's not even necessary, as we can rework the GOTM 01 scores for Global Ranking and medal purposes when we can see the full picture and take a considered view of it. That's possible at any time, as we have the raw data in the saves.

My €0.02

 

[Denniz]

My first concern on seeing the scoring algorithm and when results started to roll in was how we are going to provide a normalised score for each game so that the global rankings can be meaningful.

We always knew that the Firaxis scoring system was likely to be flawed for our purposes, and that we'd have to revisit it. Until version 1,09 it didn't have a modifier for difficulty level. Now that it has we might have to reverse it for normalisation purposes. Sure there's no modifier for victory condition, but the new factors for culture and techs add into the scores for later vcs, and we need to see what effect this has. If we determine that they are not rewarded sufficiently to compensate for a later victory then we'll have to mod the factors.

The good warmongers are often the more experienced players, and the others who pursue military victories have fewer turns in which to play badly. So, as ever, we have the challenge of distinguishing whether low spaceship or diplo or culture scores resulted from faults in the scoring system or from sub-optimal play towards those vcs.

It took a committee of elite players and some time to develop the Jason system for Civ3. That was done after quite a few games had been played on different maps, and it still raises some players' blood pressure. So I don't think we are ready to jump the gun on this right now. And it's not even necessary, as we can rework the GOTM 01 scores for Global Ranking and medal purposes when we can see the full picture and take a considered view of it. That's possible at any time, as we have the raw data in the saves.

My €0.02

I don't disagree with most of what you say. I was hoping to hear that some of those behind the Jason system are already working on something for Civ4.

Where I would think there might be some urgency would be in terms of the score-based medals and awards. A new system could change the results such that the winners would be different. What happens then? (Not that I am in any danger of being affected. )

It might be better to start out with just the Fastest Finish awards until we are sure.

The Jason system did spark some rather spirited debate. More than once, IIRC. And I didn't get started following/playing until GOTM31.

From what I have seen here and in other forums, my gut tells me the VCs aren't balanced. I have been know to trust my gut in advance of proof, but there is no reason anyone else should.


O/T, €0.01 called cents, too?

 

[Renata]

For normalization purposes (for global ranking), we could simply pick some arbitrary number and award it to the top scorer every month. Then scale everybody else by the same factor. Might not be perfect, but good enough for getting by with.

For example, pick 10,000 as the arbitrary number, and say the scores received for one game range from 100 to 20,000, with most players coming in around 8,000. For the global ranking, top player gets a 10,000, bottom player gets a 50, and most people get about 4,000.

 

[Renata]

@ Denniz -- The fact is, though, that the Jason system wasn't developed or implemented until the GOTM had been running for almost a year and a half. CivIII had been out even longer than that. This time around, with awareness up front that the scoring system probably isn't optimal, it shouldn't take that long, but it seems unrealistic to me to expect people to be working on modification before we even have any data to work with.

 

[AlanH]

O/T, €0.01 called cents, too?

Yes ......

French Centimes were around a while before US ¢ I think.

 

[Denniz]

@ Denniz -- The fact is, though, that the Jason system wasn't developed or implemented until the GOTM had been running for almost a year and a half. CivIII had been out even longer than that. This time around, with awareness up front that the scoring system probably isn't optimal, it shouldn't take that long, but it seems unrealistic to me to expect people to be working on modification before we even have any data to work with.

Fair enough. I don't really expect a quick solution. I do think people might have some ideas that can be tested as the data comes in.

To expand on what I was talking about in the OP. I think a scoring system will need to:

a) figure out a way to calculate a base score that values all the things an empire creates [pop, wonders, improvements, specialist, culture, etc.]. This is not a new idea, I have seen some discussion around this before, not to mention the whole QSC from Civ3.

b) Bonus each VC based on how quickly they are reached relative to when such a victory is possible. [There is precident from the Jason system for figuring earliest possible dates.]

Example: Bonus = Base * ((FinishTurn - BestTurn) / (EndTurn - BestTurn)). This is my theoretical contribution to the discussion. But, I don't have the math skill to work out if that would be balanced.​

Gathering ideas can't hurt, can it?

 

[Denniz]

Yes ......

French Centimes were around a while before US ¢ I think.

I'm not trying to be the ugly american. I didn't know 1/100th of a Euro was called cents, too. Of course, I did think that cents came from the English penny.

 

[AlanH]

Nah! Olde Englishe pennies were much more sensible. 240 to the Pound. 240 divides by lots of numbers, so you can have many, many fractions of a pound in old pennies without rounding errors. Try dividing a Euro or a dollar into three or six or eight equal parts! :runs for cover:

 

[Grogs]

a) figure out a way to calculate a base score that values all the things an empire creates [pop, wonders, improvements, specialist, culture, etc.]. This is not a new idea, I have seen some discussion around this before, not to mention the whole QSC from Civ3.

The in-game score (the one that displays on the lower-right during the game) already capture most of this, specifically population, land, technology, and wonders. It does lack a direct measurement of things like culture and wealth, but they are recorded somehow (you can see a graph of them on the statistics screen.) This score *doesn't* seem to be cumulative because when you wipe an AI off the map, their score drops until it hits 0 (when you wipe them out.) This reduces, but doesn't entirely eliminate the 'milking' problem. It does make it easier to deal with though.

b) Bonus each VC based on how quickly they are reached relative to when such a victory is possible. [There is precident from the Jason system for figuring earliest possible dates.]

Example: Bonus = Base * ((FinishTurn - BestTurn) / (EndTurn - BestTurn)). This is my theoretical contribution to the discussion. But, I don't have the math skill to work out if that would be balanced.​

Personally, I think that was a pretty good system. Of course the hard part is determining what the 'best finish' date is for any given map condition. It's entirely possible that changing the difficulty level would have the exact opposite effect on the 'best date' for 2 victory conditions. Going from Noble to Emperor might make the 'best possible' date for conquest/domination later and for SS/Diplomatic earlier. Most likely, I think these dates will be developed empirically. Until we have a large pool of finishes from expert players (and I don't think that anyone's a Civ4 expert yet) our margin of error would be pretty large. I think 4-5 GOTM's will help tremendously towards that end.

 

[Denniz]

The in-game score (the one that displays on the lower-right during the game) already capture most of this, specifically population, land, technology, and wonders. It does lack a direct measurement of things like culture and wealth, but they are recorded somehow (you can see a graph of them on the statistics screen.) This score *doesn't* seem to be cumulative because when you wipe an AI off the map, their score drops until it hits 0 (when you wipe them out.) This reduces, but doesn't entirely eliminate the 'milking' problem. It does make it easier to deal with though.

AlanH posted an rundown on the ingame scoring system here. The part I would point out is the weighting factors.

... The Weighting Factors are 5000 for pop, 2000 for techs and land, and 1000 for wonders...

Not only are culture, wealth, improvements, etc. not considered but the weighting favors pop.

Personally, I think that was a pretty good system. Of course the hard part is determining what the 'best finish' date is for any given map condition. It's entirely possible that changing the difficulty level would have the exact opposite effect on the 'best date' for 2 victory conditions. Going from Noble to Emperor might make the 'best possible' date for conquest/domination later and for SS/Diplomatic earlier. Most likely, I think these dates will be developed empirically. Until we have a large pool of finishes from expert players (and I don't think that anyone's a Civ4 expert yet) our margin of error would be pretty large. I think 4-5 GOTM's will help tremendously towards that end.

I would agree that this will be the tough part. Other things like map type, civs, traits, etc. will all have an effect too. There are so many variables that the mind boggles at how it can be figured out. Hopefully, someone will have a more simple idea.

 

[BeefontheBone]

Nah! Olde Englishe pennies were much more sensible. 240 to the Pound. 240 divides by lots of numbers, so you can have many, many fractions of a pound in old pennies without rounding errors. Try dividing a Euro or a dollar into three or six or eight equal parts! :runs for cover:

That's the same reason there are 360 degrees in a circle, among other things - the Babylonian counting system was based on 12s and 60s IIRC, because they have lots of factors, which makes arithmetic, particularly for everyday activities like commerce, much easier. It's only more recently that the advantages of the metric system (and decimalised counting, to a lesser extent - that's mainly just because the Arabic system became more widespread) have been realised.

As for the imperial system of weights and measures, that one beats me - 14 definitely doesn't have a lot of factors!

On-topic, I'm inclined to agree with the OP to some extent - the current system does seem to reward conquest and domination victories, particularly on the smaller map sizes where they're actually far easier to achieve than others, and to achieve quickly, which is the main factor - the exponential time bonus is huge.

 

[AlanH]

the advantages of the metric system

What advantages? Base ten is only used because we have ten fingers. If only we had evolved with twelve fingers we'd have a far better counting system.

 

[Venom3]

That's the same reason there are 360 degrees in a circle, among other things - the Babylonian counting system was based on 12s and 60s IIRC, because they have lots of factors, which makes arithmetic, particularly for everyday activities like commerce, much easier. It's only more recently that the advantages of the metric system (and decimalised counting, to a lesser extent - that's mainly just because the Arabic system became more widespread) have been realised.

Its actually 360 degrees because they estimated that there were 360 days in a year.

 

[Aeson]

I'm pretty sure the scoring system is going to be rewarding "milking" type play regardless of victory condition. The finish bonus will overshadow that with some maps, potentially by a huge margin, and will be relegated to Conquest/Domination in many cases as the victory conditions possible to trigger earliest.

The huge problem with the scoring system is it is only considering the state of the game for that turn. So someone who conquered out to domination in 1500AD, and someone who conquered out to domination in 1800AD, would both get the same score at 2050AD if they ended up both maxing pop, land, wonders, and tech by then.

 

[DaveMcW]

How about this:

The highest score wins the gold medal and 100% global ranking for that game.

The fastest spaceship launch wins the spaceship award and 100% global ranking for that game.

Anyone who won a space race victory gets the highest of these 2 global rankings:
game_score / highest_game_score
fastest_launch_turn / launch_turn

We could probably get a better score distribution if we used a function other than simple division to compare the typical player finish dates and fastest finish dates. But the main idea is to totally remove milking from fast finish game scores.

Once we have enough data to calculate "best dates" and "best scores", the scores can be compared against those instead of the highest player score.

 

[Denniz]

I think that comparing Fastest Finish (FF) by victory condition (VC) with score as a tie-breaker will probably work as is. GPR within a VC by comparing relative dates as DaveMcW says sounds workable.

The challenge, as always , is comparing different VCs. I hadn't considered the effects of static nature of the scoring system that Aeson pointed out. Right now, I don't see how we can have an overall award based on score that isn't just duplicating the domination FF award.

I guess that if everyone understands that, then highest scores becomes another category like the Cow and the FF awards. It may be "the" premier status award but no one should be expecting all VCs to compete. I don't know whether a fair system for comparing different VCs can be created, but I think it would save endless "debate" if we don't say we are doing that right now.

Maybe we need a highest scoring award for each VC to go with the FF and lowest scoring ones we have now. (Maybe just replacing the lowest scoring ones.) It is a little early to say, but I think we will see the FF scoring lower than a better "milked" later finish for most VCs. If we reward both then it brings more value to having to having a set of each type. Maybe a GPR that awards points for percentage of each high score and FF VC attempted, modified by difficulty and number of different VCs attempted.

 

[Shillen]

Maybe we need a highest scoring award for each VC to go with the FF and lowest scoring ones we have now. (Maybe just replacing the lowest scoring ones.) It is a little early to say, but I think we will see the FF scoring lower than a better "milked" later finish for most VCs. If we reward both then it brings more value to having to having a set of each type. Maybe a GPR that awards points for percentage of each high score and FF VC attempted, modified by difficulty and number of different VCs attempted.

Wow thanks for the headache first thing in the morning. I know what all your abbreviations mean but it makes it very hard to understand what you're saying when you're using them rampantly like that.

 

[Grogs]

Maybe we need a highest scoring award for each VC to go with the FF and lowest scoring ones we have now. (Maybe just replacing the lowest scoring ones.) It is a little early to say, but I think we will see the FF scoring lower than a better "milked" later finish for most VCs. If we reward both then it brings more value to having to having a set of each type. Maybe a GPR that awards points for percentage of each high score and FF VC attempted, modified by difficulty and number of different VCs attempted.

With the conditions we played this map on, I don't think milking is going to be an issue. If the fastest finish was around say, 1000 AD, you'd be losing (by my calculations) > 600 bonus points per turn. If (as is my understanding) the in-game score is normalized to a maximum of ~10,000 points, you'd have to do some pretty serious increases in population, land, etc. between turns to compensate for this, i.e. plop down 20 settlers in a turn. The only real disparity I see under those situations might be between a conqueror who razes everything and a dominator who finishes slightly slower, but that seems pretty fair.

The only place I see milking helping is in the final turns of the game. Since you're only getting <50 bonus points a turn as you get near 2050, it may be possible to increase your base score by enough for each of the final turns to offset the loss in bonus points. Since the total bonus in that case would be so low, I don't see it being even close to the fastest finish score (a 2050 Space Ship might outscore a 2030 SS, but probably not a 2000 SS and certainly not a 1900 SS.)

 

[Denniz]

Wow thanks for the headache first thing in the morning. I know what all your abbreviations mean but it makes it very hard to understand what you're saying when you're using them rampantly like that.

Sorry, occupational hazard. If we didn't use acronyms something terriable might happen. The universe might stop expanding or something.

With the conditions we played this map on, I don't think milking is going to be an issue. If the fastest finish was around say, 1000 AD, you'd be losing (by my calculations) > 600 bonus points per turn. If (as is my understanding) the in-game score is normalized to a maximum of ~10,000 points, you'd have to do some pretty serious increases in population, land, etc. between turns to compensate for this, i.e. plop down 20 settlers in a turn. The only real disparity I see under those situations might be between a conqueror who razes everything and a dominator who finishes slightly slower, but that seems pretty fair.

The only place I see milking helping is in the final turns of the game. Since you're only getting <50 bonus points a turn as you get near 2050, it may be possible to increase your base score by enough for each of the final turns to offset the loss in bonus points. Since the total bonus in that case would be so low, I don't see it being even close to the fastest finish score (a 2050 Space Ship might outscore a 2030 SS, but probably not a 2000 SS and certainly not a 1900 SS.)

Different games could play out differently. I think it is possible to get a fast SS win without building a large empire. As Aeson pointed out, Civ4 doesn't average your score over time. I think it is quite possible for the highest score to belong to a later finish date. Someone diverting from a Domination VC to a SS victory could probably out-score an earlier SS date. It really depends on how you want to play your game. It might give some people something different to play for. Time will tell, though.