New Global Ranking System for 2003

[Aeson]

With the introduction of the QSC (Quick Start Challenge), a new method for incorporating the results of the GOTM and QSC into one Global Ranking is needed. At this time, other fixes to problems in the old rankings are also being included in the change. We (The GOTM Staff) have come up with two different proposals for this change, and offer them up for discussion here.

Comments and suggestions are welcome.

Addition of the QSC to the Global Rankings:

To accomodate for the inclusion of seperate events with seperate scoring systems, the Global Rankings will use a base event value that all results in that event will be normalized to. The values of these base event scores are 100 and 20, for the GOTM and QSC respectively. Those values were arrived at as the QSC encompasses 15% of the turns that the GOTM does, and also includes 5 points for the added participation shown by those who send in their QSC writeups.

Changes to the Global Rankings System:

Normalization

Previously in the GOTM, normalization of scores was done based off of the average event score. This resulted in very high scores in the more difficult games, especially Deity ones. Also, using the average doesn't result in a score with an upper limit through the mathmatics involved. This wouldn't allow us to use the 100 and 20 point system, as often the results would be well in excess of those limits.

To 'fit' the scores into the 100 and 20 point range, each result will be divided by the highest result in that event, and then multiplied by the event value. That way each event's winner will get 100% of the event value, with everyone else receiving a percent based off of how their score compares.

Event Combination

Events will no longer be averaged together. The events scores will be added to determine the overall global ranking. This way, submitting a 'poor' event score won't bring down your average. A submission will always increase your total.

Event Count

Events will now count towards the global rankings based off of an event 'lifespan'. This lifespan is linked to the difficulty of the game. The goal here is to encourage players to play and submit even if the difficulty is one they have trouble with. The losses are just as valuable for comparison purposes as wins.

Also, all events that haven't exceeded their lifespan will continue counting towards the Global Rankings, rather than just the best 3 out of 5. This is to encourage and reward consistant gameplay.

A more involved explaination of how event lifespan is calculated will follow later in the post.

Event Fade

Because the lifespans are somewhat long, the effects of each event score on the Global Rankings will slowly fade over time. For example, an event with a 6 month lifespan will count 6/6 the first month, 5/6 the second, till it reaches 1/6 on the sixth month, leaving the rankings altogether on the seventh. This means that playing current events will always be rewarded in the rankings, while past events will still count as well, just not as much. It also means more stability in the rankings, as when an 'exceptional' score ceases to count, the effect won't be all at once.

Implementation

GlobalRanking = Event1 + Event2... + EventN

For each event Score:

EventX = (PlayerScore / TopScore) * EventValue * Life

Life = (Lifespan - Age) / Lifespan
PlayerScore = Player's score in the event
TopScore = Top score in the event
EventValue = 100 for GOTM, 15 for QSC, 5 for QSC writeup

There are two different implementations of Lifespan, '6789' and '679'.

6789 Lifespan = 4 + Difficulty
679 Lifespan = 6 for each difficulty, 7 for Emperor, and 9 for Deity

 

[Aeson]

Examples:

Here is an example of how the new system compares to the old. I have selected 5 players who's results show how different categories of playstyle are affected.

Player A consistantly plays the GOTM, and consistantly scores at or near the top of the individual game rankings.

Player B is new to the GOTM, but in the two months has turned in top flight scores.

Player C played for a few months, turning in very good results, but then hasn't submitted lately.

Player D consistantly plays the GOTM, with results that represent the average submission.

Player E hasn't played in several months.

We feel these changes greatly stabilize the Global Rankings. Player E was ranked 39 in GOTM13, and then fell entirely out of the old rankings this month. While Player B, who wasn't in the rankings 2 months ago, jumped all the way to the top. In the new systems changes are more gradual.

The old rankings were based mostly off of performance in 2 or 3 games, while the new rankings reward both performance and consistancy.

A more indepth look at the overall effects can be found here in an Excel spreadsheet.

 

[cracker]

I would add a quick link to the global ranking list on the new GOTM website so that players can find their own names in the current list:

http://www.civfanatics.net/~gotm/players/index.htm

The far right hand column of the table is included to give everyone a rough idea of approximately where they would fall in the list if we sorted the ranking over to the new system immediately. The actual numbers of relative positions will be determined based on the "fade out" system that you help us choose and then updated to include the Gotm15-Russia results when they are available in the first week of February.

We do value you commenst and questions, so make sure you ask for more details or examples if you feel there is anything you need to make this process clearer and easier for all of us to understand and use.

Also just as a matter of perspective, it is important to note that 407 different players have submitted at least one GOTM game in the past six months. The total game submissions include well over 2000 recent game results. This represents a very large pool of data on which to base some of our decisions and evaluations of current and future games.

 

[col]

I like any ranking system that puts me ahead of Aeson

But seriously this looks like an excellent peice of work and very much like ranking systems in other games that I play - chess and bridge. Consistent performance is the only way to reach the top of the new table - but there are lots of good features

- more equal weighting of each game
- encouragment for people to submit 'bad' games
- encouragement for people to not drop out

In conjunction with a scoring system that values alternate approaches to winning equally, this looks like an excellent basis for the future.

 

[DaveMcW]

Looks good. I almost missed the vote question at the very bottom.

Originally posted by Aeson
6789 Lifespan = 4 + Difficulty
679 Lifespan = 6 for each difficulty, 7 for Emperor, and 9 for Deity

I don't really see much difference between the two. The second option drops the value of Monarch and Emperor games by 1, resulting in a 15% lower weight for those levels. If we are arguing over 15%, I'll take the least complex formula.



Hmmm, maybe an example to illustrate the question...

Person A gets a normalized score of 0.70 each month for 6 months.

Person B gets a normalized score of 0.50 each month for 6 months.

Person C gets a normalized score of 0.30 each month for 6 months.

Person D gets #1 in an Emperor game during month 4 and #1 in a Deity game during month 6. They don't play any other games.

What should the rankings be at the end of the 6th month?

 

[civ_steve]

I think this is a decided improvement over the current system! A result is that many more QSC submittals will occur. This allows those who don't submit a final result due to time constraints to participate through their QSC contribution.

My only significant concern is the normalization to the Top Score. I look forward to any future communications on how that score may be determined, if changed from the existing method.

I'd recommend the 6789 lifespan determination method.

 

[Aeson]

In your example DaveMcW, the difference would be rather small. Most of the difference between the two fade methods is shown in low difficulty games early and high difficulty games later.

679:

a: 271.111 (100%)
b: 193.651 (71.4%)
c: 116.190 (42.9%)
d: 87.302 (32.2%)

6789

a: 301.389 (100%)
b: 215.278 (71.4%)
c: 129.167 (42.9%)
d: 105.556 (35%)

As you can see by the relative percent (based off the 'top' score in this example, player a), really only the 'hang on' effect for higher difficulty games shows a difference. The other players all show consistant play, and their rankings relative to each other are consistant between both the formula's because of that.

 

[Aeson]

I like any ranking system that puts me ahead of Aeson.

That was one of the problems with the old rankings. By milking 1 game all of a sudden I jump up clear to #3, my previous month's ranking being #73. No single game should have that much effect on the rankings. The new rankings still allow for jumps based off of exceptional short term results, but not to the same extent.

 

[RufRydyr]

Got a little lost in the details of the formulas and statistics, but understood the vast majority of it. 100% in favor of either new system.

Can you make an Excel spreadsheet that would allow us to plug our numbers in or is the top score component going to make it impossible to make until all games have been processed?

 

[Aeson]

The spreadsheet I linked to at the bottom of the second post will allow you to plug your own numbers in both new ranking systems, the 'top score' is determined dynamically by the function MAX(R1C:RNC). 1 and N being 'first' and 'last' rows of player data respectively.

The old system isn't set up that way though, and adding/changing any value would require changing all the values for that event because it will modify the overall average those scores are based off of. It's not a big deal if you are adding just one or two results (unless they are very extreme), but will make things come out slightly different.

 

[Aeson]

Also, just in case you are interested... You can quickly look at how the difficulty 'lifespan' works by changing the difficulty (in text form) at the top of each event column. The calculations are done based off of that as well as the 'current month'.

(after thinking what might have lead to the file request in the first place...)

I appologize if how I saved the file makes all this data less than obvious that it exists. I felt that it was easiest to make comparisons between the systems by sliding all the 'data' columns underneath the rank/name pane so that the far right 'total' column was right next to the 'name' column. Just use the slider to expose all the event data columns.

 

[Skyfish]

I vote for the 6,7,9 Lifespan system. The much higher difficulty of Deity is accounted in a better way with that system.

 

[Bolka]

Very good job, there are good improvements compared to the old system, that makes it better than the old system overall.

Though the one thing I don't like about it is if you want to have a good rank, you must play a lot. So a player missing a few games will not have a chance to have a rank equal to his ability to play the game. I prefer a ranking system that compares players by their skills than by their attendance, or amount of time they can spend playing.
Of course, there must be an 'averaging' mechanism that ensures no player can take the lead out of a couple good games, but there should also be a mechanism that allows a player to miss a few games, or that forgives the odd very poor performance.

For example, taking out of the count the two worse results/missed games out the list would help this reasonably, without losing the advantages of the new system.

My 2 cents...

 

[Moonsinger]

I don't really understand the meaning of those digits. Nevertheless, I agree with Skyfish that the higher difficulty of Deity should have a longer lifespan. Having multiple rankings for each level of difficulty may be not be a bad idea either. For example, a separate global ranking for Deity, Empire, Monarch, Regent, and so on.

 

[col]

Originally posted by Bolka
there should also be a mechanism that ...... forgives the odd very poor performance.

For example, taking out of the count the two worse results/missed games out the list would help this reasonably, without losing the advantages of the new system.

The new system does this. Since your score can only go up then playing a poor game would never leave you worse off than not playing that game.

It does encourage people to 'have a go' rather than only submit games that they feel are good ones.

If I understand the proposed system correctly ....

 

[RufRydyr]

Moderator Comments:
I have posted this note at the beginning of your message to indicate that you points are valid and important but that they really do mix the two issues:
Global Ranking vs Individual Game Scoring

It is important that we successfully separate thes two issues because your comments are almost entirely focused on the individual game scoring that we are also in the process of revising.

The reasons for the two changes are different and Milking versus speed will eventually have no impact on the Global Ranking -- cracker

This is a big improvement over the previous method, but it is still dependant on the score that is produced by the game. So killing everyone except one city and then milking to 2050 will still produce the highest scores and rankings, right? It would be nice to figure out a way to do an alternative scoring (QSC provides some excellent ideas) that will end the need to milk. I milk all my games, but it would be nice to compare scores without having to put all that time in. Maybe a scoring system weighted more towards victory date? The problem with trying to use the QSC formulas for GOTM is that then you'd focus on getting maximum Wonders, units, buildings, etcetera and basically have to milk in a different way. Trying to determine an optimum finish date after everyone has submitted and giving bonusses for finishing before then while penalizing completions after that would require the submission of many games per person.

***Perhaps an Optimum Finish Date could be set at the start of each GOTM. Instead of many people sending in their 2050 games (along with 10AD), people could send in the 10AD and the last turn they played. A bonus would be added to their score for number of years victorious before the Optimum Date, whereas an amount would be deducted for victory dates after that date AND THEN plug the Adjusted Score into the formulas above. That might end the need for milking GOTM games. [Losses would not get a bonus for finishing early, but maybe get penalized for finishing later or give a bonus for each victory type] The goal would be to finish every game with a victory as quickly as possible.

The game gives a bonus for finishing before 2050, but it's tiny. Kind of like how future techs are practically worthless to score. Maybe just adjusting the early win bonus is all that is needed. Instead of Optimum Finish Date suggested in the last paragraph, just give a set bonus for every year or turn that a player is victorious before 2050ad. The amount of the bonus could be determined after all the submissions are in. The hard part would be to have fair boni from game to game as to not upset the Global Rankings.

My 3 cents....(inflation)

 

[Ribannah]

Originally posted by Bolka
The one thing I don't like about it is if you want to have a good rank, you must play a lot. So a player missing a few games will not have a chance to have a rank equal to his ability to play the game. I prefer a ranking system that compares players by their skills than by their attendance, or amount of time they can spend playing.

The solution to this well-known dilemma is to keep two rankings.

One is for the (annual?) championship. If you play more you score higher.

The other is for ratings of playing strength. These can be adjusted over time by taking eg 90% of someones last rating + 10% of the latest performance. You'd have to play a minimum number of games a year to stay on the active list, but your rating won't deteriorate if you don't play.

The present proposal, attempting to combine the two, is a so-called 'king-of-the-hill' ranking system which has some charm but also comes with obvious drawbacks.

 

[Yndy]

@Rufrydr:

Note that this is the score calculation for the Global ranking. It refers to the comparability and summation of scores that you obtain in each GOTM. The actual score of each GOTM is another topic.

 

[Aeson]

We realize that no one ranking system is going to be able to reward both skill and participation as they would be rewarded with seperate rankings dedicated to one or the other. The goal of this discussion is to strike a balance between the two, where skill can get a good ranking as can participation. Because of this, skill and participation combined will get the best rankings of course.

The limiting factor to how much we can do in regards to the ranking is administration time required. There are several things in the works or have been proposed, but we only have so much time to split among the various tasks we already have. A full blown database with sortable lists ranking the players based on category ('skill', 'participation', spaceship, occ... ect.) and definable time frame would be ideal, but at this time rather unrealistic.

Suggestions of this sort are welcome, just bear in mind the time requirements to implement them may not make them feasible.

 

[Ribannah]

Once you have the score per game figured out, there is no extra administrational burden by maintaining both a championship ranking and performance ratings. A few lines of code would suffice.