Adjustments to exclude specific games from GOTM results

[ltcoljt]

Originally posted by Qitai


I WANT Aeson to be playing just as I WANT Sir Pleb to be playing. These two players are among the players I have great respect for and without them, the entertainment and competition value of playing the GOTM will be much reduced! Please do not make any further such suggestion. And if my opinion is of any value, then this negates your opinion that he shouldn't be ranked. So, the choice is still his!!

I want them to play as well, I just think that if they have the best dates then it is only fair for us all to have them.

 

[ControlFreak]

IMHO the best dates in themselves are not spoiler info. But the map parameters used to generate the date is spoiler info. Maybe in the future, the best date could be given but without the modifier details. (Or is it too easy to back calculate to determine what type of map was used?)

 

[Bamspeedy]

Yes, it would be easier to guess if there is a galley crossing or not, because some 'best dates' would be +-500 years or so, depending if you need to get to navigation or not.

 

[ltcoljt]

So, how is it fair for some people to have the info and others not? If I am incorrect in assuming that Aeson has this info prior to playing I humbly apologize.

 

[cracker]

Originally posted by ltcoljt
Which brings me back to an old compliant, relevant here because it is a fairness issue. The best dates are still not being issued with the .sav file. Aeson of course is privy to this information, his games therefore should be removed from the rankings. Sometimes I feel like the staff is speaking out of both sides of their respective mouths. They want to say that the best dates are not relevant but at the same time they can't be released. It doesn't take a genius to see through that one.

It might take more genius than would be evident in a first pass assessment of the situation without the facts.

I can assure you beyond all reasonable doubt that when staff members play the GOTM at my active request and with my encouragement, they have absolutely no knowledge of game content or best dates that all the other players may not have.

Their only advantage in the games is that they are being whipped by me to focus on the big picture multigame issues and in some ways this may lead them to a better perception of what MY next move might be. I am however seeing that the bright minds among the regular player pool are right on track with figuring out some of the key issues that will present in the next GOTM games even before they see the first pixel or byte of the games announcemnts or save files. It is no small coincidence that the brightest minds among the player pool as well as many of the mid level players are picking up on significant issues like trading, expansion, and upgrade planning even before these issue become apparent as key game content elements.

Adjustments that Aeson is required to make to the scoring system are based on comparative curve fitting of results across literally hundreds of games. His one game number in that data could not possibly influence the outcome of any decision in any way that is more or less significant that the influence of all the other games in the pool.

The best dates are also set based on a formula that is based on past player performance in games of that victory type under map conditions that are similar to the current game.

There are a number of people in this world that are beyond reproach and I can tell you that Aeson is one of those. If he told you that the sky was going to fall tomorrow at 11.33AM your local time then you can bet you should buy a hard hat and stay indoors right at that time.

Even with this confidence that I have in the staff and many, many of our players, their games are still subject to the same stringent statistical tests as all the other player's games. Plus they get the specific honor of knowing that I check every one of their games personally.

All of this is balanced by knowing that the staff members need to play the games to maintain a valid "player's perspective". They have to use the same download processes and the same QSC timeline and/or submissions processes that all the other players use. They have the added pressures of needing to complete their personal game commitments and still complete their real life task commitments plus the GOTM volunteer staff roles.

We have real life engineers, accountants, parents, managers, lawyers, statisticians, professors, and other professionals on the GOTM staff. The average age of the staff is between 35 and 40 years of age and for the most part these individual represent the cream of the crop in terms of people you would ask to pick up your kids from school or to make sure that an important bank deposit got to your bank before closing time on this specific date. You should have absolutely no question about their skills, motives, or abilities to represent the best interests of the entire GOTM community while always paying attention to specific personal needs of every individual GOTM player.

In many ways I am the only weak link in the process but with their help we are working on my performance every week.

 

[ltcoljt]

Gee, cracker, all you had to say was that he didn't have the dates.

I stand corrected.

 

[ControlFreak]

I think its good that Cracker had the opportunity to talk a bit about his staff. They are doing a great job and deserve to be commended. They may also be feeling the collective pressure resulting from the decisions made about Ribannah and it was a nice move from Cracker to comment on their ethics as well.

 

[ltcoljt]

I want to stress that my comments may have been a tad on the edge, but reflect my concerns with processes rather than people.

 

[TC3]

I apologize in advance for a slightly off topic post...

Civ3 is a great game, and the GOTM adds tremendous amount of enjoyment for me, even though I just play along and lurk here without submitting my monthly results. The problem, from my lurker's perspective, is that the single player game is not designed for a side-by-side comparative competition format. The software doesn't support it, and the game mechanics don't either. Yes, we all play the same maps, and yes the better players and better strategies will always perform better, but that doesn't mean that we're all on equal footing each month.

Ignore the RNG's role in things like battles and suicide galleys, and just think about things like the first 20 moves by each AI. Without the ability to script AI moves to the letter, we end up playing against slightly different opponents even though the maps are the same. Which direction they explore, who they meet and when they meet them, has a large impact on the game at higher levels.

Anyways, I enjoy this place and the GOTM because of the strategy discussions and the community of great players. Who gets the best score means little to me. If you want to have a true competition then why not do a multiplayer ladder tournament? Human against human seems to be a far better test of skill than to see who can exploit the AI weaknesses the best.

 

[denyd]

AlanH,

I can think of someone who fits your requirements: Matrix

However, I also believe in the current staff and trust there judgement in matters where they have the facts and I don't.

It's also pretty obvious that this could have been avoided if the offending party had just changed their methods. This wasn't a one shot offense and they had ample warning (so we are informed), so why continue with those tactics when you've been already been caught.

That's enough time and thought...back to the task of purging the unbelievers from 'my' continent!!!

 

[Kaiser_Berger]

This is a truly unfortunate thing, indeed. I guess it can all be related back to civ, though: no matter how productive, populous, and full of Wonders you make your capital, you're still going to probably have that one little shield of corruption sneak in somehow.

 

[Aeson]

So where would we find an impartial third party, who also knows enough about GOTM and Civ3 to be able to judge such a case? The attributes required would only seem to fit an experienced Civ3 player who enjoys the respect of the GOTM community, but who hasn't competed for a while, for whatever reason. Do they exist and would they be prepared to act as ombudsman in such a case?

Cracker fits. The assumption here seems to be that for some reason the GOTM staff as a whole would want to ban anyone, which is as far from the truth as you can get. Of course we have a vested interest in running the GOTM, which in court of law would disqualify us as a judge.

This is not a public court though. It is a privately owned forum, with rules and regulations set down for participation. Thunderfall has given us the ability to administer and oversee the competition, and we try to do so to the best of our abilities. Cracker was chosen to be the administrative head, and his decision is final in these matters as long as Thunderfall wishes it to be administered that way.

The GOTM Staff's loyalty is to the GOTM though. To try and help make sure it's the most enjoyable competition/community it can be. We are just players volunteering to do some of the less fun tasks that need to be done. There really isn't any other reason to be involved in the Staff (and plenty of reasons not to be involved in the Staff). No pay at all...

---------------------

This type of action is far and away the least fun of all. Much better to sort through 60k entries in a spreadsheet than to have to deal with the subject of cheating IMO. If one of the Staff brought up such a thing just for the heck of it... they'd probably get shot (eh... or the digital equivalent of it). Even if one of us had a vendetta against a player, there are 19 (?) people with access to the GOTM Staff forum (and CivFanatics Staff forum where this subject was also brought up), several of whom aren't involved in the GOTM or even Civ3 at all. A 'retribution' or unsupported decision just wouldn't stand without a mass falling out (which would definitely end up public) between us.

Yes, you'll just have to trust us on that. We have access to discussions and information that aren't public, and aren't made public to protect the players of the community in general. Until every GOTM participant is willing to spend 10-15 hours a month going over these discussions, looking at saves, looking at replays, corresponding with the player(s) in question (and try to do so as impartially as possible) it would just lead to far too many misunderstandings and false accusations.

Most of the players here would be able to handle it, but as shown in previous public hearings on similar matters, some just point fingers and call names (and do worse in PM's). In short, it would lead to a dying GOTM community where even the people who enjoy the competition and the interaction with the other players, would get sickened by the constant discussions on who's cheating or not. It's the reason Matrix resigned and no longer plays, and it's probably no coincidence that GOTM participation in general took a nose dive when the first public discussions on who's cheating and who isn't were going on.

It's not the point of the GOTM to have to worry about those things IMO. I really wish it wasn't part of the Staff's responsibility either.

 

[samildanach]

It appears from the details Cracker gave about the investigation that the individual caught cheating would have been able to get away with it if they had not been a regular participant in the GOTM. This is where I think peoples performance in the global ranking as opposed to their performance in individual GoTMs should be seen as the standard of excellence for the GoTM.
So say when the results of a GoTM come out, rather than focus on the scores for that GotM instead focus on the changes at the top of the global rankings incurred by the new results.
It will still be possible for someone to cheat on an individual GoTM and then leave thinking that they have conned us. But if they want the ultimate accolade of being number 1 on the Global ranking then they are going to have to hang around and risk being caught by Cracker and the team.
Cracker has already stated in another thread that he will be emphassing the Global rankings more in future, perhaps he is thinking along these lines.

Well thats my 2 cents.. Oh and congratulations moonsinger on being ranked No.1 but look down towards No.107 he's
closing fast (actually I think you might have to look on the next page )

 

[Bam-Bam]

I have read through all of this, and as a newcomer to GOTM, I miss some of the details and history of all that has transpired here, so forgive me if my comments reflect too much ignorance of the GOTM legacy, and forgive me if I am venturing too far off the topic of the thread.

I did not see what TC3 pointed out in earlier discussions--which I think is an absolutely essential point to keep in mind when we all look at the "standings" and this competition. Having been in a few comparison games--there are immense differences in game results that come about from things outside of the players' hands. Different AIs becoming dominent because one roll of the RNG gave them a leader--which tipped the balance in an AI-vs-AI war, or games where the number of leaders generated in an always war game goes from an extreme of about five leaders for the first 50 or so elite victories to ONE for 55 elite victories (52 victories for attacking units).

Yes, the top players generally produce the top scores. There have been numerous praises and questions about the scoring system for GOTM--I do not want to discredit the enormous amount of work that has gone into improving the comparative metric used for GOTM games. Is it perfect. No. Is it a fair comparative tool--mostly. What is misses is how much games are affected by how things change, and it still carries a whole lot of the high-population--lots of territory--happy people paradigm from Civ basic scoring. So be it. A medium like this needs some metric, so it has one.

"What is he getting at?", you might ask. Ok--what I have seen, and I strongly encourage, is increased emphasis on the details of what happens during the game. I think the discussion forums (spoiler threads) are great for this purpose--put the focus on the game, not on the end results. I care more about how players made their way through the games--i.e., their stories, their legends and epics--rather than how many points they have gotten. I think the community recognizes this--the QSC--even though it is about the numbers, does put the detail to the game history--not just the end. There are many instances of "variantish" type play and storytelling getting a whole lot of support and praise in these games, even if the games were nowhere near the medals. To me, that level of participation and interest in the "mighty struggles" is more indicative of the success of the GOTM than sheer numbers or numbers of winners.

Yeah--everyone has opinions. I have very strong ones about avoiding spoiler information for my games--though I am participating, I will not discuss the game with others until I am complete, even though I hope to post reports. This is my choice for my game to be judged on what I do in the game without specific assistance. That may or may not be the norm for most players, and I have no particular feelings about that. It does not make me a better or worse player or person--just is a choice I make. Whenever you make comparisons or have competitions--someone may not fare as well in the competition or comparison. And some may value the false glory of high score enough to take shortcuts that cheapens the comparison. I applaud the effort the staff takes to make sure the comparisons stay relatively clean. I am sure it is a whole lot of work.

Last point--I am hoping, when I finish this, to more of the depth of the tales from others' games like I have seen in the more recent games. Second to the pure enjoyment of the game itself, I find the telling and retelling of our wartales at the proverbial pub to make all of this worthwhile. Sorry for the long post--and potential threadjack, but TC3's post hit a nerve.

Bam-Bam

 

[whb]

What do you think of this as a possible easy improvement to the game to help prevent cheats:
(Feel free to pass this idea on to Firaxis, but do credit me as the source -- unless of course someone else had this idea before it occurred to me)

Internal RNG player-roll likelihood checking:

At the moment, the RNG preserves its seed with the idea that a save-and-reload cannot alter the outcome. This fails because the player can change his behaviour slightly to "use up" a bad roll on an inconsequential action, and then get a new roll on the action that really mattered.

A better idea might be to deliberately throw away the seed (thus allowing save-and-reload to change the roll) -- but to record the player's (but not the AI's) dice score distribution throughout the game.

So long as "high" rolls for the player are always good, and "low" rolls are always bad, a simple likelihood check on the roll-distribution tally recorded in the .SAV should highlight chronic reloaders very easily. (Of course, this check would only be made when submitting to a GOTM or other tournament)

cheers,
whb

---
Will Billingsley
Cambridge University Computer Laboratory

 

[Renata]

@whb: Very appealing idea, but I doubt it would work, as there are just so *many* dice rolls in any given game that ten or twenty or even probably fifty manipulated ones might get lost in the noise. It also doesn't address the other main types of cheating that I'm aware of: map preknowledge/ re-playing to change strategy.

Renata

 

[Bamspeedy]

whb-the problem I see with that would be that the rolls that are saved would need to be kept track of for what the situation called for when making the roll.

If someone gets very good rolls on combats and leader generation, who cares if they get bad rolls on worker automation, diplomacy, what the advisor suggests you build/research next, etc.

And there are varying degrees of how 'bad' or 'good' a number is. Rolls for culture flips may look bad, but if the odds are so low of the city actually flipping, you would need an extremely low roll, but in a battle a bad roll, might be a much higher number.

Lets say it rolls for numbers 1-1000 (I don't want to nitpick the exact numbers, which have been studied/posted before on how the RNG works exactly). For a city with low probability of culture flipping, lets say it needs a number of 10 or lower to end up being a 'bad' roll. For combats where you have say, a 50/50 shot of winning, the number would need to be lower than 500 to be a 'bad roll'. So just getting a list of numbers wouldn't help, unless you knew what each roll was used for.

Turns can sometimes take long enough as it is, and save files can get big. I can't imagine how long turns would take and save files would be if the game had to record every roll of the RNG when there are dozens made every turn, and if you play 540 turns.

 

[JonathanValjean]

Cracker, thanks for sharing those thoughts with us about your staff in your most recent post. I know that this must be a very difficult time for you, and I want you to know that, although I will miss Ribannah, I support you and your staff. It takes courage to do what you think is right, despite the dissension that invariably rears its ugly head in such a complicated matter as this one. Knowing that you and your staff have agonized about this decision for such an extensive period of time has consoled me and has increased my respect for you and your fellow GOTM co-laborers. As if you all didn't already do enough, for absolutely no pay, but strictly on account of the altruistic goodness in your hearts.

Originally posted by cracker
In many ways I am the only weak link in the process but with their help we are working on my performance every week.

It really takes a lot of courage to make a self-deprecating statement like that, and if you are partly talking about your relations with others, I really have noticed that you have made great strides in the public relations department. Sometimes, that's not so easy to do for an engineer! (I have two uncles who are engineers for the Redstone Arsenal and for NASA, and public relations experts they are not...) (Please don't take this offensively, for I certainly did not mean it as such.) Thanks for all your efforts to keep everything together behind the scenes. The GOTM truly is the greatest event in all of Civilization, and we owe it largely to you, Cracker.

 

[whb]

@Bamspeedy

Certainly you would only record rolls for interesting events (like battles, not like Advisors or Autoworkers).

I don't think the size or speed would be limiting because I doubt you would have to store all the raw numbers -- at the simplest level, you could just store the sum-to-date of the difference between a player's raw rolls and his opponent's raw rolls in battles, along with the total number of battles, and use that as a very basic check for chronic battle-outcome reloading.
Similarly, you could produce a score for the suicide galley survival-at-sea scores. There won't be enough data points in the one game, but when summed across a player's multiple entries in the GOTM, things get statistically significant very quickly.

I am sure there are more complex and more useful algorithms that could be applied too. One could also conceive of building a map-square scoring system into the game itself, to automatically evaluate the value of opening exploring moves against expected best-play. (Although Cracker seems to be doing a fine job of that one without in-game help).

Simply put, keying these metrics into the game itself would lessen the load on the mods, and make the evidence for cheating harder to refute.

There would be workarounds (eg: have more meaningless battles to cover up the cheating meaningful ones) but I still think it might be an interesting avenue for Firaxis to pursue.

whb

 

[samildanach]

Originally posted by whb

Internal RNG player-roll likelihood checking:

Firaxis may be interested implementing something like this for the PBEM community. I don't think they would do it for single player though. What would be the point? Does Firaxis even know the GoTM exists?