HOF Challenge Series II Discussion

[Denniz]

Just to clarify. If I submit an Immortal game and a Settler game for the same challenge, will both games be adjusted for difficulty before my "best game" is determined? There is no doubt that the Settler game would be faster. I will not be limited to my fastest finish when scores are normalized?

That is the intent. Of course, until we figure out a equitable adjustment, only your best date will show in the results.

 

[Aerinon]

The settings state: <snip> Not difficulty as tiebreaker. Run and run fast.

Yes, but you need to read the main challenge thread. Games will be normalized by difficulty later (after data is collected by the games submitted). Likely those who are able to complete the challenge on a harder difficulty level will get a huge bump in the standings. Especially for this one. Winning on Monarch+ will be way more impressive than winning on Settler.

 

[TheMeInTeam]

What have you smoked?
Anyone goin' over settler difficulty can't hope to be first.

For this game, you just need a few chariots or even a few warriors to conquer the closest neighbours, 2 stacks to send in different directions. game over.
Research: wheel (you got it free on settler), AH, BW. Stop.

The settler AI takes forever to research BW, you can have researched well past Alpha before they arrive to it.

The settings state: Not difficulty as tiebreaker. Run and run fast.

There's a caveat. The heading for HoF challenger II claims that somehow difficulties will be "normalized", which basically means somebody is going to be screwed by perceptions of how much going up one or more levels is worth.

 

[TheMeInTeam]

Maybe its going to be a new-age feel-good "competition" in that everyone gets a medal of some kind. I mean... waiting for the submissions before deciding how to score them kind of... makes it something less than a "World Championship of Civ 4" anyhow.

I suggest one take it as a personal challenge, rather than a cut-throat competition. Score scheme matters little to me, because in my eyes unless you play all the games at the highest level submitted victory for that game, you haven't demonstrated your skills are superior to the person who does that. But it can be arranged that you get a medal anyhow.

Actually I'm thinking about actually lowering my difficulty level for this Challenge in order to see if I can learn to optimize finish times better. At emperor level I'm challenged just to win a difficult game set-up, but if I drop to Monarch I have some luxury to experiment in trying to optimize finish dates. The best thing for me is that others will be playing the same set-ups and helping me improve my game (and that wouldn't really be expected in a cut-throat competition).

I'd like to get one of the games done in Deity, maybe the culture one, or the religious one. Other than that...

Should be fun, no matter what.

New age feel good crap destroys incentive in a lot of cases (this is a major societal problem I'd not like to see on this forum). In this case, it just eliminates any pretense of competition at all unless people are literally just ranked within difficulty tiers. That's probably only going to be meaningful at a few difficulties (like settler or deity, maybe a middle one too on that conquest no razing one), doubtful more than 1 or 2 will matter unless we have 50+ players.

There is no objective way to normalize a 2000-1000 BC settler win with a 1700 AD deity win for say standard conquest. In fact, the latter is so much more skillful that it's ridiculous.

 

[shulec]

New age feel good crap destroys incentive in a lot of cases (this is a major societal problem I'd not like to see on this forum). In this case, it just eliminates any pretense of competition at all unless people are literally just ranked within difficulty tiers. That's probably only going to be meaningful at a few difficulties (like settler or deity, maybe a middle one too on that conquest no razing one), doubtful more than 1 or 2 will matter unless we have 50+ players.

There is no objective way to normalize a 2000-1000 BC settler win with a 1700 AD deity win for say standard conquest. In fact, the latter is so much more skillful that it's ridiculous.

Ultimately, two scores from separate difficulty levels will have to be compared to create a weighted score. It will have to be a subjective decision deciding which game is better. Therefore, the winner (and loser) is chosen by subjective criteria. The HoF tables, QM, EQM, Challenge I, and Gauntlets all have preset objective criteria. Challenge II is a major departure from this framework. Criteria for comparing different levels should have been set before the contest started. I find this challenge to be a major disappointment.

On the upside, it seems that there are new players participating in this challenge. If this is what was intended with the new format, you may be accomplishing that. It remains to be seen if there will be greater overall participation.

 

[AgedOne]

My own very personal approach to Challenge Series II.
Having thought about it for a while, and read what everyone has been posting here, I have come up with my approach to the new series:
I will attempt each of the challenges at the hardest level I believe I can get a result at (and that may mean trying a level just too hard for me). I'll certainly not be dropping right down to settler level to get a result. I'm trying to impress myself, and nobody else.

Secondly, when I'm looking through the results and trying to judge who I am impressed by, and who I'm in total awe of, I will start by considering that I am an average player. Having done that, I will pretty much ignore any results at much easier levels - even though they may take up all of the top positions on the tables. I will only start being interested in the results at the same levels that I tried, or thereabouts.

As has been said most eloquently on some of the posts above, there is absolutely no known way of comparing the efforts of two players, one playing at settler level and the other at deity. There is almost no chance that anyone will formulate a way of normalizing these results across widely differing scores in a way that is acceptable to most of the players.
So I think we can only gauge players relative achievements by comparing the same difficulty levels (You might be able to get a rough idea when comparing with one level up or down, but that would be difficult)

 

[ParadigmShifter]

Yeah, normalisation should have been announced beforehand.

+0.5pts/level was my preferred choice.

 

[Sun Tzu Wu]

Difficulty level Normalization:

+0.5pts/level was my preferred choice.

I would agree that +0.5 points/Difficulty level seems to be the closest that we got in the discussion, but the HOF staff never agree to this even for the Prince to Deity range.

It remains to be seen what the final Difficulty adjustment will be, but +0.5 points/Difficulty level would be a good initial guess at it. As I recall, Denniz, seemed to be somewhat supportive of +0.5 points/Difficulty level. However, I wouldn't necessarily expect that particular level of normalization. I wouldn't expect any Difficulty level normalization guidance from the HOF staff, until all Games have a good sample of different Difficulty levels submitted.

Choosing a Difficulty level that is Both Player and Game appropriate:

Players submitting Settler Games now may be disappointed with their rankings once the Difficulty normalization is applied. In my opinion, Players should submit the highest level that they are confident of Winning relatively early. The HOF tables and especially the HOF Ad-hoc Query <http://hof.civfanatics.net/civ4/adhoc_query.php?show=summ&condition[]=1&difficulty[]=9&difficulty[]=8&difficulty[]=7&difficulty[]=6&difficulty[]=5&difficulty[]=4&difficulty[]=3&difficulty[]=2&difficulty[]=1&speed[]=3&mapSize[]=5&civ=-1&maptype=-1&era=-1&exp=0&pubID=107&submit=Go> can be useful guides to what would be an appropriate Difficulty level to submit. However, this conquest Game has some particularly difficult settings, so starting with two levels lower than what one is used to might be fruitful. All the Challenge Series II Games have a Leader that is not really suited to the Victory Condition and Opponents that will not get along with each other very well, so one might submit at the Difficulty level one is most comfortable with respect to the Victory Condition or maybe one Difficulty level lower just to make the initial Victory a bit easier to achieve.

Note that all the above must be taken with the goal of achieving the earliest possible Victory at the chosen Difficulty level. One may be disappointed with a higher Difficulty level submission that has a Win Date that isn't especially early for that Difficulty level. I would frankly be surprised if the best Difficulty level for any Game in this Series is Noble or lower; I would think that Prince or higher may be required for the top spot for each Game; the question to be answered is which Difficulty level is optimal (the sweet spot) for each Game in Challenge Series II.

Sun Tzu Wu

 

[killercane]

Lets not throw settler out the window! The space race Pacal game is not going to have the wide disparity of dates that the militaristic and cultural games will have (Deity vs. Settler with no huts). There should be a difficulty adjuster for each different game.

 

[Sun Tzu Wu]

Lets not throw settler out the window! The space race Pacal game is not going to have the wide disparity of dates that the militaristic and cultural games will have (Deity vs. Settler with no huts). There has to be a difficulty adjuster for each different games

A Difficulty level Normalization specific to each Game of the Challenge Series II may be what the HOF staff has in mind. I would prefer the same Normalization would apply to all Victory Conditions equally regardless of any perceived flaws that may result. A single Normalization that applies to all Victory Conditions would be easier to understand and apply. However, I would be fine with a Normalization that differs for each Victory Condition; it could be made more accurate, but it would be complicated and not so easy to understand when considering all Victory Conditions.

In others words, I don't agree that "There has to be a difficulty adjuster for each different games". However, I do understand the desirability of it; though I'm not sure it is worth the complexity of seven different Normalizations, one for each Victory Condition (including the Time Victory which is missing from Challenge Series II).

I agree that the Normalization formula should allow the real possibility of a Settler Game being the ultimate top Game after Normalization. However, if any Difficulty level dominates the top spot with sufficient submissions at every Difficulty level, then the Normalization formula may be suspect and might require some future adjustment.

Sun Tzu Wu

 

[killercane]

A Difficulty level Normalization specific to each Game of the Challenge Series II may be what the HOF staff has in mind. I would prefer the same Normalization would apply to all Victory Conditions equally regardless of any perceived flaws that may result. A single Normalization that applies to all Victory Conditions would be easier to understand and apply. However, I would be fine with a Normalization that differs for each Victory Condition; it could be made more accurate, but it would be complicated and not so easy to understand when considering all Victory Conditions.

In others words, I don't agree that "There has to be a difficulty adjuster for each different games". However, I do understand the desirability of it; though I'm not sure it is worth the complexity of seven different Normalizations, one for each Victory Condition (including the Time Victory which is missing from Challenge Series II).

I agree that the Normalization formula should allow the real possibility of a Settler Game being the ultimate top Game after Normalization. However, if any Difficulty level dominates the top spot with sufficient submissions at every Difficulty level, then the Normalization formula may be suspect and might require some future adjustment.

Sun Tzu Wu

I dont have any strong preferences; keeping all difficulties sounds like it will lead to a Jason score type system. I hate adding extra work for the HOF staff, and would rather it be kept simple.

I justdo not want to see people denigrating the wonderful Settler level when there are some very nice games on the tables, ie Dynamic's spaceraces and Kamino's BC cultural wins. In the end, Deity and Settler are hugely different strategy games. It may help in the future to limit the range of difficulties from Prince->Deity. According to the poll in the main forum, there are only 4 people who play Warlord/Noble. It would be interesting to hear from them on the issue.

 

[cabert]

I thought there would be a range for each game.
Like the first game could be noble->deity (it's not easy even on noble) and the religious game could be emperor->deity

 

[AgedOne]

I can't see how adding points to the players' Challenge scores after the tables for each game have been ranked on finish date is going to work. Not for Game 1.
Suppose I decide to go for a win (as Sun Tzu Wu suggests, 2 levels down from my norm) at Prince.
At the end of the contest, any win I submit will certainly be outside the top 10, which will be filled by all of the settler games.
Adjusting my score of 0.5 upwards by any reasonable amount will not bring me up to anything close to the winner on settler.

Imagine, as TMIT says above, a 1000BC settler win and a 1700AD deity win. Under the suggestion of +0.5pts per difficulty level, the settler win gets 10pts for 1st place, and the deity win gets 4.5pts. Yet the deity win is hugely more impressive to all of us than any settler win.

I do agree that the effect is not so marked for other games, with other victory conditions. The main feature where a higher level AI is a benefit is, of course, tech trading. Someone slogging through the tech tree for a space colony victory at settler isn't getting so much benefit from choosing a low difficulty level.

Probably the fairest way of running this challenge would be to separate each difficulty level into a different sub-table, and then find a way of comparing them across tables at the end. But that sounds like a load of extra unwanted work for the organisers, which none of us would wish upon them.
Difficulty ranges for each game would have been a step in the right direction, but we are already under way so that can only be a thought for the future.

 

[Infantry#14]

I think a fair way of normalization is the following. Let me explain through an example.

Suppose Player A plays a game on Settler Difficulty and Player B plays a game with the same setting on Chieftain Difficulty. Also suppose that they both finish the game on turn 200. I think a fair way to distinguish the two is to subtract 10 turns on Player B's date, so that Player A's date is 200 turns and Player B's new date is 190 turns. Now suppose Player C wins with the same setting on Warlord Diff on turn 200, then his new date is 180 turns. So, for each difficulty up, the end date get subtracts by 10 turns x Number of levels above Settler.

On Quick, Epic, Marathon Speed, the modifer should be subtract 7.5, 15 and 30 turns from the final date, respectively. If one can beat the modifer, then he should play at the lower difficulty.

The proper normalization should be the following (Normal Speed)
-Settler: Add 30 turns to final date.
-Chieftain: Add 20 turns.
-Warlord: Add 10 turns.
-Noble: None.
-Prince: Subtract 10 turns.
-Monarch: Subtract 20 turns.
-Emperor: Subtract 30 turns.
-Immortal: Subtract 40 turns.
-Deity: Subtract 50 turns.

Edit: Obviously, this normalization only applies to determining the rank on the scoreboard, not the HOF table.

So, a player playing on Settler must finish faster than another player playing on Deity by 80 turns to have the same final normalized date.

On second thought, I think this normalization may bias against lower difficulty on Quick and bias against higher diff on Marathon. Also, map size may make a difference.

Edit: I suspect that there is an optimization somewhere for each different Challenge. Maybe Monarch gets the best normalized date in one game and Chieftain in another.

 

[AgedOne]

@Infantry#14
I agree in principle. The ranking order should be based on the finish turn, after it has been modified by a normalization factor.
Working out a good, well balanced, normalization factor will be difficult - but not impossible, if someone has the time. I still think we have to have different adjustment factors depending on the victory type (mainly because of the amount of tech progress required in different games).

I do realise that it isn't going to happen for this Challenge series, though.

 

[Denniz]

My thinking to-date, from pre-challenge discussion mostly, is to adjust the finishing positions for difficulty for each of the games separately. (i.e. before the points are assigned) That way only one player gets the 10 points for the #1, 9 points for #2, etc. Adjusting for difficult will be tough enough with all the other settings are the same. Adding in other VCs, Map types, speeds, etc. would make things too complex.

I was considering something like what Infantry#14 was suggesting above. Only rather than being being an arbitrary amount used across all the Challenges games, I was thinking in terms of using information from the HOF database, including actual Challenge results, to an calculate a turn spread between difficulties for those settings.

There are a number of technical difficulties to doing this. Mostly related to presentation. I didn't want to delay the start while I worked those out. Plus I wasn't sure all the games would have enough data points to start with. The change from Prince->Deity to Any difficulty to avoid an ugly change to code shared by the Gauntlets. Plus I kind of liked making the thing wide open for participation purposes.

So my advice is to play your best game on the highest difficulty you can manage. And no sandbagging.

The first change I plan to make is to change the challenge game rankings to show more than one game per player as long a they are at different difficulties. Of course, only your best game, after adjustment, will be used for rank points.

 

[AgedOne]

@Denniz
I think that what you're suggesting is a really good practical answer to the problem. I'm pleased to hear that this is something that you can actually implement during this challenge series.
Thanks

PS. I'd better stop discussing and get on and play some of these games now!

 

[Dynamic]

It looks like with such system Monarch-Immortal levels will not be profitable.

 

[TheMeInTeam]

I love it! I actually got to see the term "sandbagging" in a civ thread .

I would think that Prince or higher may be required for the top spot for each Game; the question to be answered is which Difficulty level is optimal (the sweet spot) for each Game in Challenge Series II.

I came to this conclusion almost instantly upon seeing the rules, but the sweet spot is not plainly obvious unless we know the scoring system...

but the advice

So my advice is to play your best game on the highest difficulty you can manage. And no sandbagging.

Is pretty telling also. If we're running statistical rankings based on current HoF database submissions then that changes my perception on the feasibility of normalization significantly.

 

[Sun Tzu Wu]

I think a fair way of normalization is the following. Let me explain through an example.

...

The proper normalization should be the following (Normal Speed)
-Settler: Add 30 turns to final date.
-Chieftain: Add 20 turns.
-Warlord: Add 10 turns.
-Noble: None.
-Prince: Subtract 10 turns.
-Monarch: Subtract 20 turns.
-Emperor: Subtract 30 turns.
-Immortal: Subtract 40 turns.
-Deity: Subtract 50 turns.

... <See Infantry#14 full expression of this idea in his post above.>

This is the best idea for Normalizing Difficulty level that I've seen so far.

Also great to hear Denniz is planning to implement something like this only with real Game results to support the actual turn spread per Difficulty level.

So my advice is to play your best game on the highest difficulty you can manage. And no sandbagging.

Now that is a stern warning to Players who are playing well under their Skill (Difficulty) Level. Players planning to submit Settler Difficulty who can easily Win at Prince or higher take heed of this advice. I've added my own emphasis via the DarkRed font.

@TheMeInTeam: Very well stated!

Sun Tzu Wu