With Patch M, Xienwolf has included support for Gerike's OoSLogger. Basically this will help us to trace any causes of OoS problems that are being reported. It won't mean that we are able to instantly solve them, but it will give us a much better idea of what is going on.

In order to do this effectively however, we need to get the relevant feedback from players. With the OoSLogger enabled, any OoS occuring in Fall Further will create a file named "OOSLog.txt" in the Civ4 directory. That file tracks various values that are used to create the sync-checksums (Sync Value and Options Value as displayed in game) and will help to narrow down which are becoming divergent.

All we need from you is;

Rename the log file to something unique - preferably the name of the host, the name of the player and the date such as "HostVehem-PlayerXienwolf-27-09-07-OOSLog.txt"
Post them to this thread.

We will need logs from both sides of the divergence (at least one of the "master" version and one from the "different" version) - in games with 2 players that means both players need to post. In games with more players, it's easier if all available logs are posted as we're sure to get the necessary ones.

===

If you would like to help further and don't mind a bit of reading - take a look at this thread (http://forums.civfanatics.com/showthread.php?t=188460) and learn a bit about what the OoS errors and checksums actually mean. The explanation there goes into far more depth than I'd ever seen and I learnt a lot reading it. If more of us understand the problem, more of us can help solve it.

Playing FF patch M with FfH2 33g I was getting OoS errors when ending the first turn. Of the displayed numbers with the OoS message: the sync numbers are the same, but the options numbers are different. I generated the OoS log files as suggested, used a diff utility on them and it reported they were equal.

Anyway, I noticed that there was no OoS (on the first turn) when no AI players were in the game. So I experimented. Any time I have the Malakim, Luchuirp, or the Ljosalfar as an AI player, I get an OoS error when ending the first turn. I do not seem to get this error with any other civs. I notice that they all have life mana granted by their palace, but am not sure of any other common factor.

Of course it all may be coincidence and something else entirely is at fault.

Reading through the thread for OOS logger, I doubt that he logs any information from the Options Checksum, based on the assumption that they simply CANNOT change during the course of the game.

int CvGame::calculateOptionsChecksum()
{
PROFILE_FUNC();

int iValue;
int iI, iJ;

iValue = 0;

for (iI = 0; iI < MAX_PLAYERS; iI++)
{
for (iJ = 0; iJ < NUM_PLAYEROPTION_TYPES; iJ++)
{
if (GET_PLAYER((PlayerTypes)iI).isOption((PlayerOptio nTypes)iJ))
{
iValue += (iI * 943097);
iValue += (iJ * 281541);
}
}
}

return iValue;
}



So the only value checked is if each player has a gameoption set on or off. Those options are:

PLAYEROPTION_ADVISOR_POPUPS,
PLAYEROPTION_ADVISOR_HELP,
PLAYEROPTION_WAIT_END_TURN,
PLAYEROPTION_MINIMIZE_POP_UPS,
PLAYEROPTION_SHOW_FRIENDLY_MOVES,
PLAYEROPTION_SHOW_ENEMY_MOVES,
PLAYEROPTION_QUICK_MOVES,
PLAYEROPTION_QUICK_ATTACK,
PLAYEROPTION_QUICK_DEFENSE,
PLAYEROPTION_STACK_ATTACK,
PLAYEROPTION_AUTO_PROMOTION,
PLAYEROPTION_START_AUTOMATED,
PLAYEROPTION_SAFE_AUTOMATION,
PLAYEROPTION_NUMPAD_HELP,
PLAYEROPTION_NO_UNIT_CYCLING,
PLAYEROPTION_NO_UNIT_RECOMMENDATIONS,
PLAYEROPTION_RIGHT_CLICK_MENU,
PLAYEROPTION_LEAVE_FORESTS,
PLAYEROPTION_MISSIONARIES_AUTOMATED,
PLAYEROPTION_MODDER_1,
PLAYEROPTION_MODDER_2,
PLAYEROPTION_MODDER_3,


And I am not entirely certain WHY those are part of the gamestate. It seems like it should not matter at all to each of the other player's computers how you have your options set. If it is set for Workers to start Automated, then the routine which assigns them a task during automation ought to send information on what task they performed.


Anyway, let us know what the values are when your options are different values. At the least it can inform us roughly which/how many options are different for the both of you. This might be tied to the same issue which is causing Clams Pop-ups to happen repeatedly for some people (also a health granting resource, like Life Mana).

Could it be caused by someone getting a "first health resource" advisor pop-up? I can't imagine that causing an OOS, but it could be related to the bug I am having with being unable to disable advisor pop-ups.

Here's some data, although I do not know what it means.

Started two games with same options and got OoS with same option checksums for both games:
Host options oos options checksum: 5081619
Options on: Wait at End of Turn; Minimize Pop-Ups.
2nd computer oos options checksum: 13569492
Options on: Advisor Pop-Ups; Sid's Tips; Wait at End of Turn.

Started another two games changing 2nd computer's options to same as host. Again the oos option checksums were the same for both games:
Host options oos checksum: 4701604
Options on: Wait at End of Turn; Minimize Pop-Ups.
2nd computer oos options checksum: 13189477
Options on: Wait at End of Turn; Minimize Pop-Ups.

All games started with Malakim, Luchuirp, and Ljosalfar as AI civs (and Elohim and Bannor as player civs). All games generated OoS at end of 1st turn.

Are the graphics, audio, and other options included in the checksum?

Reading through the thread for OOS logger, I doubt that he logs any information from the Options Checksum, based on the assumption that they simply CANNOT change during the course of the game.


It is however, a desync of that checksum that I am experiancing in most games around said time. So its safe to say something in that assumption is wrong.

Edit: I wonder if it could somehow be related to dowsing's little bug O.o... just a thought.

Looking at it again, the numbers for the checksum are simply stupid. There is one number with a large multiplier which indicates the player checked, and another number with a smaller number indicating the option value checked. But both of them are 6 digit numbers!


It does seem to make more sense in my opinion to make the player based number a 6 digit number which is simple, like 300000, and the option based number a single digit simple number, like 7. Then both are prime numbers (in their scale) and you can count on no overlap between them.

Here's some data, although I do not know what it means.

Started two games with same options and got OoS with same option checksums for both games:
Host options oos options checksum: 5081619
Options on: Wait at End of Turn; Minimize Pop-Ups.
2nd computer oos options checksum: 13569492
Options on: Advisor Pop-Ups; Sid's Tips; Wait at End of Turn.

Started another two games changing 2nd computer's options to same as host. Again the oos option checksums were the same for both games:
Host options oos checksum: 4701604
Options on: Wait at End of Turn; Minimize Pop-Ups.
2nd computer oos options checksum: 13189477
Options on: Wait at End of Turn; Minimize Pop-Ups.

All games started with Malakim, Luchuirp, and Ljosalfar as AI civs (and Elohim and Bannor as player civs). All games generated OoS at end of 1st turn.

Are the graphics, audio, and other options included in the checksum?


Crunched the numbers on this one today. First listed checksums were:


Host options oos options checksum: 5081619
Options on: Wait at End of Turn; Minimize Pop-Ups.
2nd computer oos options checksum: 13569492
Options on: Advisor Pop-Ups; Sid's Tips; Wait at End of Turn.


The options listed match the Checksum that was given by the Host. The Checksum on the second computer is off by Player 9 having Advisor Popups enabled (iI = 9 & iJ = 0)


The numbers from the second set with both computers having the same options set do the same thing, the difference in the values is exactly iI = 9.



So.... somehow, Player 2's computer thinks that Player 9 (an AI) has decided to turn on Option 0. And in #erebus Moltov informed me that it was a game with 10 slots filled, making Player 9 the last AI slot (excluding the Barbarians of course).

Ok, did some more testing with numbers as they came up while Moltov played a game. Seems like there is an overflow issue with the Feats spilling data into the options. This would also explain how some people have seen the Relationship boost from Casting Trust without ever having done so, and others have noticed that one of their options changed without them personally having asked to change it.


Testing showed that having the Shrine of Sirona (sets Feat 32 to 1 at the start of each turn) resulted in activation of the Option "Right-Click Menu" (Option 16). The option was disabled again after casting Sirona's Touch (sets Feat 32 back to 0)

Visiting Bradeline's Well (the last listed Feat in the FeatInfos File) activated Feat 39 out of 40 for me. Meaning that the game thinks there should be 1 more Feat than there actually is.

After completing all non-BtS Feats (Trust through Bradeline's Well), I saved and loaded the game. All options claimed to have been loaded properly.

Sirona's Touch was not available, so that Feat didn't load properly. I no longer had a relationship boost from the other Civilizations, so there is another down the drain. Global Spell is available as well. Of all the Site Visit Feats, I was able to redo each but the last 2, Odio's Prison and Bradeline's Well.

Counted the entries in the XML and sure enough, they come out to be 39 total.


Going to test a slight reformatting of how I have things now, and if that doesn't work then I will attempt to mimic how DamageTypeInfos is handled (both DLL list and XML setup) so that the Description tags can be kept for the FeatPrereq fields on promotions. Hopefully this is the only "big issue" broken in the code right now so we can have a clean entry into 34 compat mode :)

(oops posted w/o properly attaching its below)

So, one of the infamous Vista-XP interconnect OOS. Prehaps you can do something with it.

Me(Moltov) Hei(whicheverwaytheygo) and Kraken(ur...whatever) were all in the game.
Me and Hei had identical checksums, but Krakens was... Quite a bit off.
Anyway, the file: [edited out, wasnt actualy of this oos]

Forum upload didnt wana work. *shrugs*

Edit: Seems the logger for the OOS didnt log for me. Odd.

OOS log-ushram, rogue, me(hei) 051008

hope this helps ^^


(btw really appreciate the work you guys are putting in into smoothing out these MP issues... FfH is way too good a game for MP not to work that well =P)

(that said, O seems to be much improved over M in MP stability =D we'd try to continue this game later to see if we get into one of those chronic OOSes or its a one-off issue)

OOS log-ushram, rogue, me(hei) 051008

hope this helps ^^


(btw really appreciate the work you guys are putting in into smoothing out these MP issues... FfH is way too good a game for MP not to work that well =P)

(that said, O seems to be much improved over M in MP stability =D we'd try to continue this game later to see if we get into one of those chronic OOSes or its a one-off issue)

Here is my log from the game

Strange, both of your attachments claim to be a .rar file, but they open as a Civ4 Save File.

The file you want to attach should be:

C:\Program Files\Firaxis Games\Sid Meier's Civilization 4\OOSLog.txt

odd... the one we put was the prog files\...\civ IV\bts\OOSlog
they were too big to attach up there so we zipped it up

Strange, both of your attachments claim to be a .rar file, but they open as a Civ4 Save File.

The file you want to attach should be:

C:\Program Files\Firaxis Games\Sid Meier's Civilization 4\OOSLog.txt

odd... the one we put was the prog files\...\civ IV\bts\OOSlog
they were too big to attach up there so we zipped it up

They downloaded as rar'd text files for me - I've uploaded both into your folder on the server, Xien. Might be that CFC was getting it's attachments crossed again.

I did a little differencing on them myself btw. It strongly suggests the spawn of a Gnossling triggered the OOS unless I'm mistaken.
That said, wow, a lot of stuff OOSed in the immediate time therein...

Ok, so the breakdown of differences (read the list and tell me if anything springs to mind as being tied to actions you recently took):

Next Random Seed was off. So one of you did a random number check that the other did not.

Ushram's differences from heisenburg's:


Records 1 extra deal having been negotiated during this game

According to Ushram's computer, player 1 (so not the host, the guy in the slot right after him) has 1 extra unit, who is worth 1 Asset and 1 Power. This shows up later as being a Gnosling (UnitAI_Recon). This gnosling is at map location (8,3), which tells you nothing much if you were playing on a wrapped map.

Ushram records Player 1's Adept as having a bonus XP point, this is probably the random check. There is also a warrior belonging to player 1 on that tile, and a Barbarian Giant Spider is 30 tiles south of it, with a barbarian goblin just north of the spider. Player 2 (the third human) has a worker about 15 tiles east of these barbarians (if this helps to discover where the city with the extra gnosling and a single warrior defender is).



Ushram goes on to record that Player 3 (first AI) has a warrior 1 square east of where heisenburg thinks it should be, and that Player 5 is getting 2 less food & production total (so possible a barbarian standing on a 2/2/0 tile, or at war with that warrior and HE is standing on the 2/2/0 tile)



Ushram thinks that player 5 has an extra Hunter (UnitAI_ATTACK_CITY) (this hunter is being trained, doesn't exist on the map yet). He also records a Hunter belonging to player 5 as being 4 tiles off location, and yet another hunter as being healed more than it ought to be, and a third being out of location. Plus a Worker with an extra point of experience (further cascade effect from the random number being off)

2 of Player 5's warriors are out of location, with another one being full health and no XP, when it should be half health with 5 XP

Player 6 has a workboat out of location (2 tiles north, 1 east).

Player 7 has a wolf 1 tile east of where he should be

Player 10 is missing a unit worth 13 Asset & 5 power (a Chanter - UnitAI_Explore. Had 8 XP in proper logs, but was still level 1, so must have just won a battle). Also has a scout 2 tiles west of proper, another 2 NW of proper, and a chanter with 4 extra XP.


Player 12 has a worker missing a point of XP, and a warrior who didn't move 2 tiles west (so is healed and in the wrong spot).

Player 14 has a hunter 1 tile east of proper, and another 1 tile NE. Also a worker with 1 XP missing, another worker with 1 XP too much, and a trio of hunters each 3 tiles off proper.

Player 15 has a 2 scouts 1 tile east of proper, and a slave missing an XP point and promotion (so he has had a turn since the error occured).


Essentially every barbarian is off location by a little bit. Notable oddities other than that are that Barbatos has 1 XP and 1 Promotion less for Ushram.





Overall.. tons of stuff that is very different, I would have to suspect that quite a bit of it stems from the random number being different though, and it all cascaded from there. Major change was a Chanter belonging to player 10 died to a warrior belonging to player 5 (most likely), but that could also just be a cascade effect from the RNG being off. Though I am not sure how Player 10 had a chanter gain some XP, unless the one which died happened to have Spirit Guide, since it is precisely half as much XP as the dead guy had.

Thats great analysis xienwolf, and exactly what we need to try to draw a pattern into what may be causing these.

I looked through the function that grants spellcaster xp and I cant see anything fishy in it or the random roll. Maybe it was in a roll before that?

ps anyone else find it funny that we are unable to keep states aligned in games where one of the people is named Heisenberg? No, just me? </nerd>

lol. I wonder how many people will get the joke....

There's no way to be certain...

lol :lol: fine fine i'll go change my nick :sad:

how does.... "Schrodinger's cat" sound?



well jokes aside, XD
ushram did say that the OOS error popped out just when that gnosling popped out of the gate, and he was the guy who was OOS from the two of us.
we are unsure if that's somehow caused by nate using lolvista and rogue being on windows server (essentially xp) and I on xp.
From what we read from previous OOS threads, the OS might have an (placebo?) effect on the OOS rate.
Nate's gonna go install xp (hopefully soon =P) on his partition and we'll see if it kinda helps...
thanks again for the work ^^

btw Kael-san if you'll add an OOS logger into 0.34 as well I'm sure there's quite a few of us here who would gladly test it out in MP for you as well =D

There's no way to be certain...

That made me laugh more than the actual joke. :)

btw Kael-san if you'll add an OOS logger into 0.34 as well I'm sure there's quite a few of us here who would gladly test it out in MP for you as well =D

Yeah, thats probably a pretty good idea but I dont know about including it in a major release. I'd probably put out a one off version to multi-players for testing. We will see how 0.34 goes and what we need to do.

I checked through the FF code and the code that summons Gnosling's has some Feat checks in it. If there is a feat overflow issue like Xienwolf was mentioning it could effect Gnosling summons.

Feats were all cleaned up (ideally). So I am guessing for now that isn't the issue. But the Khadi gate code could have a decent shot at containing some bad function calls, like a sorenrand buried inside an ActivePlayer if statement. Ideally not, but I haven't checked it out quite yet.


Really wished that we'd get an OOS log that had a single value out of alignment instead of a couple dozen ;) But oh well. I'll check out the gnosling a bit since it is really the most likely cause (if combat was the issue, then we'd see a lot more problems, far more often). Another question occurs then: Was this your first gate critter?

well heres the trick xien, me and them will probly be playing together more. So you will probobly have multiple OOS logs to crossrefernace and narrow it down within the week(The logs within the week, you can take your time narrowing it down... this isnt a very bad OOS, just requires a reload) XD

Edit: Also, considering the feats caused OOSes in 3 player the one time I got to see it, caused 3 different values for all 3 players. I do not belive this is remotely related to them ^.^ Atleast one could hope. Two of us WERE still syncornised.

Well, I didn't see anything suspect for OOS in the gnosling code, but I did find that it isn't counting ALL mana types for the Djinn limitation. But that is to the Khad's advantage since more mana means fewer Djinn (if I am reading it properly).

That made me laugh more than the actual joke. :)

LOL :lol:

and yes, the OOS happened right after the gnosling spwnd. Did not see anything else that popped that turn. I also remember this happening once before. ( and earlier that game if i remember correctly)

Me and two friends (all Windows XP) continued an older multiplayer save game (patch m) yesterday and ran into an repeating OOS after some turns. It was always player 1 going out of sync.

Just to be clear, Player 1, as in Not the host (Player 0), but the guy who joined right after him?

EDIT: Nice, you named the OOS logs Player 0, 1 & 7. Clears that question right up. Oddly enough, Player 0 & 7 disagree on the location of one of player 0s workers. Player 7 agrees with Player 1 as to where it should be.

Tons of differences again, but this time even the Map random number sequence is off. But that could possibly be cascaded from a different basic random and such a long time between occurances.

Were they playing broken for a while with this? There are a ton of differences with player 1.

I saw these differences to player 1 too and I have to admit it wasn't the first OOS in this particular game. I'm pretty sure it was the third one. After the first OOS we could go on some more turns before it went OOS for a second time. The third OOS error came almost immediately after reloading the game. I'm sorry, I should have remembered earlier to collect the ooslogs from the other players, but I was too buisy with a war going on. Next time, I'll post right after the first time an OOS occcurs. I was always assuming a game is in sync again after reloading it completely, by reloading all the relevant data from the hosts save, but maybe I was wrong.

EDIT: Btw, player two was human in the beginning but died because of a huge raging barbarian onslaugt. He rejoined the game as player 7. Just to clear this up. I don't think the OOS has anything to do with this. It happened much later.

I'd have to doublecheck, but I think individual unit location cannot cause OOS (hence 1 & 7 had same checksum). Thus you can remain non-synch for quite some time without it hitting your total unitcount (which is in the checksum)

hallo everybody,
I am playing frequently with a friend of mine, we always get oos. Its frustrating and we no longer play. Xienwolf advised me to put our oos logs here, which I plan to do as soon as I have access to my pc again - 2 days at max. However, I can already now summarise our game options (in 034 h) that we always use in the 2pl games: no tech trading, no tech brokering, debug, most times agressive AI, simultaneous turns, map settings differ (lately often donut), vict cond: conquest, altar, tower (rest off).
that is basically all that is activated. We always play in a team, mostly lanum & ljosalfar.
(typically, the oos occur +- turn 170-180. we normally get adepts & priests by then and it was my suspicion so far that it is related to either young forest planting, tsunami, the way XP is granted randomly to adepts & priests, or the many upgrades with a % to disappear at end of turn that are available with adepts.
As said, the oos logs will follow in short. Many thx

lol. I wonder how many people will get the joke....

Note that by asking the question you've made people consider it more, and thus changed the answer.

Ugh... hit one OOS here after weeks of relative peace :cry:

FF 043, patch D.
I'm player 1 as Illians, Ushram was player 2 as the Sheaim.
Not sure if this is caused by the old XP/vista compat issues--i'm on XP, he's on lolVista. :lol:

The OOS logs are attached.

Occurred around turn 200ish, nothing particularly noticable, just started when we had an astrologer event, and then every turn after that even after we reloaded each turn, and even appeared again after we loaded a quite a few turns back.

If you need any other info I'll be lurking around the thread to check it, or you could usually find me (Hei) or ushram (kraken121) on the erebus channel.

think it was around turn 204 or 206 Hei :mischief:

This is the first OOS log, or one from after a few reloads and re-OOSs?

EDIT: Wow, this must be from the first OOS. The only difference is that Nate thinks that Player 9 has an extra Sons of Asena and the random seed is off.

Tired of having to guess if a unit was just built and roughly where it is, so I modified the OOS logger to also output the location, name, and a few other details about each city. So next patch should include a new OOS logger to utilize (basically, if I knew the location of Player 9's cities I would know if he just built the warrior for Nate, or just lost it for Hei.

yeah , you are correct xien, that was from the first OOS. No matter how much we reloaded it kept happening for another 10-15 turns before we gave up on the game, was quite enjoying that game too :lol: any clue what is causing the problem?

Random seed is off, so something random happened which used the number in a wrong way. You said Sheiam were involved again, so I would suspect their gates, but we checked that last time and all of the calls for a random number did seem to be done properly.

Don't suppose you could load up a save of the game and tell me who the players were (what order they show up for you to select your leader on joining the game)? The 10th listed player should be Doviello, I am curious who is before & after them.

1. Kuriotates (was played by human but was killed)
2. Illians (Hei)
3. Sheaim (Ushram)
4. Luchuirp (Dead)
5. Balseraphs
6. Calabim
7. Amurites
8. Grigori
9. Kahdi (Dead)
10. Doviello
11. Svartalfar

I don't think the planar gates did ever pop a unit out yet, Ush could confirm.
And I haven't even met the Dov yet, so I've no idea what might have caused it :lol:

Yeah, i told xien on IRC that i only had 1 gate built and it did not pop anything at the time.

Using Patch F there seems to be a constant OOS when ever a Scion player spawns an Awakened. We've tried this on patch D and it seems to be ok but on patch F it's a constant OOS error.

Do you have logs? Sounds like you have it nicely narrowed down, so we'll check out the changes in Awakened spawning mechanism, but with the logs I can see more accurately what is causing the OOS (the actual spawning, one of the random checks, the location....)

Seems like i posted this on the wrong forum. Here is the OOS log for Scions the Host is the Scions player I'm using the Archos. http://forums.civfanatics.com/attachment.php?attachmentid=197014&d=1229142459

Heres an unusual one for ya.

Turn 21.

Me, Ushram.

Erebus... With mazatl/caulli/archos/scions added to internal flavormod---renamed to prevent issues. Of course.

Chizlev
Kuriotates
Sheaim
Dural
Scions
Ljosalfar
Malakim
Elohim
Balseraphs
Archos
Austrin
Doviello
Sidar
Cualli
Mazatl

Here is my half of an OOS log from turn 21 ( a game with moltov)

Having looked over our logs...

the only difference is he fought a goblin. And on my side... he DIDNT fight a goblin(although it did still exist...)
... ... ...

Edit:

Okay. I dunno what you folks broke. But holy crap, if I need something broken. I know who to go to now XD

Every time Ushram attacks that goblin, after a load off MY save. It OOS's.

O.o

Edit2:

Holy crap. So. I loaded up the save on my computer, whent in twice(I can run 2 seperate copies of civ4 at once) It still OOS's. Somethings royally messed up here. Heres the save... Its MP but. O.o

Very interesting. I am resisting temptation to look this over and try to solve it right this moment, but I do want to mention that in the next release we will have a new MP gameoption to log all random numbers. Shouldn't help with this goblin issue by the sounds of things, but maybe.


Anyway, if you leave the option enabled, then a file will be created "My Games\Beyond the Sword\Logs\RandomLogger.log" which will list every random number generated, what the maximum could have been, and what the stated reason for requesting it was. This means that we have a nice text file to run a compare on, and when we find that single line where everything starts to go wrong, we can get information to point us at the offending function immediately.


Couldn't figure out a way to re-target the logger system to combine this with the OOS logs unfortunately, but it shouldn't cause any noticeable system slowdown. Also, the logfile will delete itself whenever you start Civilization from the desktop. So if you get an OOS and quit to desktop, save the logger before you load back up again.

sounds good xien, thanks for the quick reply :)

have to say we have not come across that issue before, was quite strange indeed.

think we loaded it up 3 or 4 times and every time i attacked the goblin it happened.

:lol: haven't you heard of that goblin shaman protecting himself with the ultimate curse causing you to OOS once you killed him?