Some OOS findings

[a_civilian]

Edit: If you're playing multiplayer with a lot of AI's, please try the small modification Sephi posted below and let us know if you still encounter any OOS. It doesn't change any behavior; it just adds a small error handling check.

Original post:

I've logged 7 OOS (out of sync) instances from 4 different multiplayer games (each with 3 players and 9 AI, running 0.41m) and each error was essentially the same: a particular AI civilization decided to build one item in one player's game and a different item in the other player's game.

(One remark is that in some cases the "Soren rand value" also differed, which resulted in different unit actions, but there was always exactly one build mismatch. This indicates that the random number mismatch was almost certainly a result of the build mismatch.)

I posted this result a few weeks ago in another thread. However, just recently I decided to reanalyze my results. In particular, I looked up the offending civilization.

Here is my list.
- (Game A) Elohim: warrior versus library
- (Game A) Sidar: work boat versus longbowman (rand mismatch)
- (Game A) Sidar: temple of Kilmorph and hunting lodge versus granary (and gold)
- (Game B) Ljosalfar: warrior versus archery range
- (Game C) Elohim: pagan temple versus library
- (Game D) Ljosalfar: warrior versus archery range
- (Game D) Elohim: monument and training yard versus pagan temple (rand mismatch)

Very suspicious. Only three civilizations out of the nineteen possible, each multiple times, and most across multiple games.

Those civilizations were not particularly overrepresented over these four games either. Here is the distribution of AI civilizations across those four games: Khazad 4, Elohim 4, Luchuirp 3, Amurites 3, Ljosalfar 3, Doviello 3, Svartalfar 2, Kuriotates 2, Lanun 2, Sheaim 1, Clan 1, Illians 1, Grigori 1, Calabim 1, Bannor 1, Malakim 1, Hippus 1, Sidar 1, Balseraphs 0. This means the Elohim, Ljosalfar, and Sidar occurred only about 40% more often than the mean (8/36 compared to 3/19).

(For the record, it also happened that the leaders in each OOS instance were Einion Logos and Thessa, but this could have easily been coincidence as those leaders were vastly overrepresented in my sample.)

My conclusion is that it is very likely that some civilization-specific code, or at least code that is strongly favored by certain civilizations, is causing these errors. I spent some time looking through the code to see if I could find anything suspicious, but not having any prior knowledge of how the AI (or anything else in Civilization) is written, I didn't have much luck. Perhaps someone more familiar with the AI code might be more fortunate.

A larger sample size would be nice, but unfortunately my group hasn't had much opportunity to play. If anyone else wishes to contribute, the instructions for producing OOS logs are in this spoiler.

Spoiler :

The first time an OOS error occurs during an instance of the FFH program, it writes much of the game state in human-readable format to OOSLog.txt in the BTS folder. You can then compare different players' logs to figure out what went wrong.

Note that FFH must have permission to write to the BTS folder, so depending on your OS security settings you may have to run it as administrator. Also, the log is only created the first time you encounter an OOS until you close and restart FFH, so make sure to restart the game after each OOS. Finally, logs are not timestamped or numbered, so they will overwrite each other if you don't move or rename them.

 

[coelocanth]

I've seen OOS in games not involving those civilisations.
e.g. a game with bannor, sheiam, infernals (AI), doviello (AI), khazad, balseraphs and amurites started to have OOS every 2-3 turns.

Also if a human player switches teams when summoning hyborem/basium, OOS will always happen.
For the other players POV, hyborem has already founded Dis, but for the player who switched, they have the stack of units containing 2 settlers and no cities.
Fortunately this OOS can be resolved by reloading the autosave and having the human player choose infernals instead of their original civ when joining.

 

[deanej]

Also if a human player switches teams when summoning hyborem/basium, OOS will always happen.
For the other players POV, hyborem has already founded Dis, but for the player who switched, they have the stack of units containing 2 settlers and no cities.
Fortunately this OOS can be resolved by reloading the autosave and having the human player choose infernals instead of their original civ when joining.

This one's been known for a very long time.

 

[a_civilian]

I've seen OOS in games not involving those civilisations.
e.g. a game with bannor, sheiam, infernals (AI), doviello (AI), khazad, balseraphs and amurites started to have OOS every 2-3 turns.

Not surprising, Bannor, Sheaim, and Balseraphs were underrepresented in my sample.

 

[Sephi]

you could try if this helps. Replace Cvgameutils.py in Python folder. make sure everyone in MP game does this

 

[a_civilian]

Thanks, I will try that.

 

[Folket]

I looked in the file. Seems like longbows/champions/arqebusier are not considered is permanent defence needed.

 

[Folket]

Reading some more I found the lines that always value more advanced units higher.

 

[coelocanth]

if gc.getInfoTypeForString(sProd)!=-1:

So you added error checking to stop trying to build something that doesn't exist in the game?

Would it make sense to log the bad string in an else clause? (so it could be searched for in the code, once identified)

ps: wow that is a huge function!

 

[a_civilian]

To clarify, the error checking is to ensure that the -1 is not later passed to an array (in particular, the array of buildings/units, see getCivilizationBuildings in CvInfos.cpp). If you take the -1 index of an array in C++, it just gives you whatever is before the array in memory, which could be anything (since this array is dynamically allocated with "new") - it could be a valid unit/building and it is likely to vary between clients.

Of course, if this fixes the issue, then as you say there is a bug elsewhere in the code because the -1 should not appear in the first place.

 

[PacoDeth]

none of this really makes sense to me, but it sounds like some good headwork! Here's to hoping for some fix in the near future!

 

[a_civilian]

none of this really makes sense to me, but it sounds like some good headwork! Here's to hoping for some fix in the near future!

Well the file Sephi posted may be a fix. It just needs to be tested.

 

[Folket]

Should that not be handled by the canConstruct/canTrain/canCreate function?

I looked at getCivilizationBuildings in CvInfos.cpp

int CvCivilizationInfo::getCivilizationBuildings(int i) const
{
FAssertMsg(i < GC.getNumBuildingClassInfos(), "Index out of bounds");
FAssertMsg(i > -1, "Index out of bounds");
return m_piCivilizationBuildings ? m_piCivilizationBuildings : -1;
}

It seems to me that it makes an assert ensuring that the index is greater then -1.

 

[a_civilian]

FAssertMsg doesn't do anything unless you've compiled with some debug flag. See FAssert.h.

 

[Boernich]

I loaded a game that we'd stopped playing (2 Humans, 11 AI) due to rampant OOS problems. We are on turn 325 and it goes OOS within 15 turns usually. We tried the new file, and the game still fell out of sync, once in 10 turns and once in 18. The only difference was that the out of sync was solid and not a flashing error, dunno if that helps.