Multi-Player Bugs and Crashes, v40 & SVN after Oct 2019

Thunderbrd

C2C War Dog
Joined
Jan 2, 2010
Messages
29,801
Location
Las Vegas
Report any OOS errors or issues with Multi-Player simultaneous games here.

You CAN play Simultaneous Multi-player as a single player and get a semi-real time turn based game that processes turns a bit faster since the AI is moving when you do. We can also try to debug those as well.

If you are trying to play actual multi-player, the IP connection is the most valid but you're likely to hit some OOS errors. To minimize these,

1) Turn off events in the game options when you set things up - yes this negates flammability but you'll have lots of OOS errors all game if you don't.
2) Make sure Rev is off. This tends to cause some as well.
3) Make sure to try to avoid building any wonders other players are building, even if it lets you, especially if you are on the same team.
4) Avoid automating anything at all, including sending units beyond the reach of their movement for a turn. For some reason, this is a known issue.

Beyond that, when you DO get an OOS error (which sometimes happens in the first round even) you should:
1) report the autosave if you have one from the immediate round before the OOS takes place and you find you can repeatedly hit the same OOS error that round.

2) report the logs you get - you should have logging turned on for all systems through the config ini settings.

and more importantly:

3) save the game AFTER the OOS has taken place, have everyone get out entirely to desktop, all reload and reinitialize the game. If you try to shortcut this to get back on track, you will doom yourself to immediate OOS errors again.

If you're having a LOT of OOS errors, please post here to let me know to look into it. A 2 player game is something I can TRY to evaluate somewhat, and just knowing the era and when things are beginning to OOS a lot can help.

We're not totally clean with this yet but my wife and I can often get a pretty good game going without hitting all that many, or at least we were recently. It can only take one bad update to make everything go very south in this department. Keep me posted on your experiences here.
 
v40

Does this apply to Hotseat sessions as well? i'm playing without revolution anyways but i would like to know about the other points.
I did play a short test game before with me playing as 3 different civs and beside getting an error message (something something eventhandler(switch players), dont remember exact wording) on turn transition, which didnt seem to affect anything, it was playing fine, although i cant tell about events being managed well or not.
 
v40

Does this apply to Hotseat sessions as well? i'm playing without revolution anyways but i would like to know about the other points.
Not sure what you mean by "this". There's no special precautions,to consider, compared to setting up a single player game, when setting up a hotseat game like it is when setting up a networked game. Option choices doesn't matter in hotseat afaik.

Hotseat has more in common with single player than with networked games, as it's not a network game but completely local to a single game instance on a single computer.
e.g. You can't get OOS (Out of Sync) errors in hotseat.
I did play a short test game before with me playing as 3 different civs and beside getting an error message (something something eventhandler(switch players), dont remember exact wording) on turn transition, which didnt seem to affect anything, it was playing fine, although i cant tell about events being managed well or not.
Knowing the error message would help.

Turn on python error logging by changing some lines in ...\Documents\My Games\Beyond the Sword\CivilizationIV.ini
LoggingEnabled = 1​
Logs are written in ...\Documents\My Games\Beyond the Sword\Logs\PythonErr.log
Upload that log if it contains any lines after playing a game.

To get the error messages that would be in those logs as popups in-game:
HidePythonExceptions = 0​
in case you prefer to upload screenshot instead.
 
Last edited:
That may be true but i am wondering about things getting buggy in general. For instance in my test game i occasionally had one of my own units become invisible to myself. I was still able to select it and give orders and it wasnt really an issue but it is still worth mentioning.
I'm guessing that in similar fashion there will be instances of one player being able to see invisible units of another player.
 
For instance in my test game i occasionally had one of my own units become invisible to myself.
That happens in single player too, the flag is there but the unit model fails to render.
The graphical paging could be out of whack and hide stuff that should be visible too so that the flag is also missing. The graphical paging has been worked on lately, made more aggressive in unloading stuff to save RAM, so I won't say for sure that it is working perfectly in all cases, it depends on FoV setting and zoom level and such to decide what is inside and outside your current camera view.
I'm guessing that in similar fashion there will be instances of one player being able to see invisible units of another player.
This would be a bad bug compared to the glitchy model rendering that you described above.
If it could happen in hotseat then it could most likely happen in single player too.

▐▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▌

I'm not saying there can't be any hotseat specific bugs, just saying that we rarely hear of them, and that they are usually easy to fix the few times they are discovered, usually just a small oversight in code involved in the player switch.
I don't think any such hotseat specific bugs are currently known to the C2C community.
 
Last edited:
v40

Does this apply to Hotseat sessions as well? i'm playing without revolution anyways but i would like to know about the other points.
I did play a short test game before with me playing as 3 different civs and beside getting an error message (something something eventhandler(switch players), dont remember exact wording) on turn transition, which didnt seem to affect anything, it was playing fine, although i cant tell about events being managed well or not.
This sounds like a hotseat-specific bug, this is the right thread to report it, and it is probably easy to fix, but a full error message (at the least) is needed.
 
This sounds like a hotseat-specific bug, this is the right thread to report it, and it is probably easy to fix, but a full error message (at the least) is needed.
May have been introduced with the PPIO modmomd that is now integrated in C2C.
I rewrote most of the python code there, but I never tested my changes in hotseat, so I wouldn't be surprised if I overlooked something related to the player switch.

Would need to see the error messages to fix it though. I might run some turns in hotseat today to see if I can get the error myself.
 
I did play a short test game before with me playing as 3 different civs and beside getting an error message (something something eventhandler(switch players), dont remember exact wording) on turn transition, which didnt seem to affect anything, it was playing fine, although i cant tell about events being managed well or not.
Would need to see the error messages to fix it though. I might run some turns in hotseat today to see if I can get the error myself.
Found and fixed the error, so you don't need to provide logs. Fix is currently only on git, but will be on SVN next time it's updated.
It was pretty inconsequential, only impact it would have on the game would be that if the previous player had the strategy overlay (ctrl/alt+X) visible when ending turn, it would remain visible for the next player in the hotseat revealing the city planning of the previous player.
 
Hi. C2C 40.1.2529, direct IP connection. OOS occured when 2nd player attacked an animal. Can not reproduce after loading previous autosave.
Logs are attached.
 

Attachments

  • Beyond the Sword.7z
    3.9 MB · Views: 82
  • Logs1.7z
    577.2 KB · Views: 75
Hi. C2C 40.1.2529, direct IP connection. OOS occured when 2nd player attacked an animal. Can not reproduce after loading previous autosave.
Logs are attached.
There are some known possible OOS errors that are not directly replicateable in combat and thus have not been easily found and fixed. They should be rare. As usual with OOS errors, if they don't absolutely replicate, do your best to save reload and carry on. I'll be reviewing the combat system in some depth very soon and may find I've corrected the issue in the process but at a point after I've completed the current project I will probably start working again on OOS hunting efforts.
 
I'll be reviewing the combat system in some depth very soon and may find I've corrected the issue in the process but at a point after I've completed the current project I will probably start working again on OOS hunting efforts.
Thank you) Do you need any other OOS logs?
 
Thank you) Do you need any other OOS logs?
We usually need the OOS log of all the human players in the MP.
We then can compare this "deathclaw - Player 0 - Turn 167 OOSLog.txt" with the same log from the other players and see what value is different for the two.
If two players have different values for the same thing, like how much HP a specific unit has after combat, then that will point out at least what value is out of sync between the players. Debugging OOS is hard though, but these logs are a starting point for such debugging.
 
Some OOS logs from both players:
 

Attachments

  • Logs - player0.7z
    598.3 KB · Views: 55
  • Logs player0 001.7z
    598.4 KB · Views: 83
  • Logs player0 002.7z
    597.3 KB · Views: 66
  • Player 1.7z
    15.3 MB · Views: 76
Some OOS logs from both players:
Are you playing with simultaneous turns?
OOS on turn 995 and 1009 both involve the same spiked clubman owned by player 1 and combat with animals.
The spiked clubman is standing on different plots for the two players and an animal has been killed by the clubman for one player but not for the other.

My initial guess is that combat is giving different outcomes;
• or that hide and seek rules are not applied the same for both players, so that the animal may have been hidden for one but not for the other so only one of the players would register the attack while the other just register a move.
This is something to remember the next time I, or TB, or anyone else, look over the relevant code.

The oos on turn 975 is impossible to interpret anything from the logs as almost everything is different between the two players game state. Too many differences to pinpoint where the divergence began.
Log difference from turn 995 is the cleanest and most informative one in this set.

Having the save on the start of the turn where the OOS happened in, along with the OOS logs, could help in further debugging.

I'll attach a new zip where I deleted all logs that are of no use. The random logger can be of use if we have one from both players, but there was only from one in here so I deleted them too.
 

Attachments

  • Three OOS log sets from GiantLittleDog.7z
    434.1 KB · Views: 58
Last edited:
Are you playing with simultaneous turns?
OOS on turn 995 and 1009 both involve the same spiked clubman owned by player 1 and combat with animals.
The spiked clubman is standing on different plots for the two players and an animal has been killed by the clubman for one player but not for the other.

My initial guess is that combat is giving different outcomes;
• or that hide and seek rules are not applied the same for both players, so that the animal may have been hidden for one but not for the other so only one of the players would register the attack while the other just register a move.
This is something to remember the next time I, or TB, or anyone else, look over the relevant code.

The oos on turn 975 is impossible to interpret anything from the logs as almost everything is different between the two players game state. Too many differences to pinpoint where the divergence began.
Log difference from turn 995 is the cleanest and most informative one in this set.

Having the save on the start of the turn where the OOS happened in, along with the OOS logs, could help in further debugging.

I'll attach a new zip where I deleted all logs that are of no use. The random logger can be of use if we have one from both players, but there was only from one in here so I deleted them too.
There could still be an issue in withdrawal somehow.

Thing is a combat can appear to be the first sign of an OOS if the random generator is already out of synch when the combat takes place. So it could also be anywhere in the AI. As noted, the random logs would help to narrow in on the first out of synch random the game experienced and that gives a good starting place to build direct logs reporting every single change to a variable working back from that moment and when you find THAT log went out of sync, you find exactly where the problem is, usually a variable without a default established at declaration being sent through a pointer reference parameter. Or something equally as random.
 
There could still be an issue in withdrawal somehow.

Thing is a combat can appear to be the first sign of an OOS if the random generator is already out of synch when the combat takes place. So it could also be anywhere in the AI. As noted, the random logs would help to narrow in on the first out of synch random the game experienced and that gives a good starting place to build direct logs reporting every single change to a variable working back from that moment and when you find THAT log went out of sync, you find exactly where the problem is, usually a variable without a default established at declaration being sent through a pointer reference parameter. Or something equally as random.
True, something else could have gone out of sync before those combats started going out of sync, something that we haven't or can't included in that OOS log.
Simultanious turns may also muck things up, so that the animal has moved away before the attack on one computer but havent on the other yet. Simultanious turns seems to me scary when it comes to OOS issues, though I don't know how it's coded, so it's just a feeling more than anything.
Should I enable C2C logging to provide these logs?
It can't hurt, I think the random logger files can be useful.
Debugging OOS is difficult, one would have to run the game on at least two computers while debugging it to get the correct conditions for OOS to happen. I only have one computer so I won't be able to help out much on it, unless I and TB cooperate on it, but I would think it easier for him to just do that with his wife rather than with me. ^^
TB is also swamped with other big modding projects in C2C, so I doubt we will be able to fix these issues anytime soon. But having saves and logs archived here will allow us to use it at a later stage.
 
Simultanious turns may also muck things up, so that the animal has moved away before the attack on one computer but havent on the other yet. Simultanious turns seems to me scary when it comes to OOS issues, though I don't know how it's coded, so it's just a feeling more than anything.
Simultaneous turns is the only reason OOS errors can happen actually - synchronization is entirely for the sake of managing simultaneous turns. Everything in processing during simultaneous turns must take place in exactly the same order on both computers UNLESS it is a purely UI interaction - but even then what happens during that interaction must go to the message system that keeps the results of such an interaction taking place at the same time on all systems. In short, an animal cannot have moved away on one computer and not on another when an event happens because the events must happen on all systems in the same exact order.

Debugging OOS is difficult, one would have to run the game on at least two computers while debugging it to get the correct conditions for OOS to happen. I only have one computer so I won't be able to help out much on it, unless I and TB cooperate on it, but I would think it easier for him to just do that with his wife rather than with me. ^^
TB is also swamped with other big modding projects in C2C, so I doubt we will be able to fix these issues anytime soon. But having saves and logs archived here will allow us to use it at a later stage.
Yeah it's tough. VERY tough. Once I get the next big project(s) complete I'll want to play soon thereafter so will HAVE to work on OOS resolutions for a while to do that. And yeah it drives my wife nuts the loading and reloading necessary to debug these things but it should be easier now that the game loads so much faster than it used to. I'll have to remind myself how to create the logging that I'll need for hunting down issues though. Been years now since I last set those up and I have a feeling what remnants I left some modder probably felt the need to clean up since they were turned off and probably looked pretty random and unnecessary.
 
Simultaneous turns is the only reason OOS errors can happen actually
Well, I got plenty of OOS errors without simultaneous turns when I used to play MP with a friend. So they can obviously happen when not using it.
I can't stand simultaneous turns, to think that my victim may move away just before I release the right mouse button to attack it so that I waste my units turn by making a move I didn't really want to do goes against so much of what I like about a TBS game.

Other annoyances is when enemy units shows up late in a turn after I've moved anyone who could have reacted to it that turn. Or that I don't notice a criminal unit sneaking out of one of my cities late in the turn long after I've already concluded that there are no criminal units around my cities to deal with. One has to always reevaluate the state of the turn in case something changed in an area you already had evaluated, and hitting end turn is something you don't want to do long before the other human players in a simultaneous MP game.
 
Last edited:
Top Bottom