Debugging loading errors/CTDs

[avain]

Hi all!

I'm writing here in a hope that I can gather some useful ideas as to how to debug a CTD that happens during loading a saved game.

The situation is the following:
- I'm doing a massive modpack that involves a lot of SDK and Python changes (Quot Capita mod)
- When starting a new game it can be played, loaded, etc. without problems.
- When exiting and reentering Civ, and trying to load a saved game one of the following happens:
a) It loads ok. I can load it several times afterwards, no problem.
b) It loads ok. After loading, I try to load it again, there is a CTD

I didn't get a Python exception.

Can this be a memory leak?
Did you have similar problems?
How can this one be traced and debugged?

Thanks for helping.

[Kael]

I would ping TAfirehawk about it. He leads the Visa mod and they have doen a ton of working getting around the MAF (memory allocation failure) errors with large saves. As I recall that was a crash during saves, not during loads but I may have that wrong.

You don't mention if you are on BtS or not. If your not I would highly reccomend you move to the BtS codebase to help with this issue (they rewrote the save process).

A few questions:

1. Do you sometimes get a crash and sometimes get a good save off of the same save game?

2. Do you get crashes off of tiny save games (say a new map at duel size)?

3. What is the filesize of save games that cause crashes when they load?

4. How much memory does your computer have?

5. What Civ version / patch level are you using?

[avain]

Originally Posted by Kael

I would ping TAfirehawk about it. He leads the Visa mod and they have doen a ton of working getting around the MAF (memory allocation failure) errors with large saves. As I recall that was a crash during saves, not during loads but I may have that wrong.

You don't mention if you are on BtS or not. If your not I would highly reccomend you move to the BtS codebase to help with this issue (they rewrote the save process).

A few questions:

1. Do you sometimes get a crash and sometimes get a good save off of the same save game?

2. Do you get crashes off of tiny save games (say a new map at duel size)?

3. What is the filesize of save games that cause crashes when they load?

4. How much memory does your computer have?

5. What Civ version / patch level are you using?

First, Kael, I'm really happy you responded, thanks once more!

Yes, I'm on BtS, 3.13. Also, I'm aware of the MAF problem with large saves, but that's not the case here (although I'd like to know a solution to that issue, too - but it was, at least for me, not related to the loading-saving process)

1. Exactly, that's the most frustrating. It seems the saves are 'OK', they just sometimes fail to load. As if Civ itself is in an OK state to load multiple times the savegame, or to break after the first succesful load -> reload.

2. Yes, it is independent of map size, save size.

3. Any or not relevant.

4. 2 Gb. (Vista)

5. BtS 3.13

Also, I had a chance to test the mod on a different machine with 1Gb, it produces the same symphtoms. Also, it was an XP and I got civ crash report after exiting civ (so not via CTD, but normally). Not so in Vista.

[Seven05]

What compiler are you using? If you're using Visual Studio (2003 or 2005 express) and you enable the crash dump in your civ4 ini file you can open up the dump file in visual studio. Unfortunately you won't have the symbols for any of the files except the GameCoreDLL but you'll be able to see if the most recent calls were in your DLL and if so which method was running.

My experience with failed load games have all been related to XML changes or not doing a clean build of the DLL after changing header files (especially enums). However, in those cases it failed to load any game and was always after I made changes.

[avain]

Originally Posted by Seven05

What compiler are you using? If you're using Visual Studio (2003 or 2005 express) and you enable the crash dump in your civ4 ini file you can open up the dump file in visual studio. Unfortunately you won't have the symbols for any of the files except the GameCoreDLL but you'll be able to see if the most recent calls were in your DLL and if so which method was running.

My experience with failed load games have all been related to XML changes or not doing a clean build of the DLL after changing header files (especially enums). However, in those cases it failed to load any game and was always after I made changes.

Thanks, I might try something along those lines.

The only thing that discourages me is that the thing WORKS most of the time, but then it suddenly fails. I mean, I start Civ say 10 times, and try to load exactly the same save. The first load is ALWAYS ok, the second either fails, or it's ok for the time being.
What can I do when it works 50% of the time (same save), and it concerns loading?...

I just found a better wording for the error: Loading from the Main Screen is 100% ok, loading the same from the loaded game fails 50% of the times (but if it's ok, I can load it seemingly infinite times again and again).

PS: I really appreciate all your help!

[Kael]

Intermittant problems are the worst. I really can't think about what would be different between 2 saves of the same game that would cause occasional crashes during load.

[avain]

Well, I tried with Visual Studio:
- resource sharing violation, at seemingly different places
- also, not surprisingly, a null pointer afterwards...
- and of course, cannot trace it really, because it ends in the exe...

What I've seen thou, is that most of the times it's related to deleting/freeing up arrays, might be coincidence thou...

[Seven05]

When debugging you should also look several steps back (at least three) to get an idea of what started the chain reaction. I had a consistent game crash on turn 3 using newely created gamespeeds, looking at the dump file it was pointing to errors in the exe but about 4 or 5 calls back there was a reference to the interface and when I looked into that it was the turn number/calendar display and it turns out my fatal CTD was caused by using 'non-standard' values for the number of months per turn (like 9, 13, 22)... fancy that, an XML error

It will normally be a wild goose chase, followed by a forehead slapping 'duh' moment of enlightenment.

Now, if it is an error with your arrays have you added any? And if so do you create & destroy them in the same methods as the original game uses for their arrays? Or, have you altered the way objects are created, initialized, reset or destroyed? Or perhaps you've added in code that is interfering with the default destructor, or overriding it, or terminating at the wrong time and preventing the destructor from being called (or finishing) thus triggering the crash. Could be pointers too, Civ4 is riddled with pointers and misusing them can be hard to trace since you're likely pointing to data it just isn't the right data.

[avain]

Now at least I have a consistent error on a 2nd load attempt.
The debug points at an isBusy call (I'm not at home, so cannot check now) in I guess CvUnit.cpp, which fails I think because the unit's <this> is a null pointer(?).
Also, I checked the call stack back as long as I could, one strange thing is that there's a malloc call on I guess CvGame(?), and the size variable is a really big one (like 1 222 222 222), also, I've seen a read call have the stream pointer altered after the call (if the sequence is correct), prior to calling it seems like ok, after it it's like 0ffffffffff.

I believe I'm using arrays in civ fashion, but the next step will be that I'll try disabling the custom writes and reads to the savegame, and check if the error returns...

I cannot rule out XML errors, since this error seems to happen in the late(r) game saves, just started games load ok 2 times in a sequence...