HOW TO: Isolate Crashes

[talchas]

Whats odd is that logMsg shouldn't be associated with any class - it shouldn't be CvUnit::logMsg (unless you want it to be). Of course, its entirely possible that I've made it be that somehow given that my knowledge of C++ is mostly based on java knowledge + a bunch of fiddling.

When you stick logMsg in the headers, do you stick it inside the "class CvUnit {...}" section? If so, that may be why its thinking that it is defined in the object.

 

[Houman]

Actually I have figured out a way, or better to say Mexico told me ,
Delete all Logmsg() methods in all classes and set it only once in Defines.ccp and set only once its definition un Defines.h.

Thats it, now you can use it frm all classes without any problem.

Regards
Houman

 

[Jean Elcard]

Actually I have figured out a way, or better to say Mexico told me ,
Delete all Logmsg() methods in all classes and set it only once in Defines.ccp and set only once its definition un Defines.h.

Thats it, now you can use it frm all classes without any problem.

Regards
Houman

Could you please be more specific about this? Can't find the file defines.cpp. Do I have to create it first?

 

[Shqype]

. .. .. .. .. .Okay, you have finally done it. After many long hours you have the mod you always wanted to make. But then you, or someone else, try to play and it crashes. What do you do?


Isolation and Resolution

. .. .. .. .. .There are two steps to fixing problems, isolating to find out what piece of code is causing the problem and changing that code to resolve the issue. This article will only deal with isolation. Resolution is going to be specific to your code so I can’t offer general advice. The hardest part of fixing problems is isolating the issue, typically if you can do that resolution is easy.


The Changelog

. .. .. .. .. .Before we start talking through the steps it is important that you keep a changelog of the changes you are making between versions. Players will often give you a good indication of what was going on during the crash and if you have a good changelog you will be able to drill into the issue right away.

. .. .. .. .. .Changelogs become even more important as you start writing new functions. In these cases crashes may not be a problem in the function that crashes so much as in “bad” data passed to them. It is extremely frustrating to be working on a new crash in a function that hasn’t changed in forever and to have forgotten that you made a change to a related module that passed data to it.

Special note, the SDK changelog:

. .. .. .. .. .The changelog is good advice for modding, but if you are going to be making SDK changes it is mandatory. Not only is it required to help troubleshoot issues but you will need the changelog to convert all of your changes into the next SDK when Firaxis releases future patches.

. .. .. .. .. .The following is the format I use for SDK changes:



. .. .. .. .. .It’s pretty self explanatory. It contains why the change was made, the version it was checked into, the file and function that was changed and the actual code. Notice when the code is changed the original code is always left in but commented out. I never remove “real” code, it is too important as a reference.


Step 1: Get a save

. .. .. .. .. .To fix a problem you are going to need a save game right before a reproduceable crash. Load up the save and verify that the crash occurs consistently. If the crash doesn’t happen then we can’t go any further. We will talk about intermittent issues later in this article.


The Little Hammer

. .. .. .. .. .Once we have a reproducible crash you can begin to isolate exactly what is going on in the save that is triggering the crash. I usually go through the following steps:

1. Remove all players except the player and one other in the Worldbuilder. See if the crash still occurs. If the Crash still occurs go to 2, if it doesn’t go to step 3.

2. If the crash still occurs with only the player and one other, we will need to delete more stuff. Remove all players except the player and one other again, but this time remove all of the units and cities for two existing players except for an empty city each. Make sure neither of these cities is finishing production the turn the crash occurs. If the crash still occurs go to the next section on ‘The Big Hammer’. If the crash doesn’t occur go to step 3.

3. Now that we have a save game where we can stop the crashes by removing a certain portion of the environment we have only to add pieces back until the crash happens again. This can get time consuming but I typically reload and only delete half of what I did the previous time (when the crash didn’t occur) and see of the crash still doesn’t happen. If it doesn’t repeat this step but delete even less. If it does repeat this step but only delete half of what I just added back in until I narrow down on the specific unit/action/city/etc that is causing the issue.


The Big Hammer

. .. .. .. .. .The Little Hammer doesn’t always work. Sometimes you can remove just about everything in the save game and it still crashes. The Big Hammer is another method of isolation that works at the mod level instead of within the save game. It is very useful for isolating python and SDK issues.

. .. .. .. .. .As above you will need a reproducible crash to test with.

1. If you are using an SDK mod rename the ‘CvGameCoreDLL.dll’ to ‘CvGameCoreDLL old.dll’. If the crash still occurs then you can assume that the problem isn’t in the SDK modified sections. If the crash no longer occurs then you have an SDK issue. You will have to go through your changelog (you do have a changelog right!) and review the changes that have been made in your latest version.

. .. .. .. .. .I keep all of the old CvGameCoreDLL.dll files from my versions so I can walk back through the changes until I find the first CvGameCoreDLL.dll that is broken, review the changes that were checked into that one and I should have my issue isolated.

2. If you aren’t using an SDK mod or removing the CvGameCoreDLL.dll didn’t fix your problem your next suspect is python. To rule out a python issue rename your python directory to ‘python.old’ then reload your save and see if the crash still occurs.

. .. .. .. .. .If it still occurs with your CvGameCoreDLL.dll and your python directory out of your way then you have to be looking at an XML issue and you should probably go back to the Little Hammer section and see what is still around to delete.

. .. .. .. .. .Assuming renaming the python directory keeps the crash from occurring you can start adding python file back until the crash returns and then see what changed in the last python file added since the last version. You could also swap in python directories from older version of your mod to see if those have the same issue (to prove if this crash is a new issue or if it has been around before).


Intermittent Issues

. .. .. .. .. .Intermittant issues are the worst! If they are extremely intermittent then they are almost impossible to trap because they may happen on only 1 out of 20 times. You may make a change that you think will fix it and load the save 40 times without a problem and you still wont know if you actually did anything to help with the issue or if you just got lucky.

. .. .. .. .. .Fortunatly since Civ4 saves the random generator these issues are less common. Even “random” events will occur exactly as before if no one consumes a random number they didn’t in the original.

. .. .. .. .. .For example I had a reproduceable crash that occured if I fortified my unit and ended the turn. If I used the unit to attack and kill a nearby barbarian the crash wouldn’t happen. So I spent hours trying to figure out what was up with that barbarian or the attacking unit that could have anything to do with the error. Then I tried to attack another unit with one of mine, again there was no crash. All I was doing with the attack was eating up some of the random numbers so that the actual random event that caused the crash didn’t occur. The lesson I learned from this was, if you have crashes related to random functions you need to make sure your actions are identical each time you reload the save, or your results won’t be meaningful.


Practice makes Perfect

. .. .. .. .. .I hope this helps. I plan on updating this article with me information and suggestions as I come across them. But don’t wait for me, post your own tips and tricks for isolating issues with mods in this thread!

Once again, this is a very helpful guide, Kael. But what does one do when he can identify when the crash occurs, but not what causes it?

BTS TAM (upgraded from Vanilla CIV 4) will give you a crash whenever a player advances to the Imperial Age via technology. Many users have come across this when they, or an AI player, discovers a technology from the Imperial Age. TAM has only 4 eras, Ancient, Bronze, Classical, and Imperial, all others were removed. The problem with this "CTD" is that the game never crashes, it just engages in an infinite loop and you're forced to shut down the game manually (thus no crash-to-desktop). It usually occurs during the AIs turn, and players find that the game hangs while "Waiting for other civilizations."

Similarly, if you gift yourself an Imperial Age tech/techs in the worldbuilder and then click on a city, you will get this infinite loop of python errors:



The PythonErr.log shows this, repeatedly:

Traceback (most recent call last):

  File "CvScreensInterface", line 705, in forceScreenRedraw

  File "CvMainInterface", line 719, in redraw

  File "CvMainInterface", line 1394, in updateSelectionButtons

RuntimeError: unidentifiable C++ exception
ERR: Python function forceScreenRedraw failed, module CvScreensInterface

It's obvious that the problem is with the Imperial Age, but it's difficult to find what is causing the infinite loop. Seven05 suggested to analyze the dump file, but because this is an infinite loop and not a CTD, there is no dump file to analyze. With Python and DLL removed, this still occurs.


In your experience and CIV 4 modding wisdom, what do you think could be causing this problem? And do you have any ideas how I could fix it?

 

[Kael]

Usually isolating the crash is the hard part, fixing it is easier (and hard to give general advice to since it depends on the cause).

In your case you probably have looping tech chain. Tech A requires Tech B which Requires Tech C which requires Tech A. When the AI considers what tech to research it walks down the tech line to see what they lead to. I suspect its walking through that looping tech chain forever which is causing the spinlock.

Check it out and see what you can find. If you don't see any issues post back here and send me another PM ping and I'll download TAM and take a look.

 

[Shqype]

Awesome Kael, thanks for your reply.

Only problem is, there was no such "bug" in vanilla TAM, and when I upgraded to BTS I didn't change the techs at all, save adding the necessary new tags that came with BTS.

 

[Shqype]

Kael, looking through CvPlayer.cpp I found the following which may be relevant to the issue TAM is having:

const TCHAR* CvPlayer::getUnitButton(UnitTypes eUnit) const
{
    return GC.getUnitInfo(eUnit).getArtInfo(0, getCurrentEra(), (UnitArtStyleTypes) GC.getCivilizationInfo (getCivilizationType()).getUnitArtStyleType())->getButton();
}

And, perhaps more importantly, this:

void CvPlayer::setCurrentEra(EraTypes eNewValue)
{
    CvCity* pLoopCity;
    CvUnit* pLoopUnit;
    CvPlot* pLoopPlot;
    int iLoop;
    int iI;

    if (getCurrentEra() != eNewValue)
    {
        EraTypes eOldEra = m_eCurrentEra;
        m_eCurrentEra = eNewValue;

        if (GC.getGameINLINE().getActiveTeam() != NO_TEAM)
        {
            for (iI = 0; iI < GC.getMapINLINE().numPlotsINLINE(); iI++)
            {
                pLoopPlot = GC.getMapINLINE().plotByIndexINLINE(iI);
                pLoopPlot->updateGraphicEra();

                if (pLoopPlot->getRevealedImprovementType(GC.getGameINLINE().getActiveTeam(), true) != NO_IMPROVEMENT)
                {
                    if ((pLoopPlot->getOwnerINLINE() == getID()) || (!(pLoopPlot->isOwned()) && (getID() == GC.getGameINLINE().getActivePlayer())))
                    {
                        pLoopPlot->setLayoutDirty(true);
                    }
                }
            }
        }

        // dirty all of this player's cities...
        for (pLoopCity = firstCity(&iLoop); pLoopCity != NULL; pLoopCity = nextCity(&iLoop))
        {
            if (pLoopCity->getOwnerINLINE() == getID())
            {
                pLoopCity->setLayoutDirty(true);
            }
        }

        //update unit eras
        for(pLoopUnit = firstUnit(&iLoop); pLoopUnit != NULL; pLoopUnit = nextUnit(&iLoop))
        {
            gDLL->getEntityIFace()->updateGraphicEra(pLoopUnit->getUnitEntity(), eOldEra);
        }

        //update flag eras
        gDLL->getInterfaceIFace()->setDirty(Flag_DIRTY_BIT, true);

        if (getID() == GC.getGameINLINE().getActivePlayer())
        {
            gDLL->getInterfaceIFace()->setDirty(Soundtrack_DIRTY_BIT, true);
        }

        if (isHuman() && (getCurrentEra() != GC.getGameINLINE().getStartEra()) && !GC.getGameINLINE().isNetworkMultiPlayer())
        {
            if (GC.getGameINLINE().isFinalInitialized() && !(gDLL->GetWorldBuilderMode()))
            {
                CvPopupInfo* pInfo = new CvPopupInfo(BUTTONPOPUP_PYTHON_SCREEN);
                if (NULL != pInfo)
                {
                    pInfo->setData1(eNewValue);
                    pInfo->setText(L"showEraMovie");
                    addPopup(pInfo);
                }
            }
        }
    }
}

Does this ring any bells in regards to the issue we are having? The techs and everything else seems fine, but when you enter the Imperial Era (which corresponds to and replaces the Renaissance Era), the game goes into an infinite loop where the AI is attempting to do something it cannot, or you get the python error messages when clicking on a city as noted a few posts above.

If you're able to look at TAM to give us a hand, TAM BTS v0.73 is out.

 

[Kael]

I downloaded and took a look at the mod (great job by the way, I havent played TAM in ages and its even better than I remembered, I love the new icons). The CtD is coming from the worker unit, in your CIV4UnitArtStyleTypeInfos.xml you have the late era art definition of the workers set to ART_DEF_UNIT_WORKERMODERN, which doesn't exist in your art definitions.

You can either add that definition to your unit art definitions or change ART_DEF_UNIT_WORKERMODERN to be the same as the Early and Middle art definition.

 

[Shqype]

Kael has once again demonstrated his excellence in all things CIV 4. Many, many thanks Kael, on behalf of the TAM development team and TAM lovers everywhere!

 

[woodelf]

All Hail Kael. We can now play TAM for BtS!!!

 

[Gaius Octavius]

I downloaded and took a look at the mod (great job by the way, I havent played TAM in ages and its even better than I remembered, I love the new icons). The CtD is coming from the worker unit, in your CIV4UnitArtStyleTypeInfos.xml you have the late era art definition of the workers set to ART_DEF_UNIT_WORKERMODERN, which doesn't exist in your art definitions.

You can either add that definition to your unit art definitions or change ART_DEF_UNIT_WORKERMODERN to be the same as the Early and Middle art definition.

Believe it or not, I had an almost identical problem with a scenario I was doing. It was absolutely silly because I was worried about some kind of major error, and there were no usual "XML failed to load" kinds of messages. Crazy.

 

[Kailric]

This is my logging function that I use in the SDK. Just stick this at the top of the cpp file after all of the #include statements to use it. Also, if you use it in multiple files, you need to give it a different name for each file (or do it properly and give it a header file that is included, but this is simpler, especially if each file uses its own logfile).

void logMsg(char* format, ... )
{
        static char buf[2048];
        _vsnprintf( buf, 2048-4, format, (char*)(&format+1) );
        gDLL->logMsg("ai.log", buf);
}

This would log the messages that use this function to "Documents and Settings\<username>\My Documents\My Games\<whatever your civ4 dir is called>\logs\ai.log" not "program files\Firaxis Games\Civ4\anything".

you would call this exactly like printf - logMsg("message to log"); or logMsg("log:%d",aNumber);


Also, instead of using a changelog like Kael's, you might want to use version control if you can deal with it. It would allow you to save all the versions of the files so you know exactly what changes you made. I personally use Subversion for the spells code I wrote for FFH.

I put this in my CvUnits.cpp code and did all you said with adding logMsg.. but I am not finding a ai.log anywhere.. whats the deal?

 

[Gaius Octavius]

Here is an issue I've been experiencing as of late that might cause others undue trouble as well. I've been modding the SDK and testing it out in a modular scenario. Trouble is, whenever I try to load a saved game I get a weird crash almost immediately. New games load perfectly, however, and if I first start a new game from scratch, then reload the old saved game, it comes up fine.

I went through my SDK changes and was stumped, since nothing appeared to be wrong. I finally switched from a modular to a regular mod, and the crash vanished. Since it was not an intermittent issue I have to assume it was caused somehow by the modular loading system, so I'd advise anyone with similar problems who is also using the modular system to convert to a regular mod and see if the crash still occurs. I don't know why this happens, but I've heard that it's happened to other people, too.

Am I wrong here, or is the modular system just prone to do these kinds of seemingly random things?

 

[deanej]

I have a crash that I can't really explain. When I play as the Borg civ (note: may actually be due to the leader I tested it on, not sure) in my Star Trek mod and select the IDIC tech to research, I immediately get a CTD. Researching IDIC with other civs works fine. As far as I know this did not occur prior to 3.17. Please advise.

 

[PsiCorps]

I knew i'd seen a thread about isolating crashes just couldn't remember the title. Now to investigate why the mod is crashing.

 

[BvH]

Great turorial! Just one thing: I was wondering if anyone knows what might cause a mod to crash as soon as I end the turn. I'm playing as Montezuma with my new civ as the AI (it crashes before the game loads if I play as my new civ), and although the tutorial is undoubtedly excellent, it doesn't do me much good, because the mod ALSO crashes when I open WorldBuilder. It's kinda driving me insane. Anyway, thanks in advance.

 

[The_J]

A very often occuring cause is missing ethnic unit art.
In the CIV4UnitArtStyleTypeInfos.xml you can define all ethnic units, but it does not get crosschecked, if these entries are present in the ArtDefines. If not, and such a unit is build, the game will crash.