A guide to debugging crashes with a custom DLL

[ls612]

BNW has been out for over a month now and the source for the DLL has been available in one form or another since last year, and many people are now making their own DLLs for their mods. However, it seems that many don't know how to debug crashes and issues with their mods (especially with DLL changes, which can easily cause crashes). This guide will focus on debugging your mod using Visual Studio's built in debugging features as well as optionally a modcomp to allow you to debug crashes others have using minidumps.

To Begin:
If you want to debug your mod with a custom DLL make sure that you copy the CvGameCoreDLL_FinalRelease2.pdb file to your mod's root directory (the same place the DLL goes) and import that into VFS. This is necessary if you want to attach to the process and debug in real-time. After you do that rebuild the mod and launch civ 5. When you get to the menu screen tab out of it and go back into Visual Studio. In VS (any version, for this demo I'm using VS 2012 Ultimate but it is similar for other versions) and under the Tools menu select 'Attach to Process'. You will see a window similar to this.



Live Debugging:
Select the civ 5 process (for most of us it will be CivilizationV_DX11.exe) and click 'Attach'. Now you will be attached to the game process, but your mod isn't loaded yet so that isn't too useful for now. After that go and load your mod. If you have done everything right with your DLL and PDB you should see the following line in the output window.

'CivilizationV_DX11.exe' (Win32): Loaded 'C:\Users\*Your Username*\Documents\My Games\Sid Meier's Civilization 5\MODS\New Horizons (v 1)\CvGameCore_Expansion2.dll'. Symbols loaded.

Except with your mod's name. This means that VS can now use breakpoints you set in your mod's C++ code. I will now demonstrate an example of this using my mod and some code purposely designed to crash.

To set a breakpoint in Visual Studio click on the left side of the code editor and you will get a red dot on a line of code.

WARNING: I highly recommend that you only debug with Civ 5 in Windowed mode, otherwise you can end up stuck as apps running in fullscreen don't play nice with the Visual Studio debugger.

Now when the program tries to execute that line of code it will break execution there and give manual control to the debugger.

When it does that, you have three main options; Step Over, Step Out and Step into. The first will execute every function on that line and then break at the next line of code in that document. Step Into will go to the function being called (assuming you have the source for that) and break there. Step Out will finish execution of the current function and break when control is given back to the calling function.

Now I try and use a settler. When I do I will get a popup in VS saying that I got an unhandled exception. Click Break, and you will see something like this.



The code editor will show where your mod just crashed, and you will also get windows for the call stack, local variables, and other debug info. After looking a bit I can see that my call to this function is not properly set up, and I am not ensuring that eCapturingPlayer is not NO_PLAYER. Now I know exactly what caused that crash and what I need to do to fix it. This is incredibly useful for making your mod more stable.

Post-Mortem Debugging:

You can also debug crashes other people have had after the fact using minidumps. I have created a modcomp which will create nice usable minidumps when the game crashes using your mod. Note that this will override Firaxis' default dump creation, so if you have a bug with civ 5 itself and not your mod you might want to disable this modcomp.

To use a minidump, place the CvMiniDump.dmp file (found in the folder where your civ 5 EXE is) in the same folder as your DLL and PDB file, then open it in Visual Studio and click 'Start Debugging'. You will then see a very similar screen to the one you see debugging live, and can see what went wrong with your mod. This is very useful for diagnosing issues end-users of your mod experience, whereas it is far more useful to debug live if you are experiencing the issue yourself.

 

[Putmalk]

Thank you for this. I'll look into this. Will be useful!

 

[Putmalk]

Okay I'm getting around to needing this - CvGameCoreDLL_FinalRelease2.pdb. Where is that located? I can't find that (although for some reason I remember having it...)

 

[S3rgeus]

Thanks for the post! Very helpful thing to have a guide for. I've been doing this for a while though, and I never included the .pdb file in my mod. It's been a few weeks since I last ran live debugging though, so I can't remember if I got that full stack trace like you did in the screenshot. (I definitely got some kind of stack trace, just not sure if it was as 'accurate'.)

 

[ls612]

Okay I'm getting around to needing this - CvGameCoreDLL_FinalRelease2.pdb. Where is that located? I can't find that (although for some reason I remember having it...)

It should be wherever the output from your build is (ie, where the DLL goes after you compile). You may however have at some point either changed where things go in Project Properties or suppressed the LNK4099 warning, which will prevent a PDB from being generated.

 

[ls612]

Thanks for the post! Very helpful thing to have a guide for. I've been doing this for a while though, and I never included the .pdb file in my mod. It's been a few weeks since I last ran live debugging though, so I can't remember if I got that full stack trace like you did in the screenshot. (I definitely got some kind of stack trace, just not sure if it was as 'accurate'.)

You will not get anything accurate from the debugger without the symbols for your DLL. IIRC if you don't have those breakpoints you set in code will not be triggered either.

 

[biship]

Sorry for the necro....
Running my own compile of 4/30 pull from Github of CPP. Symbols seem loaded.

How do I see the variable values in VS2015 (from the dump, postmortem)?

Dump reports a 0xC0000005.
https://blogs.msdn.microsoft.com/calvin_hsia/2004/06/30/what-is-a-c0000005-crash/


Call Stack:

 	CvGameCore_Expansion2.dll!CvPlayer::GetGreatPersonExpendGold() Line 23959	C++
 	CvGameCore_Expansion2.dll!CvUnit::GetMaxHitPoints() Line 14648	C++
 	CvGameCore_Expansion2.dll!CvLuaUnit::lGetCombatDamage(lua_State * L) Line 912	C++

CvLuaUnit.cpp

#if defined(MOD_BALANCE_CORE)
	//for visual feedback, take care that we show the precise value
	CvCity* pkCity = CvLuaCity::GetInstance(L, 8, false);
	if (pkCity && pkCity->HasGarrison())
	{
		CvUnit* pGarrison = pkCity->GetGarrisonedUnit();
		int iGarrisonShare = (iResult*2*pGarrison->GetMaxHitPoints()) / (pkCity->GetMaxHitPoints()+2*pGarrison->GetMaxHitPoints());
		iResult -= iGarrisonShare;
	}
#endif

CvUnit.cpp

int CvUnit::GetMaxHitPoints() const
{
	VALIDATE_OBJECT
#if defined(MOD_UNITS_MAX_HP)
	int iMaxHP = getMaxHitPointsBase();

	iMaxHP *= (100 + getMaxHitPointsModifier());
	iMaxHP /= 100;
	
	iMaxHP += getMaxHitPointsChange();

	return iMaxHP;
#else
	return GC.getMAX_HIT_POINTS();
#endif
}

getMaxHitPointsModifier:

int CvUnit::getMaxHitPointsModifier() const
{
	return m_iMaxHitPointsModifier;
}

GetGreatPersonExpendGold:

int CvPlayer::GetGreatPersonExpendGold() const
{
	return m_iGreatPersonExpendGold; (debugger says unable to read memory)
}

Autos & Locals look to be vars, but everything is reporting it can't read memory. I kinda thought the dmp file would contain the var values at dump time.

I don't see the point attaching to the civ5 process as won't that give me the current run-time vars.

I would have expected the actual error reported - integer overflow, undefined var etc.
I guess m_iMaxHitPointsModifier is either huge or undefined or something.

How do I go about at least finding the var values, then perhaps I can find the unit (and its properties)?

debug output window:
Exception thrown: read access violation.
this was nullptr.

 

[whoward69]

The CvMiniDump.dmp, .dll and .pdb (symbols) files ALL have to the same build - ie the CvMiniDump.dmp has to have been generated by the .dll that the .pdb file relates to.

When you have a reported crash (0x5 IIRC is null pointer related) that has nothing to do with code that contains pointers, typically the .dmp file is not for that .dll/.pdb combo

 

[biship]

The CvMiniDump.dmp, .dll and .pdb (symbols) files ALL have to the same build - ie the CvMiniDump.dmp has to have been generated by the .dll that the .pdb file relates to.

When you have a reported crash (0x5 IIRC is null pointer related) that has nothing to do with code that contains pointers, typically the .dmp file is not for that .dll/.pdb combo

Thanks for the reply. To your first point, they are all the same version. I compiled the dll and pdb, and I am using the dll (from which the dmp was created from).

It seems I can reproduce this, so I attached VS2015 to Civ5.exe, and set the breakpoint.

If I step over the line, it crashes civ5 (as expected).

However, all the vars in Auto & Local still show it can't read memory.
EDIT: If I add a var to the watch list, I get: iMaxHP identifier "iMaxHP" is undefined
(even after it has stepped over return iMaxHP

What should I be doing to finding whats wrong?

EDIT: The crash is occurring when I mouse over a Civ's city I am at war with.
EDIT2: Only for one of their cities. (the rest continue out of the method without a crash).

 

[S3rgeus]

Thanks for the reply. To your first point, they are all the same version. I compiled the dll and pdb, and I am using the dll (from which the dmp was created from).

It seems I can reproduce this, so I attached VS2015 to Civ5.exe, and set the breakpoint.

If I step over the line, it crashes civ5 (as expected).

However, all the vars in Auto & Local still show it can't read memory.
EDIT: If I add a var to the watch list, I get: iMaxHP identifier "iMaxHP" is undefined
(even after it has stepped over return iMaxHP

What should I be doing to finding whats wrong?

EDIT: The crash is occurring when I mouse over a Civ's city I am at war with.
EDIT2: Only for one of their cities. (the rest continue out of the method without a crash).

Many variables will often be optimized away by the compiler and unavailable in the debugger session (because they don't actually exist in the running program). If you recompile with optimization disabled (if VS2015 is like VS2013, that's done by: Right Click the DLL project in the solution explorer -> Properties -> Configuration Properties -> C/C++ -> Optimization) and reproduce the issue again, then those variables should be available when debugging that unoptimized build.


Also, I never followed up on this before (back in 2013), but you definitely don't need to include the pdb in your mod. As long you as you, the developer, have it available on your PC somewhere that Visual Studio can find it, then you should be able to debug dumps generated by the corresponding DLL on other users' machines.

 

[biship]

Thanks S3rgeus.

I managed to get VS2015 to show the vars disabling the optimizations.

The call stack shows one more routine (my post was updated), and now it dumps on GetGreatPersonExpendGold.

debug output window:
Exception thrown: read access violation.
this was nullptr.

Debugger says its unable to read memory for m_iGreatPersonExpendGold.
I guess that's the problem.

I don't see anything in the call stack calling GetGreatPersonExpendGold, but I really am not a programmer, so its not surprising.

 

[whoward69]

Don't forget that an errant pointer won't show up until the code attempts to read something from it (in this case m_iGreatPersonExpendGold), so it's not that the contents of m_iGreatPersonExpendGold that is incorrect but the pointer that is being dereferenced by the attempt to obtain a value for m_iGreatPersonExpendGold.

The only pointer in that code is pGarrison. The HasGarrison() method is new and the GetGarrisonedUnit() method has been totally rewritten (although you can't tell this just by looking at CvCity.h as #if/#else/#endif blocking has been dumped). Also, from a quick scan of the code, it is possible that HasGarrison() could return true while the unit it thinks is the garrison has been killed, hence causing GetGarrisonedUnit() to return null.

 

[gia]

Is it possible to Edit and continue with this? or just to have a better look at a crash?

 

[Serp]

Could someone add information how to read the CvMiniDump.dmp created by that modcomp?
You write "in the same folder as your DLL and PDB file, then open it in Visual Studio and click 'Start Debugging'."
I put the dmp into my BuildOutput folder, since there are the DLL and PDB file, but when I open it with VS the "start debugging" button is greyed out and when I hit "debug with native only" instead, I receive an error that Civ exe was not found.

So how to proceed? @ls612

edit:
also how does setting breakpoints work?
I attached the process and if the game crashes, I'm able to break and see the problematic line.
But if the game does not crash, everything in VS is just blank, also the line that I should see when the mod is loaded, is not there (I did not put the pdb file into the mod, because of the post of S3rgeus and the fact that it works if it crashes). I can go to Debug->set New Breakpoint, but this window is a joke
There is no "output", only after civ5 was closed. In "Modules" I see my custom dll loaded. But I also have no code window.
I put in the pdb file now, but it seems nothging changed, also in modules I also get the info "C:\...\CvGameCoreDLL_Expansion2Win32Mod.pdb: Symbols loaded.". But still no clue how/whre to put breakpoints in VS.

edit2:
found the output window, had to enable it with View->Output menu.
Symbols are loaded and I get every ~20 seconds:
"First-chance exception at 0x76dd4192 in CivilizationV.exe: 0xE06D7363: Microsoft C++ Exception." and while an actual game is running I get alot "First-chance exception at 0x00e29bcb in CivilizationV.exe: 0xC0000005: Access violation writing location 0x00000000." no clue what this means, but the game is running fine. Still no clue how to add breakpoints.

 

[FlammieJerynZ]

Hey, sorry about the delay in reply to you here and for my ignorance, but I have a question.
This method also works with Crashes related with .lua files (where the Crash is provoked by lua51_Win32)? Or only must be manually tracked in the internal files there to fix it?
Thank you in advance and I will appreciate any answer if possible!