Tracking down a freeze bug

[OrionVeteran]

I have traced down an unusual error that occurs when a city has been captured. I ran the turn manually, captured the city and the following message comes up (See attached picture). When I click Burn baby burn or Yes! Install new governor, the game freezes up!

Does anyone know where in the SDK or python the popup message is generated from?

 

[The_J]

I have no idea where it's generated, but I've experienced while dealing with something else that queued up popups after this popup might lead to a freeze.

 

[OrionVeteran]

I have no idea where it's generated, but I've experienced while dealing with something else that queued up popups after this popup might lead to a freeze.

I can run it manually and get the error when I make a selection on the popup or...

I can get the error using AIAutoPlay that does not display the popup at all.

Either way, the freeze up is occurring. I have tried to disable the python code that runs for onCityAcquired and onCityAcquiredAndKept. No different! I still get the freeze up. Sometimes, the error will cause a CTD, but not too often. No doubt about it, this is a bad bug that must be resolved. The only good news is I have stumbled on when and where it occurs. So now it can be worked. I have been looking for this bug for over 2 weeks!

 

[embryodead]

File: CvDLLButtonPopup.cpp

Methods, in this order:
1. launchButtonPopup
2. launchRazeCityPopup
3. OnOkClicked
(button popup type is BUTTONPOPUP_RAZECITY all that time)

But that's unlikely to help you much (as you say, the freeze isn't connected to the human popup as it happens on autoplay as well; it's more likely happening in CvCity.cpp or CvPlayer.cpp). What you need to do is compile a debug DLL and attach the debugger. Then when the freeze happens you select Debugger->Break in Visual Studio and check the Call stack. You'll know exactly what happens, function by function. You can track back, or go forward, step by step.

If that doesn't work (i.e. it lands you in the assembly) you can insert a breakpoint in launchButtonPopup method and go step by step from there. Obviously you'll want to do this with a save where this happens 100% of the time.

Here's a tutorial by Asaf that includes a section on Debugging; it gives more details on things I mentioned, if needed.

 

[The_J]

I can run it manually and get the error when I make a selection on the popup or...

I can get the error using AIAutoPlay that does not display the popup at all.

That might not be relevant (but I'm not really sure ). I experienced my bug also when a city was auto razed, so also at a point when no popup was displayed.

But embryodead is sure right, debug dll should help a lot.

 

[OrionVeteran]

Yes, a debug DLL would be a real big help here. I had trouble trying to create a debug DLL last week and I failed in the attempt. I'm not sure what I am doing wrong in the Microsoft Visual C++ 2008 Express edition.

 

[OrionVeteran]

File: CvDLLButtonPopup.cpp

Methods, in this order:
1. launchButtonPopup
2. launchRazeCityPopup
3. OnOkClicked
(button popup type is BUTTONPOPUP_RAZECITY all that time)

But that's unlikely to help you much (as you say, the freeze isn't connected to the human popup as it happens on autoplay as well; it's more likely happening in CvCity.cpp or CvPlayer.cpp). What you need to do is compile a debug DLL and attach the debugger. Then when the freeze happens you select Debugger->Break in Visual Studio and check the Call stack. You'll know exactly what happens, function by function. You can track back, or go forward, step by step.

If that doesn't work (i.e. it lands you in the assembly) you can insert a breakpoint in launchButtonPopup method and go step by step from there. Obviously you'll want to do this with a save where this happens 100% of the time.

Here's a tutorial by Asaf that includes a section on Debugging; it gives more details on things I mentioned, if needed.

After thinking about your comments, I realized that none of the mods I have merged have made any changes at all to the file CvDLLButtonPopup. The mod worked OK in earlier phases. Something else must have caused the issue. I'm curious as to why you think CvCity.cpp or CvPlayer.cpp is the likely cause, and which functions within those two files could cause a freeze up when razing or keeping a city? The reason I ask is because those two files are heavily modified.

 

[embryodead]

After thinking about your comments, I realized that none of the mods I have merged have made any changes at all to the file CvDLLButtonPopup. The mod worked OK in earlier phases. Something else must have caused the issue. I'm curious as to why you think CvCity.cpp or CvPlayer.cpp is the likely cause, and which functions within those two files could cause a freeze up when razing or keeping a city? The reason I ask is because those two files are heavily modified.

As I said, CvDLLButtonPopup is irrelevant if the freeze happens on autoplay (that popup is used for human players only). But since you can reproduce the freeze happening after that popup is closed, it means the freeze has got something to do with city capture in general, and that is handled within CvCity.cpp and CvPlayer.cpp (specifically CvPlayer::acquireCity).

Be aware that while CTDs and freezes don't necessarily have to come from DLL code, but debugging that code can help identify other problems, e.g. calls from Python with bad arguments. Sometimes it can be something entirely different - I just remember I had a freeze on city capture as well, after a lot of fruitless debugging I finally found out it happened when a particular unit captured a city. Turned out it was bad unit NIF. Not sure what was wrong with it, but it was my own Blender creation and doing it from scratch fixed the problem.

 

[OrionVeteran]

As I said, CvDLLButtonPopup is irrelevant if the freeze happens on autoplay (that popup is used for human players only). But since you can reproduce the freeze happening after that popup is closed, it means the freeze has got something to do with city capture in general, and that is handled within CvCity.cpp and CvPlayer.cpp (specifically CvPlayer::acquireCity).

Be aware that while CTDs and freezes don't necessarily have to come from DLL code, but debugging that code can help identify other problems, e.g. calls from Python with bad arguments. Sometimes it can be something entirely different - I just remember I had a freeze on city capture as well, after a lot of fruitless debugging I finally found out it happened when a particular unit captured a city. Turned out it was bad unit NIF. Not sure what was wrong with it, but it was my own Blender creation and doing it from scratch fixed the problem.

After more troubleshooting, I think have discovered the source of the problem. Its a stack attack against a city. Specifics: The attacking stack has cannons. If all units are selected and attack the city, razing the city will freeze up the game. However, If I attack with the cannons first, using ranged bombardment; and then select the remaining units to attack and take the city, then razing the city does not freeze the game. I was able to repeat both scenarios several times. I assume at this point the function updateStackCombat, in the CvUnit.cpp file, has a problem. I think making the AI stack attack use the siege weapons first, could solve the problem. So, at this point, I am optomistic about a possible solution. ...First time in 2 weeks.

 

[embryodead]

Before making a new solution, try to find the original source of the problem. This kind of freeze shouldn't happen at all. Any changes to CvUnit, CvCity etc.? Any custom graphics involved (like new cannons?). Any Python that could involve siege units?

 

[OrionVeteran]

Before making a new solution, try to find the original source of the problem. This kind of freeze shouldn't happen at all. Any changes to CvUnit, CvCity etc.? Any custom graphics involved (like new cannons?). Any Python that could involve siege units?

I have Dale's Combat Mod v2.2 for BTS 3.19, by Stolenrays & Roamty installed. Other than Cannons, there were no other siege units owned by the attacker, that participated in the attack. There was a siege tower that was destroyed, owned by the defending city. I don't believe the seige tower is the problem, since it was destroyed under both scenarios. The cannons are stock BTS cannons. This leads me to believe it is a problem with stack attack. The city was captured by a single grenadier unit, leaving several remaining attacking units on the plot adjacent to the city.

I found this in the updateStackCombat function:

if (pPlot->isActiveVisible(false))
{
	ExecuteMove(0.5f, true);
	[COLOR="Red"]// RevolutionDCM - stack attack - this next line is one source of stack attack CTD's[/COLOR]
	gDLL->getEntityIFace()->AddMission(&kBattle);
}

 

[embryodead]

I'm not familiar with DCM code, but I recall it has a "stack attack" option, so some code is surely modified there. Something could went wrong there, e.g. with the merge. That's a bit of work but you could try making a non-DCM DLL and see if the problem persist.

 

[OrionVeteran]

I'm not familiar with DCM code, but I recall it has a "stack attack" option, so some code is surely modified there. Something could went wrong there, e.g. with the merge. That's a bit of work but you could try making a non-DCM DLL and see if the problem persist.

Yes, Check my update to my last post.

 

[stolenrays]

I'm building a debug dll right now.

 

[embryodead]

If I get it right, RevDCM acknowledges that stack attack is broken and causes CTDs? Can you just turn it off in DCM options?

 

[stolenrays]

I tried debugging twice and got several asserts when attacking a city with stack attack on. In addition, I also got an unhandled exception. It would not point me to the source code unfortunately. Not sure why or if I was doing something wrong, but I can't fix it if I don't know the source code cause of the problem.

It seems as if Embryodead is correct. Most of the other mods have "fixed" the bug by just turning the feature off in the Gameoption.xml. I never use it as well, so that's probably why I never encountered the asserts/CTDs.

 

[OrionVeteran]

I tried debugging twice and got several asserts when attacking a city with stack attack on. In addition, I also got an unhandled exception. It would not point me to the source code unfortunately. Not sure why or if I was doing something wrong, but I can't fix it if I don't know the source code cause of the problem.

It seems as if Embryodead is correct. Most of the other mods have "fixed" the bug by just turning the feature off in the Gameoption.xml. I never use it as well, so that's probably why I never encountered the asserts/CTDs.

I'm going to try and track this bug down and I'm getting close. I'll let you know when I find the exact location.

 

[OrionVeteran]

Locating the actual line of code causing the CTD in the SDK Stack Attack function has proven very difficult to pin down. However, disabling Stack Attack definitely fixed the CTD. I thought the freeze up was a related issue, but evidently not, as the game still freezes up at a point "before" it ran Stack Attack. So, I still have work to do to find the cause of the freeze up. I have talked with the historian, and we will develop our own version of stack attack for BTS 3.19. Whether or not it can be put into the pending release of OGI in time for Christmas is doubtfut, but who knows. For now, the first priority is finding the freeze up and fixing it.

 

[stolenrays]

Disapointing on the short-term for fixing the CTD, but I will replace the current stack attack with your new one when it is finished.

 

[OrionVeteran]

What can cause a bunch of these errors in the audio.log file?

[5935.578] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5935.578] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.
[5955.343] WRN: FSoundQueueMgr::GetNextAvailableSoundQueue(): Ran out of soundqueues.