Single Player bugs and crashes v36 plus (SVN) - After the 24th of October 2015

Thunderbrd · Dec 6, 2015

strategyonly said:
@ TB

its happening alot lately, forces leave their city with alot of troops and only leave 1 unit in city when i am attacking with really less than 2 units??

If that save is from just before they do so, I may well be able to find the issue. I presume they aren't attacking after you've captured the city either right?

alberts2 said:
Kp

There is a limit of 8192 for things like SelectionGroups.
Reusing old Id's is a bad idea, there could be old references to them stored somewhere and reusing them could lead to odd bugs.

There are a few ways to fix that but most of them could increase turn times and/or memory usage. But it has been a long time since that issue came up so it doesn't have a high priority.

Splitting barbs and animals should have been done long ago and i personally don't care if it breaks compatibility. You and me made changes which could cause odd behavior in old saves but luckily nobody really noticed it. That basically means compatibility should be broken more often to avoid odd bugs in old saves.

Wow... it would be a big step to break compatibility.

I get there being a limit of the # of selection groups but I would think for sure I've seen numbers on their IDs being much much higher than 8192. I could be wrong.

The thing is, this is happening on a pretty early on game and selection groups are often created then destroyed very quickly when units leave other groups, join back etc. We may end up seeing this problem manifest before or within Galactic even if the system isn't all that stressed.

I'm not against splitting barbs and animals and we could/should start a thread to discuss what all we want to get out of a compatibility break if we're going to do it.

There will be a lot of little things involved in splitting animals and barbs... will take some serious effort to do this right.

tmv said:
In #353, I mentioned a problem with Felis Superior. The problem still exists with SVN 8921.

To elaborate, Felis Superior requires Effect - Big Cat Trainer, which obsoletes at Aviation. Felis Superior also requires Homo Superior.

I wanted to find out if it was possible to build Felis Superior if you actually didn't research Aviation until you were halfway through TH, but Aviation is a prerequisite for Homo Superior. One of the research chains goes as follows:

Aviation -> Rocketry -> Jet Propulsion -> Advanced Rocketry -> Guided Weapons -> Communication Networks -> Neural Networks -> Machine Learning -> Augmented Reality -> Ubiquitous Computing -> Gene Enhancement -> Mainstream Cloning -> Rapid Recuperation -> Bionics -> Cybernetics -> Species Uplifting -> Homo Superior

And another thing: In the Civilopedia, you cannot open a promotion page anymore. If you click on a promotion, the main page stays blank and you get back to the top of the promotion list. This happens in SVN 8921 as well.

Sorry, this is an easy fix but I tend to forget the simple stuff. I'll try to remember this time.

Koshling said:
If Python ever gets invoked on any but the main thread a crash is likely to result because the game engine does not have any multi-threading support. This means that all multi-thread support has to be 'hidden' strictly inside the code we run in the DLL or else all hell breaks loose (usually manifesting as a heap, corruption which fits these symptoms). Since we have no control over what happens in a Python call (what the invoked Python will itself then call) it means it is always unsafe to invoke a Python call on any but the main thread. The DLL code goes to some lengths to avoid this happening (stuff gets queued from the background threads to be invoked on the main thread explicitly), but it may well be we missed a few cases. To address this there is a workaround, and a debugging process (so workaround for a user experiencing it, process to follow for someone trying to fix it):

1) To work around it disable multi-threading. There's a one line setting in the config XML to achieve this, which I'm sure someone can look up in the XML and report back here (set the thread count to 1)

2) To try to get to the bottom of it put a debug breakpoint on the assertion failure that spots non-main-thread Python invocation to find out where in the code its making a Python call from where it shouldn't. Having found that point stop it happening, either by removing the need to call Python there somehow, or by queueing it onto the work queue to be executed by the main thread (there is already code to do this in various places, but I forget the exact details)

I wondered if disabling multithreading would work initially.

The solution is at least a start as to how to go about it. Since I'm not familiar with working with multi-threading I'm not sure I will be able to do the ideal thing, which will be to queue it onto the work queue as you explained, but I should be able to find and disable it... God (or maybe DH) knows what the python is trying to do here.

Thanks though Koshling... very nice to see you in the main forum again!

strategyonly · Dec 6, 2015

Thunderbrd said:
If that save is from just before they do so, I may well be able to find the issue. I presume they aren't attacking after you've captured the city either right?

Yes, its the turn before i go for the all out taking the city, and in my game, i took the city and yes they ran away like crying babies . . :cry:

alberts8 · Dec 6, 2015

Thunderbrd said:
Wow... it would be a big step to break compatibility.

Some changes already did that but you can still load and play earlier saves. The result is bugs and most of them are never noticed so the illusion of full compatibility remains.

An example for that is rev-8573. All saves from earlier versions are corrupt because of bugs but if you load an save from a earlier version with rev-8573 or later they are even more brocken. That basicly means fixing those bugs broke the compatibility because earlier save don't work as they did before.

Thunderbrd · Dec 6, 2015

alberts2 said:
Some changes already did that but you can still load and play earlier saves. The result is bugs and most of them are never noticed so the illusion of full compatibility remains.

An example for that is rev-8573. All saves from earlier versions are corrupt because of bugs but if you load an save from a earlier version with rev-8573 or later they are even more brocken. That basicly means fixing those bugs broke the compatibility because earlier save don't work as they did before.

Ok, well... I've started a thread to discuss what all needs to be done if we're going to break compatibility. The thread is HERE.

The Barb/Animal split will in itself be quite a task. A little daunting tbh. But I THINK I would be able to pull it off.

I'm about to introduce a new Command and I have some question in my mind as to whether it should've been made as a Mission instead. I may want to get a little feedback on this next commit on that subject and see if you agree that it should be converted to a Mission. Doing so would be a little confusing at points for me as to how I should go about it. But if we're going to be breaking compatibility, it would be during the brackets of this compat breakage zone that I'd want to re-implement it so I can safely remove the command I'm introducing.

Thunderbrd · Dec 6, 2015

Koshling said:
If Python ever gets invoked on any but the main thread a crash is likely to result because the game engine does not have any multi-threading support. This means that all multi-thread support has to be 'hidden' strictly inside the code we run in the DLL or else all hell breaks loose (usually manifesting as a heap, corruption which fits these symptoms). Since we have no control over what happens in a Python call (what the invoked Python will itself then call) it means it is always unsafe to invoke a Python call on any but the main thread. The DLL code goes to some lengths to avoid this happening (stuff gets queued from the background threads to be invoked on the main thread explicitly), but it may well be we missed a few cases. To address this there is a workaround, and a debugging process (so workaround for a user experiencing it, process to follow for someone trying to fix it):

1) To work around it disable multi-threading. There's a one line setting in the config XML to achieve this, which I'm sure someone can look up in the XML and report back here (set the thread count to 1)

2) To try to get to the bottom of it put a debug breakpoint on the assertion failure that spots non-main-thread Python invocation to find out where in the code its making a Python call from where it shouldn't. Having found that point stop it happening, either by removing the need to call Python there somehow, or by queueing it onto the work queue to be executed by the main thread (there is already code to do this in various places, but I forget the exact details)

Ok, the last line here is the call that's causing trouble:

Code:

void CvUnit::move(CvPlot* pPlot, bool bShow)
{
	PROFILE_FUNC();

	FAssert(canMoveOrAttackInto(pPlot) || isMadeAttack());

	CvPlot* pOldPlot = plot();

	changeMoves(pPlot->movementCost(this, plot()));

	//GC.getGameINLINE().logOOSSpecial(16, getID(), pPlot->getX_INLINE(), pPlot->getY_INLINE());
	setXY(pPlot->getX_INLINE(), pPlot->getY_INLINE(), true, true, bShow && pPlot->isVisibleToWatchingHuman(), bShow);

	//change feature
	FeatureTypes featureType = pPlot->getFeatureType();
	if(featureType != NO_FEATURE)
	{
		CvString featureString(GC.getFeatureInfo(featureType).getOnUnitChangeTo());
		if(!featureString.IsEmpty())
		{
			FeatureTypes newFeatureType = (FeatureTypes) GC.getInfoTypeForString(featureString);
			pPlot->setFeatureType(newFeatureType);
		}
	}

	if (getOwnerINLINE() == GC.getGameINLINE().getActivePlayer())
	{
		if (!(pPlot->isOwned()))
		{
			//spawn birds if trees present - JW
			if (featureType != NO_FEATURE)
			{
				if (GC.getASyncRand().get(100) < GC.getFeatureInfo(featureType).getEffectProbability())
				{
					EffectTypes eEffect = (EffectTypes)GC.getInfoTypeForString(GC.getFeatureInfo(featureType).getEffectType());
					gDLL->getEngineIFace()->TriggerEffect(eEffect, pPlot->getPoint(), (float)(GC.getASyncRand().get(360)));
					gDLL->getInterfaceIFace()->playGeneralSound("AS3D_UN_BIRDS_SCATTER", pPlot->getPoint());
				}
			}
		}
	}

	CvEventReporter::getInstance().unitMove(pPlot, this, pOldPlot);

This is happening during the city build multithreading. As soon as the unit is built, it's answering to a contract and immediately runs off to answer it - the thread is still not the main thread (it's thread 3 apparently.)

Not sure what's being done on unitMove in python but surely the potential for this issue has been with us for a while.

I don't want to simply delete this event report... I'm not sure if that's even a fix really. And placing it on a queue ... :shrug:

I think this probably would enable one with understanding to envision how this is taking place:

Code:

 	CvGameCoreDLL.dll!IFPPythonCall(const char * callerFn=0x05838960, const char * moduleName=0x0583894c, const char * fxnName=0x05838944, void * fxnArg=0x1c8be630, long * result=0x715eefb4)  Line 721 + 0x63 bytes	C++
 	CvGameCoreDLL.dll!CvDllPythonEvents::postEvent(CyArgsList & eventData={...}, const char * eventName=0x05838f04)  Line 71 + 0x24 bytes	C++
 	CvGameCoreDLL.dll!CvDllPythonEvents::reportUnitMove(CvPlot * pPlot=0x630341b4, CvUnit * pUnit=0x7a010458, CvPlot * pOldPlot=0x630466dc)  Line 805	C++
 	CvGameCoreDLL.dll!CvEventReporter::unitMove(CvPlot * pPlot=0x630341b4, CvUnit * pUnit=0x7a010458, CvPlot * pOldPlot=0x630466dc)  Line 374	C++
>	CvGameCoreDLL.dll!CvUnit::move(CvPlot * pPlot=0x630341b4, bool bShow=true)  Line 7325	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::groupMove(CvPlot * pPlot=0x630341b4, bool bCombat=false, CvUnit * pCombatUnit=0x00000000, bool bEndMove=false)  Line 5531	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::groupPathTo(int iX=88, int iY=28, int iFlags=4)  Line 5697	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::continueMission(int iSteps=0)  Line 2512 + 0x22 bytes	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::startMission()  Line 2403 + 0xa bytes	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::activateHeadMission()  Line 8039 + 0x8 bytes	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::insertAtEndMissionQueue(MissionData mission={...}, bool bStart=true)  Line 7760 + 0x8 bytes	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::pushMissionInternal(MissionTypes eMission=MISSION_MOVE_TO, int iData1=88, int iData2=28, int iFlags=4, bool bAppend=false, bool bManual=false, MissionAITypes eMissionAI=MISSIONAI_CONTRACT, CvPlot * pMissionAIPlot=0x62f8e664, CvUnit * pMissionAIUnit=0x00000000)  Line 696 + 0x34 bytes	C++
 	CvGameCoreDLL.dll!CvSelectionGroup::pushMission(MissionTypes eMission=MISSION_MOVE_TO, int iData1=88, int iData2=28, int iFlags=4, bool bAppend=false, bool bManual=false, MissionAITypes eMissionAI=MISSIONAI_CONTRACT, CvPlot * pMissionAIPlot=0x62f8e664, CvUnit * pMissionAIUnit=0x00000000)  Line 661	C++
 	CvGameCoreDLL.dll!CvCity::completeOrderProcessing(const OrderData * pOrderData=0x715efd2c)  Line 19991	C++

Seems to me that units shouldn't be off and running while multithreading is still engaged. Also a little odd that the unit is already moving while the city processing is still taking place.

Dancing Hoskuld · Dec 6, 2015

There is an onUnitMove test to do with the Natural Wonders in Python.

Thunderbrd · Dec 6, 2015

I'll have to wait for advisement to see what should be done then. I suppose it has to do with determining if the natural wonder is discovered.

Still seems odd that the unit is moving during the city's processing.

Pit2015 · Dec 7, 2015

Ok i found a big bug/problem! The AI is not building any community discussion building, makeing the AI fall back alot because it takes them to long to build knowledge inherti... so the AI falls back alot in research, they cant build myths this way from the beginning, they are to late, thats also why the AI sucks on education now, they miss those buildings and build them to late, make the AI so that there main priority is to build community discussions building asap, so then can use there hunting awards to build myth buildings and keep up with research!

So again look into the savegame that joseph provided, there you can see the problem, no AI civ has community discussions at all as it looks like, so they miss animal ... requirment to be able to build myths and only some AI civs have knowledge inherti... now very late, so they can build myths to late and they suck on education, looks like no AI civ has a community discussions building, so again:

Look into this savegame file "TEST-GW BC-18750.7z" in http://forums.civfanatics.com/showthread.php?t=552901 page 2, i made all AI civs visible, looks like AI sucks a bit on education now... just klick the civs city and look into the propertys.

Please fix/improve the AI to build community discussions asap.

If you cant fix that i will give the scenario a free community discussion building to all civs. Or consider to make the animal myth requirment free to all civs from start so the AI can build myths directly, that will improve the AI a lot.

stachnie · Dec 7, 2015

SVN 8919, Prehistoric era. Strange things happen when I try to capture AI city:

- when I capture it with Neanderthal Warriors, it gets Barbarian one with my units in

- when I capture it with some other unit (Great Hunter, Stone Axeman), I may kill the last defender and go into the city but it is still AI city.

It is the first time I play with Hide and Seek on, so maybe this is the problem? E.g. I tried to protect a couple of subdued animals with a Great Hunter and they were killed by a duck which did not care about my GH unit and moved into the same place.

S.

Thunderbrd · Dec 7, 2015

Pit2015 said:
Ok i found a big bug/problem! The AI is not building any community discussion building, makeing the AI fall back alot because it takes them to long to build knowledge inherti... so the AI falls back alot in research, they cant build myths this way from the beginning, they are to late, thats also why the AI sucks on education now, they miss those buildings and build them to late, make the AI so that there main priority is to build community discussions building asap, so then can use there hunting awards to build myth buildings and keep up with research!

So again look into the savegame that joseph provided, there you can see the problem, no AI civ has community discussions at all as it looks like, so they miss animal ... requirment to be able to build myths and only some AI civs have knowledge inherti... now very late, so they can build myths to late and they suck on education, looks like no AI civ has a community discussions building, so again:

Look into this savegame file "TEST-GW BC-18750.7z" in http://forums.civfanatics.com/showthread.php?t=552901 page 2, i made all AI civs visible, looks like AI sucks a bit on education now... just klick the civs city and look into the propertys.

Please fix/improve the AI to build community discussions asap.

If you cant fix that i will give the scenario a free community discussion building to all civs. Or consider to make the animal myth requirment free to all civs from start so the AI can build myths directly, that will improve the AI a lot.

I saw a note in the AI the other day that indicated that the AI may not be doing anything to enhance positive properties. That's perhaps at the root of the issue.

There's a patch that can be enacted in the xml... putting an AI weight on the building for now should get them to value it.

We'll need to reprogram the AI to see value in enhancing positive properties though and that may take a bit to address. I'm currently fairly lost when it comes to following and reading some of the property valuation coding though and have already asked for some assistance in that regard. If I can come to understand what its doing and know that it's doing as its intended I may be able to come up with a solution to this and some other factors I'm thinking aren't working quite right where that's concerned.

stachnie said:
SVN 8919, Prehistoric era. Strange things happen when I try to capture AI city:

- when I capture it with Neanderthal Warriors, it gets Barbarian one with my units in

- when I capture it with some other unit (Great Hunter, Stone Axeman), I may kill the last defender and go into the city but it is still AI city.

It is the first time I play with Hide and Seek on, so maybe this is the problem? E.g. I tried to protect a couple of subdued animals with a Great Hunter and they were killed by a duck which did not care about my GH unit and moved into the same place.

S.

The first part is correct and accurate. HN units will, if they do capture a city, capture it for the barbarian player.

The second part is disturbing. You're saying that non-HN units that would normally capture a city aren't capturing at all now? I'll have to look into that asap.

H&S shouldn't be the problem but with the third issue, the Great Hunter trying to protect your subdued animals failed to do so because he was invisible to the duck. Hunters should not be used to protect anything as on H&S, hunters are now a stealth unit in their own right and moreso the Great ones. However, if you DO need to use them for this purpose, use the Standout buildup to remove all invisibility from the unit. That's generally the use for that status promo is to ensure that they will be seen so they can defend.

stachnie · Dec 7, 2015

Thunderbrd said:
The first part is correct and accurate. HN units will, if they do capture a city, capture it for the barbarian player.

Is this a new feature? This is my first game since v.36 so I was not aware of the fact that Neanderthal Warriors are HN (and that captured cities become Barbarian).

H&S shouldn't be the problem but with the third issue, the Great Hunter trying to protect your subdued animals failed to do so because he was invisible to the duck.[...] However, if you DO need to use them for this purpose, use the Standout buildup to remove all invisibility from the unit.

O.K., thanks.

S.

Toffer90 · Dec 7, 2015

stachnie said:
Is this a new feature? This is my first game since v.36 so I was not aware of the fact that Neanderthal Warriors are HN (and that captured cities become Barbarian).

Neanderthal Warriors have been HN for quite some time.
That cities captured by HN units become barbarian was introduced alongside the hide and seek module. Happy gaming. ^^

stachnie · Dec 7, 2015

Toffer90 said:
Neanderthal Warriors have been HN for quite some time.

So I must have overlooked this. It makes them even more useful (if one can build them, of course).

That cities captured by HN units become barbarian was introduced alongside the hide and seek module. Happy gaming. ^^

Interesting feature - it makes HN units strong enough to capture cities even more useful. Thanks!

S.

dummy1929 · Dec 7, 2015

Koshling said:
If Python ever gets invoked on any but the main thread a crash is likely to result because the game engine does not have any multi-threading support. This means that all multi-thread support has to be 'hidden' strictly inside the code we run in the DLL or else all hell breaks loose (usually manifesting as a heap, corruption which fits these symptoms). Since we have no control over what happens in a Python call (what the invoked Python will itself then call) it means it is always unsafe to invoke a Python call on any but the main thread. The DLL code goes to some lengths to avoid this happening (stuff gets queued from the background threads to be invoked on the main thread explicitly), but it may well be we missed a few cases. To address this there is a workaround, and a debugging process (so workaround for a user experiencing it, process to follow for someone trying to fix it):

1) To work around it disable multi-threading. There's a one line setting in the config XML to achieve this, which I'm sure someone can look up in the XML and report back here (set the thread count to 1)

2) To try to get to the bottom of it put a debug breakpoint on the assertion failure that spots non-main-thread Python invocation to find out where in the code its making a Python call from where it shouldn't. Having found that point stop it happening, either by removing the need to call Python there somehow, or by queueing it onto the work queue to be executed by the main thread (there is already code to do this in various places, but I forget the exact details)

Which config xml? I would like to try this.

Toffer90 · Dec 7, 2015

dummy1929 said:
Which config xml? I would like to try this.

All multi-thread configurations are found here:
\Assets\XMLA_New_Dawn_GlobalDefines.xml

Spoiler :

alberts8 · Dec 7, 2015

There are some possible coding issues in the new Ambush command leading to slowdowns at the moment in case anybody wonders about increased turn times.

dummy1929 · Dec 7, 2015

Toffer90 said:

All multi-thread configurations are found here:
\Assets\XMLA_New_Dawn_GlobalDefines.xml

Spoiler :

Thanks!

strategyonly · Dec 7, 2015

alberts2 said:
There are some possible coding issues in the new Ambush command leading to slowdowns at the moment in case anybody wonders about increased turn times.

Thats what i have been saying for sometime now, thx for confirming that . . . .SO

JosEPh_II · Dec 7, 2015

alberts2 said:
There are some possible coding issues in the new Ambush command leading to slowdowns at the moment in case anybody wonders about increased turn times.

Curious, do these affect games that have not activated the Option? Is there even an Option to turn Ambush Off?

JosEPh

Dancing Hoskuld · Dec 7, 2015

Thunderbrd said:
H&S shouldn't be the problem but with the third issue, the Great Hunter trying to protect your subdued animals failed to do so because he was invisible to the duck. Hunters should not be used to protect anything as on H&S, hunters are now a stealth unit in their own right and moreso the Great ones. However, if you DO need to use them for this purpose, use the Standout buildup to remove all invisibility from the unit. That's generally the use for that status promo is to ensure that they will be seen so they can defend.

This is a worrying development, why don't hunters protect their catch any more? What about scouts are they the same? How are you supposed to get those animals you capture back home now? Historically hunters did not take a bunch of fighters out with them when they hunt.

Single Player bugs and crashes v36 plus (SVN) - After the 24th of October 2015

C2C War Dog

C2C Supreme Commander

Emperor

C2C War Dog

C2C War Dog

Deity

C2C War Dog

Emperor

Theorist

C2C War Dog

Theorist

C2C Modder

Theorist

Chieftain

C2C Modder

Emperor

Chieftain

C2C Supreme Commander

TBS WarLord

Deity

Similar threads