Increase your mod speed by factor 3! Remove PythonCallbackDefines.xml and all related

[Cybah]

I've finally figured out the python performance killer:

PythonCallbackDefines.xml is decreasing your game speed by factor 3!


Tests (huge map, 10 AI players, 1 human player on AI Autoplay):

Vanilla BTS + Better BTS AI + BUG Mod: Reaching round 83 after 2 minutes

Mod with PythonCallbackDefines + Better BTS AI + BUG Mod: Reaching round 83 after 7 minutes

Mod without PythonCallbackDefines + Better BTS AI + BUG Mod: Reaching round 83 after 2 minutes



All of your python additions will NOT slow your mod as long as they don't need an entry within this file!

I have ton's of custom python addons in my mod which do not need the Callback file - and the speed is NOT increasing by removing those custom features.

Try it out and enjoy the bigger maps again.

 

[primordial stew]

Good to know

1 tip I picked up from one of Zappara's postings was to use cPickle instead of pickle. It wouldn't be hard to measure but I didn't. Just happy that it seemed faster.

Has anyone tried using the python profiler to see if there is anything interesting from that?

 

[Cybah]

Update: Route Restricter like the following (which need a callback) seem to be the root of the evil:

	def unitCannotMoveInto(self,argsList):
		ePlayer = argsList[0]
		iUnitId = argsList[1]
		iPlotX = argsList[2]
		iPlotY = argsList[3]
        
        ## Locomotive Route Restriction
		pUnit = gc.getPlayer(ePlayer).getUnit(iUnitId)
		iTrainType = gc.getInfoTypeForString ( "UNIT_RAILROAD_ARTILLERY" )
		if pUnit.getUnitType() == iTrainType:
			pPlot = CyMap().plot(iPlotX, iPlotY)
			iRoute = gc.getInfoTypeForString ( "ROUTE_RAILROAD" )
			if pPlot.getRouteType() != iRoute:
				return True
        
        ## End Edit
		return False

	<Define>
		<DefineName>USE_UNIT_CANNOT_MOVE_INTO_CALLBACK</DefineName>
		<iDefineIntVal>1</iDefineIntVal>
	</Define>

This alone is decreasing my mod's speed by over 200%. I guess it gets even worse with more units on the map.

 

[Cybah]

	<Define>
		<DefineName>USE_CAN_TRAIN_CALLBACK</DefineName>
		<iDefineIntVal>1</iDefineIntVal>
	</Define>

is decreasing the game speed by up to 50% (depending on the code in gameutils).

 

[EmperorFool]

Using cannotMoveInto() is going to chew huge amounts of performance because it is used during path-finding which is called a lot as more units get built. You would do well to have a C++ hacker create an XML attribute to limit units to routes so that code can be moved out of Python. The real price is more likely the call to Python rather than the Python being executed. Whenever you need to interface between layers (C++ and Python) there is a certain amount of overhead you pay regardless of the complexity of the called function.

 

[davidlallen]

In terms of sdk mods to restrict movement on certain routes, you may wish to look at JRouteNative. It hasn't been updated to 3.19, and in fact I would love if somebody did update it since Fury Road relies on it.

 

[keldath]

i can try and update this...

 

[zappara]

Good to know

1 tip I picked up from one of Zappara's postings was to use cPickle instead of pickle. It wouldn't be hard to measure but I didn't. Just happy that it seemed faster.

Has anyone tried using the python profiler to see if there is anything interesting from that?

According to Python docs cPickle can be up to 1000 times faster than pickle. I can't remember who originally suggested this change but saying that it could lead up to 50% faster turns was enough reason for me to use this optimization.

Update: Route Restricter like the following (which need a callback) seem to be the root of the evil:

	def unitCannotMoveInto(self,argsList):
		ePlayer = argsList[0]
		iUnitId = argsList[1]
		iPlotX = argsList[2]
		iPlotY = argsList[3]
        
        ## Locomotive Route Restriction
		pUnit = gc.getPlayer(ePlayer).getUnit(iUnitId)
		iTrainType = gc.getInfoTypeForString ( "UNIT_RAILROAD_ARTILLERY" )
		if pUnit.getUnitType() == iTrainType:
			pPlot = CyMap().plot(iPlotX, iPlotY)
			iRoute = gc.getInfoTypeForString ( "ROUTE_RAILROAD" )
			if pPlot.getRouteType() != iRoute:
				return True
        
        ## End Edit
		return False

You could optimize the above code by moving

iTrainType = gc.getInfoTypeForString ( "UNIT_RAILROAD_ARTILLERY" ) and iRoute = gc.getInfoTypeForString ( "ROUTE_RAILROAD" )

call to init section (and/or onLoadGame function) and place those to a new variables ie.

self.RAILROADARTILLERY = gc.getInfoTypeForString ( "UNIT_RAILROAD_ARTILLERY" )
self.ROUTERAILROAD = gc.getInfoTypeForString ( "ROUTE_RAILROAD" )

and then change the code to format

	def unitCannotMoveInto(self,argsList):
		ePlayer = argsList[0]
		iUnitId = argsList[1]
		iPlotX = argsList[2]
		iPlotY = argsList[3]
        
        ## Locomotive Route Restriction
		pUnit = gc.getPlayer(ePlayer).getUnit(iUnitId)
		if pUnit.getUnitType() == self.RAILROADARTILLERY:
			pPlot = CyMap().plot(iPlotX, iPlotY)
			if pPlot.getRouteType() != self.ROUTERAILROAD:
				return True
        
        ## End Edit
		return False

So instead of using gc.getInfoTypeForString() function to retrieve the same info type on every turn on every unit you just retrieve it only once, at the start of the game or when the game is loaded and then compare two integer values in those checks, which is faster method. If you have lots of similar checks in your mod, changing them this way can get you some speed increase.

 

[Agent327]

You can't just remove the PythonCallbackDefines.xml; I tried it with MOO2Civ and it definitely improves speed... but it lets me build buildings that already have been built and reduces build times as well (plus it removed a building from the build list).

 

[Cybah]

it's depending on your custom python code. sure, you will have to remove some code/features of your mod. I guess there are buildings in your mod you can only build with special conditions which got set by python.

 

[Agent327]

Quite, according to deanej a lot of FF features were python-defined, unfortunately. So I guess it'll be of little help for FF-based mods...

 

[EmperorFool]

Get some smart cookie to port the FF Python features to the SDK.

 

[Dom Pedro II]

Ok, to clarify, you mean that it's moving more slowly when you set these callbacks to active right? Because I can't see why, if the game isn't actually calling the python functions, it would be decreasing the speed that much.

 

[Cybah]

it's depending on your python code. if you don't use any code which needs a callback entry (many python don't need this file at all), there won't be a big change in game speed. but I recommend to use as few as possible callbacks. some callbacks are worse than others.