[MODCOMP] Unit Statistics

Thank You Sir

 

I've got some new error messages:

Code:

Traceback (most recent call last):

  File "CvEventInterface", line 30, in onEvent

  File "BugEventManager", line 254, in handleEvent

  File "BugEventManager", line 267, in _handleDefaultEvent

  File "CvUnitStatisticsEventManager", line 179, in onUnitLost

  File "UnitStatisticsUtils", line 518, in onUnitLost

  File "UnitStatisticsTools", line 96, in checkHighScoresAllUnits

TypeError: unsubscriptable object
ERR: Python function onEvent failed, module CvEventInterface

and

Code:

Traceback (most recent call last):

  File "CvScreensInterface", line 942, in handleInput

  File "CvStatisticsScreen", line 733, in handleInput

  File "CvStatisticsScreen", line 89, in interfaceScreen

  File "CvStatisticsScreen", line 265, in drawHighScoresScreen

  File "CvStatisticsScreen", line 318, in drawHighScoreList

  File "UnitStatisticsTools", line 202, in checkTop10

TypeError: unsubscriptable object
ERR: Python function handleInput failed, module CvScreensInterface

UnitStatisticsTools.py isn't changed by me.

 

Uncommenting lines 51 and 171 should do the trick.

(In the file UnitStatisticsTools.py)

 

Thank you

 

can this mod be merged with BUG?

 

It's possible; you'd have to manually merge all files which exist in both mods (mostly python ones). These are CvMainInterface.py, CvScreenInterface.py, SdToolKit.py/SdToolKitAdvanced.py, and possibly some more...

 

Now that I have added a DLL option to BUG via BULL, I am considering adding this mod. What is the performance like for it, especially during wars? I see a few things scanning the code that give me pause.

First, the log of everything the unit does is kept as one long string. While that should make pickling fast, it will grow considerably over time. Is this a high-priority feature of the mod? I would think the coolest parts are the stats themselves, but I haven't used it yet.

Second, each time an event happens, each statistic for a unit is modified individually using sdObjectSetVal(). Each time this is done, the script data is unpickled and repickled. When a value is incremented, it requires unpickling once to get the current value and then writing the new value as above. Take the example of moving a unit:

• Increment Unit movement counter
• Increment Player movement counter
• if has cargo
  • Incremenet Unit cargo movement counter
  • Increment Player cargo movement counter

Ignoring the checks for high scores, this requires 4 unpickles and 2 pickles. It would be far better to store all the stats in a single object so you could unpickle it once, modify all the necessary values, and then repickle it to store it.

Rewriting this code would go a long way to addressing any performance problems it has, if any.

 

I did some (very crude) performance tests recently: Click me.

Storing all information in the same object was the biggest cause for performance loss in older versions. The number of pickling/unpickling operations may increase if each unit is updated individually, but each operations will be a hundred or a thousand times faster (depending on how many units you are tracking). That's a good trade-off, if you ask me.

The different statistics can be enabled/disabled in the config file, and movement tracking is disabled by default. It just happens too often and isn't that interesting.

I rather enjoy looking at the log of some of my units at the end of the game; it's not much of a performance issue since it will only grow long for a handful of units, and even when attacking with them, I don't perceive a slowdown.

 

I agree, and I wasn't proposing combining units into a single object. Instead, I'm saying to combine all the individual data values of a unit into a single object. It's already stored this way by SD-Toolkit: a dictionary. This would allow you to make multiple updates to a unit's values with only a single pair of unpickle/pickle operations.

Here's how the same movement tracking in the example above would be handled:

• Unpickle UnitStats object [high cost]
• Increment Movement counter [instant]
• Increment Cargo Movement counter [instant]
• Pickle UnitStats object [high cost]

The pickling and unpickling take the longest amount of time, and their cost doesn't change depending on how many values you alter. This allows you to pay that cost exactly once per unit. The same goes for the PlayerStats object. If an event causes multiple high score values to be altered, this could provide the same time savings.

Actually, this isn't true. The whole data structure is unpickled for every read (sdObjectGetVal) and unpickled and repickled for every write (sdObjectSetVal). Whether your store the units together or separately, you must pay the pickler to access that data. You are correct that splitting them into separate objects reduces the cost of each pickle operation, and this was indeed a smart move.

 

I see what you mean. There is indeed some room for improvement in the code.

On the other hand, I'm happy enough with the performance at the moment and don't intend a complete overhaul for a performance gain that might not even be noticeable.

 

Didn't know you were still around Teg We broke your UnitStats in Fall Further and I might bug you for some assistance in precisely HOW we managed to do so in a few weeks (sorting out other bugs now unfortunately). I'm just not so hot on python processes, and don't much care to learn them all right at the moment (pickles taste good, if they are Dill and not Bread & Butter, and they can conduct electricity, until it cooks them, but I don't think they are meant to be good in programs... that's just me though)

 

If you ever want to do non-stopwatch performance testing, you might find this Timer class to be pretty handy. I wrote it for BUG to test some of the more beefy data-crunching and initialization to ensure I didn't introduce performance concerns.

Extracted from BugUtil.py; save as CvTimer.py. I changed it to use CvUtil.pyPrint() to log results.

Code:

# CvTimer.py

import time
import CvUtil

class Timer:
	"""
	Stopwatch for timing code execution and logging the results.
	
	timer = CvTimer.Timer('function')
	... code to time ...
	timer.log()
	
	In a loop, log() will display each iteration's time. Since Timers are started
	when created, call reset() before entering the loop or pass in False.
	Use logTotal() at the end if you want to see the sum of all iterations.
	
	timer = CvTimer.Timer('draw loop', False)
	for/while ...
		timer.start()
		... code to time ...
		timer.log()
	timer.logTotal()
	
	A single Timer can be reused for timing loops without creating a new Timer
	for each iteration by calling restart().
	"""
	def __init__(self, item, start=True):
		"""Starts the timer."""
		self._item = item
		self.reset()
		if start:
			self.start()
	
	def reset(self):
		"""Resets all times to zero and stops the timer."""
		self._initial = None
		self._start = None
		self._time = 0
		self._total = 0
		return self
	
	def start(self):
		"""Starts the timer or starts it again if it is already running."""
		self._start = time.clock()
		if self._initial is None:
			self._initial = self._start
		return self
	
	def restart(self):
		"""Resets all times to zero and starts the timer."""
		return self.reset().start()
	
	def stop(self):
		"""
		Stops the timer if it is running and returns the elapsed time since start,
		otherwise returns 0.
		"""
		if self.running():
			self._final = time.clock()
			self._time = self._final - self._start
			self._total += self._time
			self._start = None
			return self._time
		return 0
	
	def running(self):
		"""Returns True if the timer is running."""
		return self._start is not None
	
	def time(self):
		"""Returns the most recent timing or 0 if none has completed."""
		return self._time
	
	def total(self):
		"""Returns the sum of all the individual timings."""
		return self._total
	
	def span(self):
		"""Returns the span of time from the first start() to the last stop()."""
		if self._initial is None:
			CvUtil.pyPrint("Warning: called span() on a Timer that has not been started")
			return 0
		elif self._final is None:
			return time.clock() - self._initial
		else:
			return self._final - self._initial
	
	def log(self, extra=None):
		"""
		Stops the timer and logs the time of the current timing.
		
		This is the same as calling logTotal() or logSpan() for the first time.
		"""
		self.stop()
		return self._log(self.time(), extra)
	
	def logTotal(self, extra="total"):
		"""
		Stops the timer and logs the sum of all timing steps.
		
		This is the same as calling log() or logSpan() for the first time.
		"""
		self.stop()
		return self._log(self.total(), extra)
	
	def logSpan(self, extra=None):
		"""
		Stops the timer and logs the span of time covering all timings.
		
		This is the same as calling log() or logTotal() for the first time.
		"""
		self.stop()
		return self._log(self.span(), extra)
	
	def _log(self, runtime, extra):
		"""Logs the passed in runtime value."""
		if extra is None:
			CvUtil.pyPrint("Timer - %s took %d ms" % (self._item, 1000 * runtime))
		else:
			CvUtil.pyPrint("Timer - %s [%s] took %d ms" % (self._item, str(extra), 1000 * runtime))
		return self

You could time AI turns pretty easily by creating and starting a timer for each AI in onBeginPlayerTurn() and stopping and logging it in onEndPlayerTurn(). I believe that the AI does all its moves between these events, unlike human players who move all their units before BeginPlayerTurn is fired.

Code:

import CvTimer

...

def onBeginPlayerTurn(self, argsList):
	'Called at the beginning of a players turn'
	iGameTurn, iPlayer = argsList
	global turnTimer
	turnTimer = CvTimer.Timer("Turn for Player " + iPlayer)

def onEndPlayerTurn(self, argsList):
	'Called at the end of a players turn'
	iGameTurn, iPlayer = argsList
	global turnTimer
	turnTimer.log()

This would allow you to compare with and without UnitStats. To time the UnitStats code by itself would require more work.

• Add False to the Timer() constructor call above so it is created without being started.
• Add calls to turnTimer.start() and stop() at the top and bottom respectively of each UnitStatsUtil main function.
  
  Code:

  def logUnitCreation(self, objUnit, iPlayerID):
      turnTimer.start()
      ...
      turnTimer.stop()
  

  You'd need to make sure it has access to the turnTimer created in the begin/end player turn events via importing, and make sure that each "return" in the functions calls stop() before doing so.
• Change log() above to logTotal() so it adds up all the UnitStats calls during the turn.

 

I'm not really active anymore, but still check for new versions of FfH or posts in subscribed threads occasionally

I just had a look at the Fall Further download, and without trying it out (working on different OS atm), I can tell you that there are two settings in PythonCallbackDefines.xml that will definitely mess up UnitStatistics: USE_ON_UNIT_LOST_CALLBACK and USE_COMBAT_RESULT_CALLBACK both have to be set to 1.

 

Yeah I mostly assume it is callback related. I need to sit down sometime and isolate all areas where a callback is needed and ensure that they specifically link to a UnitStats callback, seperate from the normal ones (one exists, I just don't think it is included everywhere it is required right now)

 

Exactly one year after the last update, a new version is up. Nothing changed except for compatibility with 3.19.

 

@Teg Navanis:
USE_COMBAT_RESULT_CALLBACK isn't defined in your own PythonCallbackDefines.xml

 

It doesn't exist in BtS. Kael added some switches to Fall from Heaven so that python could be disabled wherever possible (which apparently speeds the game up a bit).

 

Yes it does.

I'm completely paranoid with python callbacks (I went as far to add 15 more switches in the SDK and then turned them all off), how much is the callback this modcomp uses going to hurt me performance-wise? Has anyone ever run some concrete profile tests?

 

I get the following python exception when merging. Any suggestions?

I think my main problem is that I can't figure out how to merge my mod, which also has a customeventmanager.

 

When I use this mod in Vanilla civ, the main interface disappears. Any idea why this might be happening?