1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

[MODCOMP] Unit Statistics

Discussion in 'Civ4 - Mod Components' started by Teg_Navanis, Mar 13, 2006.

  1. Ket

    Ket Composite Of A Composite

    Joined:
    Jul 26, 2006
    Messages:
    900
    Location:
    Austin, TX
    Thank You Sir
     
  2. Caesium

    Caesium Radiant!

    Joined:
    Jan 14, 2006
    Messages:
    526
    I've got some new error messages:

    Code:
    Traceback (most recent call last):
    
      File "CvEventInterface", line 30, in onEvent
    
      File "BugEventManager", line 254, in handleEvent
    
      File "BugEventManager", line 267, in _handleDefaultEvent
    
      File "CvUnitStatisticsEventManager", line 179, in onUnitLost
    
      File "UnitStatisticsUtils", line 518, in onUnitLost
    
      File "UnitStatisticsTools", line 96, in checkHighScoresAllUnits
    
    TypeError: unsubscriptable object
    ERR: Python function onEvent failed, module CvEventInterface
    
    and

    Code:
    Traceback (most recent call last):
    
      File "CvScreensInterface", line 942, in handleInput
    
      File "CvStatisticsScreen", line 733, in handleInput
    
      File "CvStatisticsScreen", line 89, in interfaceScreen
    
      File "CvStatisticsScreen", line 265, in drawHighScoresScreen
    
      File "CvStatisticsScreen", line 318, in drawHighScoreList
    
      File "UnitStatisticsTools", line 202, in checkTop10
    
    TypeError: unsubscriptable object
    ERR: Python function handleInput failed, module CvScreensInterface
    
    UnitStatisticsTools.py isn't changed by me.
     
  3. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    Uncommenting lines 51 and 171 should do the trick.

    (In the file UnitStatisticsTools.py)
     
  4. Caesium

    Caesium Radiant!

    Joined:
    Jan 14, 2006
    Messages:
    526
    Thank you :)
     
  5. suli

    suli Chieftain

    Joined:
    Aug 17, 2005
    Messages:
    10
    can this mod be merged with BUG?
     
  6. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    It's possible; you'd have to manually merge all files which exist in both mods (mostly python ones). These are CvMainInterface.py, CvScreenInterface.py, SdToolKit.py/SdToolKitAdvanced.py, and possibly some more...
     
  7. EmperorFool

    EmperorFool Deity

    Joined:
    Mar 2, 2007
    Messages:
    9,633
    Location:
    Mountain View, California
    Now that I have added a DLL option to BUG via BULL, I am considering adding this mod. What is the performance like for it, especially during wars? I see a few things scanning the code that give me pause.

    First, the log of everything the unit does is kept as one long string. While that should make pickling fast, it will grow considerably over time. Is this a high-priority feature of the mod? I would think the coolest parts are the stats themselves, but I haven't used it yet.

    Second, each time an event happens, each statistic for a unit is modified individually using sdObjectSetVal(). Each time this is done, the script data is unpickled and repickled. When a value is incremented, it requires unpickling once to get the current value and then writing the new value as above. Take the example of moving a unit:

    • Increment Unit movement counter
    • Increment Player movement counter
    • if has cargo
      • Incremenet Unit cargo movement counter
      • Increment Player cargo movement counter
    Ignoring the checks for high scores, this requires 4 unpickles and 2 pickles. It would be far better to store all the stats in a single object so you could unpickle it once, modify all the necessary values, and then repickle it to store it.

    Rewriting this code would go a long way to addressing any performance problems it has, if any.
     
  8. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    I did some (very crude) performance tests recently: Click me.

    Storing all information in the same object was the biggest cause for performance loss in older versions. The number of pickling/unpickling operations may increase if each unit is updated individually, but each operations will be a hundred or a thousand times faster (depending on how many units you are tracking). That's a good trade-off, if you ask me.

    The different statistics can be enabled/disabled in the config file, and movement tracking is disabled by default. It just happens too often and isn't that interesting.

    I rather enjoy looking at the log of some of my units at the end of the game; it's not much of a performance issue since it will only grow long for a handful of units, and even when attacking with them, I don't perceive a slowdown.
     
  9. EmperorFool

    EmperorFool Deity

    Joined:
    Mar 2, 2007
    Messages:
    9,633
    Location:
    Mountain View, California
    I agree, and I wasn't proposing combining units into a single object. Instead, I'm saying to combine all the individual data values of a unit into a single object. It's already stored this way by SD-Toolkit: a dictionary. This would allow you to make multiple updates to a unit's values with only a single pair of unpickle/pickle operations.

    Here's how the same movement tracking in the example above would be handled:

    • Unpickle UnitStats object [high cost]
    • Increment Movement counter [instant]
    • Increment Cargo Movement counter [instant]
    • Pickle UnitStats object [high cost]

    The pickling and unpickling take the longest amount of time, and their cost doesn't change depending on how many values you alter. This allows you to pay that cost exactly once per unit. The same goes for the PlayerStats object. If an event causes multiple high score values to be altered, this could provide the same time savings.

    Actually, this isn't true. The whole data structure is unpickled for every read (sdObjectGetVal) and unpickled and repickled for every write (sdObjectSetVal). Whether your store the units together or separately, you must pay the pickler to access that data. You are correct that splitting them into separate objects reduces the cost of each pickle operation, and this was indeed a smart move.
     
  10. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    I see what you mean. There is indeed some room for improvement in the code.

    On the other hand, I'm happy enough with the performance at the moment and don't intend a complete overhaul for a performance gain that might not even be noticeable.
     
  11. xienwolf

    xienwolf Deity

    Joined:
    Oct 4, 2007
    Messages:
    10,589
    Location:
    Location! Location!
    Didn't know you were still around Teg :) We broke your UnitStats in Fall Further and I might bug you for some assistance in precisely HOW we managed to do so in a few weeks (sorting out other bugs now unfortunately). I'm just not so hot on python processes, and don't much care to learn them all right at the moment (pickles taste good, if they are Dill and not Bread & Butter, and they can conduct electricity, until it cooks them, but I don't think they are meant to be good in programs... that's just me though)
     
  12. EmperorFool

    EmperorFool Deity

    Joined:
    Mar 2, 2007
    Messages:
    9,633
    Location:
    Mountain View, California
    If you ever want to do non-stopwatch performance testing, you might find this Timer class to be pretty handy. I wrote it for BUG to test some of the more beefy data-crunching and initialization to ensure I didn't introduce performance concerns.

    Extracted from BugUtil.py; save as CvTimer.py. I changed it to use CvUtil.pyPrint() to log results.

    Code:
    # CvTimer.py
    
    import time
    import CvUtil
    
    class Timer:
    	"""
    	Stopwatch for timing code execution and logging the results.
    	
    	timer = CvTimer.Timer('function')
    	... code to time ...
    	timer.log()
    	
    	In a loop, log() will display each iteration's time. Since Timers are started
    	when created, call reset() before entering the loop or pass in False.
    	Use logTotal() at the end if you want to see the sum of all iterations.
    	
    	timer = CvTimer.Timer('draw loop', False)
    	for/while ...
    		timer.start()
    		... code to time ...
    		timer.log()
    	timer.logTotal()
    	
    	A single Timer can be reused for timing loops without creating a new Timer
    	for each iteration by calling restart().
    	"""
    	def __init__(self, item, start=True):
    		"""Starts the timer."""
    		self._item = item
    		self.reset()
    		if start:
    			self.start()
    	
    	def reset(self):
    		"""Resets all times to zero and stops the timer."""
    		self._initial = None
    		self._start = None
    		self._time = 0
    		self._total = 0
    		return self
    	
    	def start(self):
    		"""Starts the timer or starts it again if it is already running."""
    		self._start = time.clock()
    		if self._initial is None:
    			self._initial = self._start
    		return self
    	
    	def restart(self):
    		"""Resets all times to zero and starts the timer."""
    		return self.reset().start()
    	
    	def stop(self):
    		"""
    		Stops the timer if it is running and returns the elapsed time since start,
    		otherwise returns 0.
    		"""
    		if self.running():
    			self._final = time.clock()
    			self._time = self._final - self._start
    			self._total += self._time
    			self._start = None
    			return self._time
    		return 0
    	
    	def running(self):
    		"""Returns True if the timer is running."""
    		return self._start is not None
    	
    	def time(self):
    		"""Returns the most recent timing or 0 if none has completed."""
    		return self._time
    	
    	def total(self):
    		"""Returns the sum of all the individual timings."""
    		return self._total
    	
    	def span(self):
    		"""Returns the span of time from the first start() to the last stop()."""
    		if self._initial is None:
    			CvUtil.pyPrint("Warning: called span() on a Timer that has not been started")
    			return 0
    		elif self._final is None:
    			return time.clock() - self._initial
    		else:
    			return self._final - self._initial
    	
    	def log(self, extra=None):
    		"""
    		Stops the timer and logs the time of the current timing.
    		
    		This is the same as calling logTotal() or logSpan() for the first time.
    		"""
    		self.stop()
    		return self._log(self.time(), extra)
    	
    	def logTotal(self, extra="total"):
    		"""
    		Stops the timer and logs the sum of all timing steps.
    		
    		This is the same as calling log() or logSpan() for the first time.
    		"""
    		self.stop()
    		return self._log(self.total(), extra)
    	
    	def logSpan(self, extra=None):
    		"""
    		Stops the timer and logs the span of time covering all timings.
    		
    		This is the same as calling log() or logTotal() for the first time.
    		"""
    		self.stop()
    		return self._log(self.span(), extra)
    	
    	def _log(self, runtime, extra):
    		"""Logs the passed in runtime value."""
    		if extra is None:
    			CvUtil.pyPrint("Timer - %s took %d ms" % (self._item, 1000 * runtime))
    		else:
    			CvUtil.pyPrint("Timer - %s [%s] took %d ms" % (self._item, str(extra), 1000 * runtime))
    		return self
    
    You could time AI turns pretty easily by creating and starting a timer for each AI in onBeginPlayerTurn() and stopping and logging it in onEndPlayerTurn(). I believe that the AI does all its moves between these events, unlike human players who move all their units before BeginPlayerTurn is fired.

    Code:
    import CvTimer
    
    ...
    
    def onBeginPlayerTurn(self, argsList):
    	'Called at the beginning of a players turn'
    	iGameTurn, iPlayer = argsList
    	global turnTimer
    	turnTimer = CvTimer.Timer("Turn for Player " + iPlayer)
    
    def onEndPlayerTurn(self, argsList):
    	'Called at the end of a players turn'
    	iGameTurn, iPlayer = argsList
    	global turnTimer
    	turnTimer.log()
    
    This would allow you to compare with and without UnitStats. To time the UnitStats code by itself would require more work.

    • Add False to the Timer() constructor call above so it is created without being started.
    • Add calls to turnTimer.start() and stop() at the top and bottom respectively of each UnitStatsUtil main function.
      Code:
      def logUnitCreation(self, objUnit, iPlayerID):
          turnTimer.start()
          ...
          turnTimer.stop()
      
      You'd need to make sure it has access to the turnTimer created in the begin/end player turn events via importing, and make sure that each "return" in the functions calls stop() before doing so.
    • Change log() above to logTotal() so it adds up all the UnitStats calls during the turn.
     
  13. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    I'm not really active anymore, but still check for new versions of FfH or posts in subscribed threads occasionally :)

    I just had a look at the Fall Further download, and without trying it out (working on different OS atm), I can tell you that there are two settings in PythonCallbackDefines.xml that will definitely mess up UnitStatistics: USE_ON_UNIT_LOST_CALLBACK and USE_COMBAT_RESULT_CALLBACK both have to be set to 1.
     
  14. xienwolf

    xienwolf Deity

    Joined:
    Oct 4, 2007
    Messages:
    10,589
    Location:
    Location! Location!
    Yeah I mostly assume it is callback related. I need to sit down sometime and isolate all areas where a callback is needed and ensure that they specifically link to a UnitStats callback, seperate from the normal ones (one exists, I just don't think it is included everywhere it is required right now)
     
  15. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    Exactly one year after the last update, a new version is up. Nothing changed except for compatibility with 3.19.
     
  16. Arian

    Arian No more ghostbusting!!

    Joined:
    May 10, 2008
    Messages:
    2,088
    Location:
    The Netherlands
    @Teg Navanis:
    USE_COMBAT_RESULT_CALLBACK isn't defined in your own PythonCallbackDefines.xml :confused:
     
  17. Teg_Navanis

    Teg_Navanis King

    Joined:
    Jan 21, 2006
    Messages:
    737
    It doesn't exist in BtS. Kael added some switches to Fall from Heaven so that python could be disabled wherever possible (which apparently speeds the game up a bit).
     
  18. Afforess

    Afforess The White Wizard

    Joined:
    Jul 31, 2007
    Messages:
    12,239
    Location:
    Austin, Texas
    Yes it does.

    I'm completely paranoid with python callbacks (I went as far to add 15 more switches in the SDK and then turned them all off), how much is the callback this modcomp uses going to hurt me performance-wise? Has anyone ever run some concrete profile tests?
     
  19. stolenrays

    stolenrays Deity

    Joined:
    Aug 2, 2009
    Messages:
    2,061
    I get the following python exception when merging. Any suggestions?

    I think my main problem is that I can't figure out how to merge my mod, which also has a customeventmanager.
     

    Attached Files:

  20. JoeyB98

    JoeyB98 King of Ohio

    Joined:
    Sep 19, 2009
    Messages:
    383
    Location:
    Earth
    When I use this mod in Vanilla civ, the main interface disappears. Any idea why this might be happening?
     

Share This Page