RFC - the OOP edition (programming mumbo-jumbo)

[Baldyr]

I'm currently learning the finer points of Python programming and CivIV modding. One thing I just realized is that pickling (even with the C++ module cPickle) and storing values in the CyGame instance is very slow. And Rhye uses pickling to store a lot of values!

Since I've also read up on Object-Oriented Programming I was able to simply replace a call to one of Rhye's un-pickling functions with storing that value in the class instance. This would seem much more efficient, not?

So it could be possible to optimize RFC quite a bit by taking out some of the most used pickling functions and replacing them with class fields. Like instead of storing the list lStability in the dictionary scriptDict, pickling the whole dictionary, and then saving it in the game instance - it could simply be a field of the Stability class: Stability.lStability (and used like self.lStability).

Makes sense?

 

[civ_king]

SO

pickling is slow
therefore you want to do it with out pickling?
then it becomes faster?

 

[Baldyr]

Hopefully, yes.

Pickling means that you turn something (like a dictionary with various values) into a string (a series of letters). The way this is useful in CivIV is that you then can store this string in the game instance (the actual game session) and thus the data will get stored on save - and can be retrieved again on load-up. But it should be sufficient to only do this on save/load, not whenever you need to store or fetch a value during game-play.

But I'm no expert, I'm just wondering if I got all this right...

 

[merijn_v1]

Rhye made the pickles much faster. He imported pickles as cpickles. If you can make it faster, go ahead. I would love to load America (or the Dutch in RFCE) in just 30 minutes.

 

[Baldyr]

Rhye made the pickles much faster. He imported pickles as cpickles. If you can make it faster, go ahead. I would love to load America (or the Dutch in RFCE) in just 30 minutes.

Yeah, I figured out that Rhye uses cPickle (the C++ equivalent of the Python pickle module) in RFC. While timing some of my own code I happened to notice how much of the lag in RFC was actually caused by pickling...

I wasn't really proposing to rewrite the whole mod or anything, but rather check if I got the principle right. Then if I do end up doing something along these lines - and it would speed up autoplay - all the better!

 

[Ready_set_go]

Try your best. It would be amazing to get this right, and then speed up the game overall. It would be amazing, as they said, to load America in 30 minutes.

 

[Baldyr]

I might actually try to time how long all the pickling takes on a given game turn. This would of course vary from turn to turn (like when stability is calculated and set every third and sixth turn). But it would be interesting to see if it adds up to full seconds or if its less than that.

Lets see... 30 min to load 345 turns allows for 5,21 seconds per game turn - on average. I don't think there is more pickling involved as the game progresses though, but its rather the amount of cities and units and stuff that needs to be processed. So even shaving off a full second from pickling might not do the trick.

I haven't timed how long a America start takes on my system, but how long does it take for you guys? Like a full hour?

I also wonder if it wouldn't be possible to have the game skip some processes during autoplay, or at least perform them less often. Because as long as the outcome is believable and not entirely predestined it shouldn't matter if the AI has actually "played by the rules" or not...

 

[civ_king]

about an hour though this is probably because I have it autosave every turn

 

[Baldyr]

about an hour though this is probably because I have it autosave every turn

Yeah, autosaving adds to the lag... I would probably turn off autosaves while waiting for an America game to start.

 

[civ_king]

Do you know. How much it adds to the time to generate?

 

[Baldyr]

Do you know. How much it adds to the time to generate?

No, I think this would have to be timed in the SDK (involves C++ and ability to recompile the DLL). But I do believe that saving a game session gets increasingly time consuming with the number of game objects that need to be saved along with it.

So while the lag in between turns increases with time, so does the time it takes to save the game. You could test it yourself, if you don't need to use your computer for a couple of hours.

 

[Omega124]

I think that removing the autosave every turn would do wonders against the lag when loading up certain civs.

 

[Baldyr]

I found a clever way of timing the pickling done by RFC and am busy generating an America start. This far I can report that the total pickling time per game turn start off at about 150 milliseconds and but quickly doubles and triples. By the Middle-Ages its regularly, but not every turn, at 1000 ms (thats one full second) and rising...

I can't run through the whole thing at this sitting, but I will try to get some 1800th century times soon. I guess it will average at one second per turn however. (It has already passed the 2 seconds mark occasionally.)

edit: Looking at the debug log from the entire autoplay I can see that the total time of pickling processes on any given turn varies from 0,5 to 4 seconds - per turn - in the 1800th century. I also noticed a mistake I made and thus I haven't even been timing one of the main modules that performs pickling (RFCUtils)... I might do another test some day and count the total pickling time for the entire autoplay - and also calculate a average time per turn. But if we play with the notion that pickling takes up 1 second lag on average, then it would add up to about 345 seconds (almost 6 minutes) for the America spawn.

 

[Baldyr]

Continuing the technical discussion, here's how I think pickling could be avoided other than on pre-save:

The Stored Data module already has a class called StoredData, so it would be easy to simply make the entire scriptDict into a field of that class:

StoredData.scriptDict

The StoredData module is already imported by CvRFCEventHandler that assigns a StoredData instance to the field:

CvRFCEventHandler.data

So the scriptDict could be accessed through:

CvRFCEventHandler.data.scriptDict

The trick is to import the CvRFCEventHandler instance to all the other modules now performing data storage operations via pickling. Because then all the pickling code could be substituted with something like:

        def getStability( self, iCiv ):
                scriptDict = CvRFCEventHandler.data.scriptDict
                return scriptDict['lStability'][iCiv]

and:

        def setStability( self, iCiv, iNewValue ):
                scriptDict = CvRFCEventHandler.data.scriptDict
                scriptDict['lStability'][iCiv] = iNewValue

And this would be much quicker! The only pickling would be done by adding the onPreSave event to the Event Handler. This would only be performed when the game is saved and would probably only take like 1/10 second...

I'd have to wrap my head around how exactly the Event Handler operates if I were to attempt this, but it should be doable. (I actually did something similar last night when I replaced the pickle name in all the RFC modules with an imported custom module that took over all pickling duties with cPickle - while timing the processes.)

 

[Baldyr]

Ok, I actually think I've got it!

Since all the modules contain classes and the Event Handler stores each class instance as a field, all I have to do is pass the Event Handler instance as a argument to each class constructor. Like this:

self.rnf = RiseAndFall.RiseAndFall(self)

Then add the actual constructor to each module:

        def __init__ (self, CvRFCEventHandler):
                self.scriptDict = CvRFCEventHandler.data.scriptDict

And finally, change the data storage methods into:

        def getStability( self, iCiv ):
                return self.scriptDict['lStability'][iCiv]

and respectively:

        def setStability( self, iCiv, iNewValue ):
                self.scriptDict['lStability'][iCiv] = iNewValue

If I haven't overlooked anything, this seems to be almost too simple... So I think I'll have to give this a try, then!

 

[Baldyr]

So, I tested my theory in practice and it turns out that it isn't sufficient to build this from the Event Handler level and down. I should have made the scriptDict an attribute of the CvEventManager instance instead. Also, I have no idea as how to make the interface and the different screens access the values stored in the scriptDict.

Is the CvScreensInterface module called directly from the SDK/DLL?

I now realize that pickling the scriptDict and storing it as a string in the CyGame object is useful for all the various systems to access the same stored values...

I guess I could have a separate setup for the stuff not connected to the Event Handler/Manager, that uses the tried and tested pickling setup. Then the scriptDict would only be pickled and stored once every turn - and thus the state of affairs would only be updated in-between turns. I doubt this sacrifice is worth a second or two worth of lag per turn in the late game...

I'll have to think about this further and perhaps revisit the idea sometime in the future, when I have a better understanding of how the game is built.

 

[Baldyr]

I actually think I managed to resolve the issues with this setup. I'm looking at about 10-15% decrease in lag in between turns - more if I disable autosaves in the CivilizationIV.ini file.

The replacement Python files are attached below - they go in the \Rhye's and Fall of Civilization\Assets\Python\ folder. But anybody who wants to test this should also enable Python exceptions (error messages) in the CivilizationIV.ini file (found in \My Documents\My Games\Beyond the Sword\):

HidePythonExceptions = 0
LoggingEnabled = 1

Because not doing this, you won't be able to tell whether or not there are any glitches. And then any testing would be a waste of time.

Please report back with the results of any testing!

 

[Baldyr]

...and has anyone actually tested this successfully? (I see 10 downloads.)

If you installed the files and everything is working just a usual (and you have pretty much played an entire game) the answer would be a yes!

Hopefully it was slightly faster too, but this ca be a bit subjective unless you actually time it!

 

[Sendou]

Brief test - I just installed this mod and loaded the Persians on the new 'Unlocked' mode. It works fine (er, at least until your first turn) and loads a little faster (not spectacularly so, 10% like you said). I disabled autosaving in the .ini file as well.

However, I notice that my Immortals are footsoldiers and Alexander the Great has a new skin. I like it, but you should have said that there were other mods included.

Thankyou, this mod looks promising already.

EDIT: OK, I've now tried to play several games as teh Persians. The problem is, sometimes it loads fine, but other times it gives me this:


Uploaded with ImageShack.us

Or this:


Uploaded with ImageShack.us

On the times when it loads fine, though, I haven't noticed a single problem. I eventually switched to the Mongols, conquered my own Persia - worked totally fine.

 

[Baldyr]

You probably just have the wrong mod version, or something.

The principle is also implemented in RFC: Epic/Marathon - try using that instead.