Using scriptData vs. new variables in DLL - performance

[embryodead]

Suppose you read/write a few variables in Python hundreds of times per turn using scriptData and cPickle. This takes a few seconds each turn. Would it be much faster to add those variables to CvPlayer, along with get/set functions exposed to Python? Common sense says yes, as there would direct access with no serialization (and scriptData is a few kB), but I'm not sure if this is worth the effort.

 

[Afforess]

If you are saving that much information, yes, I recommend you start saving it in the DLL. get/set calls take up effectively no time, as does the read/write calls.

 

[davidlallen]

There are three separate parts to the question.

1. Pickling/unpickling. This is time consuming and should only be done on a load/save respectively. If you find "seconds per turn" happening, please make sure pickling is not done on each data access or each turn. This would be a huge waste.

2. Storing the data. If you are storing simple data, I am sure it is fractionally faster to store them in C++ data members (in the sdk) instead of in python; but here we would be counting microseconds. I doubt this will make a perceptible difference.

3. Operating on the data. Non-programmers often over generalize, "Python is always too slow". If you have some complex algorithm which needs multiple lookups and list traversals, python is slow. However, in most cases you can find a simpler algorithm or a slightly different design, which is fast regardless of what language you use. If you absolutely have to redo some time consuming calculation many times per turn, you are better off doing it in C++ code (in the sdk) instead of in python.

 

[Baldyr]

Since I reckon we're talking about RFC here, this is my own proposal on how to get rid of the endless pickling.

First we make the scriptDict a field of the StoredData instance:

Spoiler :

class StoredData:

        def __init__():
                self.scriptDict = {      #------------RiseAndFall
                            'iNewCiv': -1,
                            'iNewCivFlip': -1,
                            'iOldCivFlip': -1,
                            'tTempTopLeft': -1,
                            'tTempBottomRight': -1,
                            'iSpawnWar': 0, #if 1, add units and declare war. If >=2, do nothing
                            'bAlreadySwitched': False,
                            'lColonistsAlreadyGiven': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #active players
                            'lAstronomyTurn': [500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500], #active players
                            'lNumCities': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players to contain Byzantium too
                            'lLastTurnAlive': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players to contain Byzantium too
                            'lSpawnDelay': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #active players
                            'lFlipsDelay': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'iBetrayalTurns': 0,
                            'lLatestRebellionTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'iRebelCiv': 0,
                            'lExileData': [-1, -1, -1, -1, -1],
                            'tTempFlippingCity': -1,
                            'lCheatersCheck': [0, -1],
                            'lBirthTurnModifier': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lDeleteMode': [-1, -1, -1], #first is a bool, the other values are capital coordinates
                            'lFirstContactConquerors': [0, 0, 0], #maya, inca, aztecs
                            'bCheatMode': False,
                            #------------Religions
                            'iSeed': -1,
                            #------------UP
                            'iImmigrationTurnLength': 0,
                            'iImmigrationCurrentTurn': 0,
                            'iLatestFlipTurn': 0,
                            'lLatestRazeData': [-1, -1, -1, -1, -1],
                            #------------AIWars
                            'lAttackingCivsArray': [0, 0, 0, 0, 0, -1, 0, -1, 0, 0, 0, -1, -1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, -1, 0, 0],
                            'iNextTurnAIWar': -1,
                            #------------Congresses
                            'bCongressEnabled': False,
                            'iCivsWithNationalism': 0,
                            'bUNbuilt': False,
                            'lInvitedNations': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                            'lVotes': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lTempActiveCiv': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                            'lTempReqCity': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                            'iLoopIndex': 0,
                            'lTempReqCityHuman': [-1, -1, -1, -1, -1],
                            'tempReqCityNI': -1,
                            'tempActiveCivNI': -1,
                            'lTempAttackingCivsNI': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                            'iNumNationsTemp': 0,
                            'lBribe' : [-1, -1, -1],
                            'lCivsToBribe': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                            'tTempFlippingCityCongress': -1,
                            'lMemory': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players + barbarians (minors and barbs are not used, but necessary for not going out of range)
                            #------------Plague
                            'lPlagueCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players + barbarians
                            'lGenericPlagueDates': [-1, -1, -1, -1],# -1],
                            'lFirstContactPlague': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], #total players + barbarians
                             #------------Victories
                            'lGoals': [[-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1],
                                       [-1, -1, -1]],
                            'lReligionFounded': [-1, -1, -1, -1, -1, -1, -1],
                            'iEnslavedUnits': 0,
                            'iRazedByMongols': 0,
                            'lEnglishEras': [-1, -1],
                            'lGreekTechs': [-1, -1, -1],
                            'lNewWorld': [-1, -1], #first founded; circumnavigated (still unused)
                            'iNumSinks': 0,
                            'lBabylonianTechs': [-1, -1, -1],                                    
                            #'iMediterraneanColonies': 0,
                            'iPortugueseColonies': 0,
                            'iFrenchColonies': 0,
                            'lWondersBuilt': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'l2OutOf3': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                            #------------Stability
                            'lBaseStabilityLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lPartialBaseStability': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lStability': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lOwnedPlotsLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lOwnedCitiesLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lCombatResultTempModifier': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lGNPold': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lGNPnew': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lGreatDepressionCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lStatePropertyCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lDemocracyCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                            'lStabilityParameters': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #2+3+2+3+3
                            'lLastRecordedStabilityStuff': [0, 0, 0, 0, 0, 0], # total + 5 parameters
                        }

The instance is of course created in CvRFCEventHandler (as before):

                self.data = StoredData.StoredData()

CvRFCEventHandler.data.scriptDict now points to the scriptDict. We include it as an argument in the constructor of the other module classes:

                self.rnf = RiseAndFall.RiseAndFall(self.data.scriptDict)

And we also create a RFCUtils instance in the Event Handler:

                self.utils = RFCUtils.RFCUtils(self.data.scriptDict)

This is what the init definition would look like for all the classes (modules) what handle stored data:

        def __init__(self, scriptData=None):
                if scriptData != None:
                        global scriptDict
                        scriptDict = scriptData

The way to get all the modules that are accessing RFCUtils (and other modules like RiseAndFall) to share the scriptDict reference is to simply not include the argument to the constructor (the same as before):

#Another module

import RFCUtils
utils = RFCUtils.RFCUtils()

To access values in scriptDict we change the methods to:

        def getStability( self, iCiv ):
                return scriptDict['lStability'][iCiv]

        def setStability( self, iCiv, iNewValue ):
                scriptDict['lStability'][iCiv] = iNewValue

I haven't had time to test this solution yet, but do you think my theory is sound?

Also, I believe that the temp values currently stored in the scriptDict could be taken out and stored as class fields of the module/class that is using them. Because I don't think other modules need to access them, and the they don't need to be saved and thus pickled onPreSave.

 

[General Tso]

For Python class objects that contain data that needs to be saved I just created the methods GetGamestate and RestoreGameState. GetGameState creates a tuple of pickleable objects for the data that needs to be saved then returns it. RestoreGameState receives a similar tuple and restores the object's values. Since GetGameState and RestoreGameState only get called when the game is be saved or loaded, there is no slowdown during actual game play. The actual calls are done within a GameState object then the whole GameState object is pickled and saved as scriptdate to the CyGame object.

 

[Baldyr]

For Python class objects that contain data that needs to be saved I just created the methods GetGamestate and RestoreGameState. GetGameState creates a tuple of pickleable objects for the data that needs to be saved then returns it. RestoreGameState receives a similar tuple and restores the object's values. Since GetGameState and RestoreGameState only get called when the game is be saved or loaded, there is no slowdown during actual game play. The actual calls are done within a GameState object then the whole GameState object is pickled and saved as scriptdate to the CyGame object.

This pretty much sums it up.

By the way, I just tested my own theory and it seems to work...

 

[embryodead]

Yes, the idea is to get rid of excess pickling in RFC, which is done on every read/write, probably over 1000 times per turn. Even if it saves a second per turn of mid- to late- game, it'd be worth it IMO, esp. with 1500 turn games and autoplay. Just moving stability to CvPlayer cuts that down at least by half, though if there's a way to handle variables from numerous classes and pickle them all only on load/save, that would be better. I'd love to see a code example, but my assumption was most mods use SdToolKit, which does the same what RFC does, i.e. pickles data on every access.

The theory is sound Baldyr. My own experience with using class variables for storing global data is rather bad - it would pass between games, even though new instance was created, and I additionally cleared it on load, game start and so on, but I was probably doing it wrong.

I'm not sure what temp values in StoredData you want to get rid of but apart from tTempFlippingCity all seem to be needed (maybe that one is too, I'm not even sure why is it used at all).

@General Tso
If I get it right, every single class has those methods, and all are called on load/save?

 

[Baldyr]

The theory is sound Baldyr. My own experience with using class variables for storing global data is rather bad - it would pass between games, even though new instance was created, and I additionally cleared it on load, game start and so on, but I was probably doing it wrong.

I actually did it and it seems to work. Right now I'm timing autoplay to see what the difference between the default method for storing data and my own is. (In a Roman autoplay it was a about 10% improvement, down 10s to 92s from 102s. I'm guessing something like a Turkish start will yield even better results.)

I'm currently only pickling and storing values on preSave. But I have autosave enabled for every single turn, so this is handicapping my own setup.

 

[embryodead]

Just a note on timing: with autoplay you need larger data set though because it's quite random, with Americans I'd get like 30-50 minutes with plain RFC, depending on how the game goes. Better to time the same turn with the BUG timer etc.

 

[Baldyr]

Actually, I find that autoplay is quite good for timing lag. (The time will be about the same every time.) With longer loads it evens out, but you're of course right about the randomness of it...

In a moment anyone will be able to test for themselves. See you at the RFC boards...

 

[Baldyr]

So I experienced a 13% reduction for wait in between turns running autoplay as the Turks. Lets just say that there can be something like 10-15% reduction of lag by not pickling everything at every juncture. It may of course be less, since it would depend on the actual game session.

But, there occurred an exception about the game not being able to read MercenaryData on the first Turkish turn. It turns out that the Mercenaries mod (which is incorporated into RFC) uses SdToolkit to save its data. But how is that even possible when Rhye uses the same CyGame instance to store the scriptDict?

gc.getGame().setScriptData( pickle.dumps(scriptDict) )

From SdToolkit.py:

CyGameInstance = gc.getGame()
...
	CyGameInstance.setScriptData( pickle.dumps(cyTable) )

I'm guessing I managed to bug out the Mercenaries feature by storing the scriptDict on preSave:

                gc.getGame().setScriptData(pickle.dumps(self.data.scriptDict))

So I guess I'll just have to use sdToolkit to save the scriptDict, then? (But I still don't understand how this can work with the regular RFC setup...)

 

[General Tso]

@General Tso
If I get it right, every single class has those methods, and all are called on load/save?

All classes that contain anything that needs to be saved.

I also automated it further for data that doesn't need to be converted to something different before pickling. I give the class that's being saved a variable called gs_attributes. It is a tuple of strings with each string containing the name of an attribute. Then I just cycle through this list using the Python builtin functions setattr and getattr to load and save data. That way if you don't have any data that needs to be converted (like pointers) all you need is a gs_attributes attribute in any object that needs saved. Then just cycle throught all the objects when loading or saving and use gs_attributes as described above.

 

[Baldyr]

Hmm... I have a problem with aliasing. Or rather not aliasing. I thought that by pointing to a field in a class in another module, changing the content of this field in another module will also be reflected in the original field. Instead, it seems that by using the global command I've created a copy of the original field that doesn't affect the original.

So every module now have its own scriptDict. Which is not what I intended to do.

Well, it was too simple. Maybe if I centralize all scriptDict operations to the StoredData module, then?

 

[embryodead]

How about getting rid of the class in StoredData so that's it's not instantiated, and put all data read/write & load/save functions in there.

 

[Baldyr]

How about getting rid of the class in StoredData so that's it's not instantiated, and put all data read/write & load/save functions in there.

This is actually what I'm working on right now.

But I think I'll keep the StoredData class for now and pass along the instance to every other module, making the reference a global value in those modules.

All the get/set and save/load methods become part of the StoredData class. (Its easier to keep them as class methods due to the share number of them.)

 

[Baldyr]

No... that is no good since even if we import the get/set methods from StoredData we need a scriptDict to use. And by making it a field of the class instance it won't be available to modules not connected to the Event Handler. And by making it a global value it won't be the same scriptDict but rather some sort of copy.

Lets see... How to make a method invocation like this in another module:

self.setStabilityParameters(i, 0)

...use the scriptDict in the StoredData module (global) or instance (field)? The object here is to not edit every single line of code in every single module accessing the scriptDict... Because that would be a lot of work and result in a epic bug hunt.

What if we substitute the pickling/unpickling lines in the get/set methods with a call to the original StoredData instance created in the Event Handler? Like this:

        def __init__(self, pData=None):
                if pData != None:
                        global StoredData
                        StoredData = pData

        def setStability( self, iCiv, iNewValue ):
                scriptDict = StoredData.scriptDict
                scriptDict['lStability'][iCiv] = iNewValue
                StoredData.storeScriptDict(scriptDict)

And add this method to the StoredData class/module:

        def storeScriptDict(self, scriptDict):
                self.scriptDict = scriptDict

This would be valid, right? And yeah, the StoredData class constructor would look like this:

Spoiler :

        def __init__( self ):
                """Initialise the global script data dictionary for usage."""
                self.scriptDict = {      #------------RiseAndFall
                                    'iNewCiv': -1,
                                    'iNewCivFlip': -1,
                                    'iOldCivFlip': -1,
                                    'tTempTopLeft': -1,
                                    'tTempBottomRight': -1,
                                    'iSpawnWar': 0, #if 1, add units and declare war. If >=2, do nothing
                                    'bAlreadySwitched': False,
                                    'lColonistsAlreadyGiven': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #active players
                                    'lAstronomyTurn': [500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500], #active players
                                    'lNumCities': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players to contain Byzantium too
                                    'lLastTurnAlive': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players to contain Byzantium too
                                    'lSpawnDelay': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #active players
                                    'lFlipsDelay': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'iBetrayalTurns': 0,
                                    'lLatestRebellionTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'iRebelCiv': 0,
                                    'lExileData': [-1, -1, -1, -1, -1],
                                    'tTempFlippingCity': -1,
                                    'lCheatersCheck': [0, -1],
                                    'lBirthTurnModifier': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lDeleteMode': [-1, -1, -1], #first is a bool, the other values are capital coordinates
                                    'lFirstContactConquerors': [0, 0, 0], #maya, inca, aztecs
                                    'bCheatMode': False,
                                    #------------Religions
                                    'iSeed': -1,
                                    #------------UP
                                    'iImmigrationTurnLength': 0,
                                    'iImmigrationCurrentTurn': 0,
                                    'iLatestFlipTurn': 0,
                                    'lLatestRazeData': [-1, -1, -1, -1, -1],
                                    #------------AIWars
                                    'lAttackingCivsArray': [0, 0, 0, 0, 0, -1, 0, -1, 0, 0, 0, -1, -1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, -1, 0, 0],
                                    'iNextTurnAIWar': -1,
                                    #------------Congresses
                                    'bCongressEnabled': False,
                                    'iCivsWithNationalism': 0,
                                    'bUNbuilt': False,
                                    'lInvitedNations': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                                    'lVotes': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lTempActiveCiv': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                                    'lTempReqCity': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                                    'iLoopIndex': 0,
                                    'lTempReqCityHuman': [-1, -1, -1, -1, -1],
                                    'tempReqCityNI': -1,
                                    'tempActiveCivNI': -1,
                                    'lTempAttackingCivsNI': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                                    'iNumNationsTemp': 0,
                                    'lBribe' : [-1, -1, -1],
                                    'lCivsToBribe': [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
                                    'tTempFlippingCityCongress': -1,
                                    'lMemory': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players + barbarians (minors and barbs are not used, but necessary for not going out of range)
                                    #------------Plague
                                    'lPlagueCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #total players + barbarians
                                    'lGenericPlagueDates': [-1, -1, -1, -1],# -1],
                                    'lFirstContactPlague': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], #total players + barbarians
                                     #------------Victories
                                    'lGoals': [[-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1],
                                               [-1, -1, -1]],
                                    'lReligionFounded': [-1, -1, -1, -1, -1, -1, -1],
                                    'iEnslavedUnits': 0,
                                    'iRazedByMongols': 0,
                                    'lEnglishEras': [-1, -1],
                                    'lGreekTechs': [-1, -1, -1],
                                    'lNewWorld': [-1, -1], #first founded; circumnavigated (still unused)
                                    'iNumSinks': 0,
                                    'lBabylonianTechs': [-1, -1, -1],                                    
                                    #'iMediterraneanColonies': 0,
                                    'iPortugueseColonies': 0,
                                    'iFrenchColonies': 0,
                                    'lWondersBuilt': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'l2OutOf3': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],
                                    #------------Stability
                                    'lBaseStabilityLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lPartialBaseStability': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lStability': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lOwnedPlotsLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lOwnedCitiesLastTurn': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lCombatResultTempModifier': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lGNPold': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lGNPnew': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lGreatDepressionCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lStatePropertyCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lDemocracyCountdown': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                                    'lStabilityParameters': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], #2+3+2+3+3
                                    'lLastRecordedStabilityStuff': [0, 0, 0, 0, 0, 0], # total + 5 parameters
				}

 

[embryodead]

No... that is no good since even if we import the get/set methods from StoredData we need a scriptDict to use. And by making it a field of the class instance it won't be available to modules not connected to the Event Handler. And by making it a global value it won't be the same scriptDict but rather some sort of copy.

That's why I said, drop the class. Then you have a module with one, truly global scriptData and loads of functions that are all available to all modules that do from StoredData import *, much like some Cv* modules provided in Civ4.

If you really want you can make a singleton or similar class, there's several ways to do that I found in google, the simplest one would be to just initialize a single instance in the StoredData module i.e.:

sd = StoredData()

then all modules import that instance, without creating new ones:

from StoredData import sd

Init works so far, though I haven't moved the get/set functions yet.

 

[Baldyr]

Ok, I think I solved the issues, finally. This must have been my tenth try... I'll post the files in my original thread on the RFC board, so that people can test for themselves.

 

[embryodead]

Well, my solution worked as well, either way, the jar of 1000 pickles is gone

 

[Baldyr]

Well, my solution worked as well, either way, the jar of 1000 pickles is gone

Yeah! What exactly did you do? (It would probably be harder for me to find it in the next RFC Marathon update.)

You mentioned a sort of "modders RFC edition" in a post awhile ago? What do you say about we join up and make RFC Marathon the next generation of RFC? That would mean minimal changes to actual gameplay (except balancing the different levels of play) but added support for modding and customization. (I believe this is already what you're doing, but I'm yet to actually look at your work.)

What I'm looking to do myself is to start going through Rhye's Python code and see what improvements could be done for performance. And also divide key features into easily accessible methods, instead of the 100s of lines of code that needs to be processed as is now. Unless you're already doing this I could share with you the fruits of my own labor. And then I'd be able to use RFC Marathon for my own future modding. (The final version of PyScenario would be a add-on to RFC Marathon, obviously.)

And there would of course not be any more pickling involved, but that is pretty much a given.