[TUTORIAL] How to Set Persistent Variables in Pure Python

[CSD]

Spoiler Background :

I wanted to solve the infinite tech exploit without making changes to the DLL (and preserving Oracle/Liberalism mechanics) which meant python was needed.

After many failed attempts to prevent the exploit, I finally developed a viable alternative: add to a "freeTechsAllowed" counter per player for completed free tech granting buildings/projects/techs and check if it is greater than zero when a free tech is gained. If true, decrement counter; else, remove their ill-gained tech and recalculate the player's era.

Everything seemed to work perfectly after quick testing with the aid of worldbuilder, until I was struck with this horrible thought that my humble counter couldn't survive between sessions.

Spoiler :

It didn't.

This meant that anyone who was due a free tech on load would be sorely disappointed.

"Up in the sky, look: It's a bird! It's a plane! It's ScriptData! And it can save the day, once per instance* (data serialization imported separately)!"

It's a persistent string intended for modding, which technically I could save my counter to directly, but if I want to later add a second persistent variable or more, I'm in a pickle.

Thankfully, I found a relatively simple solution with little overhead to setting multiple variables per ScriptData string which is clearly presented below for the benefit of my fellow python modders.

Spoiler Tutorial :

There is a single persistent variable (ScriptData) for each instance of several classes (i.e. CyCity, CyGame, CyPlayer, CyPlot, CyUnit) which has getter and setter methods:

• STRING getScriptData()
• VOID setScriptData(STRING)

Fairly straightforward, but also limited to a single value unless the string is serialized data (essentially a long line of bytes subdivided to represent some data structure).

The pickle library is already used for some data serialization in Civ4, so I've decided just to stick with it, but it's worth noting there are alternatives.

There are many methods in pickle but only two will be required here:

• OBJECT loads(STRING)
• STRING dumps(OBJECT)

The loads method should take getScriptData as its argument to load an object from ScriptData, and setScriptData should take dumps as its argument to save an object to ScriptData.

But what object allows us to effortlessly map multiple variables? A dictionary!

First, we need to import pickle, or python will throw shade.

Also, pickle needs to know what object type we want, so we pass an empty dictionary: cyInstance.setScriptData(pickle.dumps({})).

Then we load a copy of our dictionary into a local variable: dLocal = pickle.loads(cyInstance.getScriptData()).

Now we can insert any variable we want into the dictionary and set its value in the same line of code: dLocal["anyVariable"] = bar.

Furthermore, we can take the current or default value of our variable as an argument: dLocal["anyVariable"] = foo(dLocal.get("anyVariable", bar)).

If we get the current value of our variable without a default, make sure to define it first or it will return None: bar = dLocal.get("anyVariable").

Finally, remember to save your dictionary: cyInstance.setScriptData(pickle.dumps(dLocal)).

Spoiler Generic Example :

import pickle
game = gc.getGame()

# Initialize dictionary for game
game.setScriptData(pickle.dumps({}))

# Choose default values
key1Default = 0
key2Default = ""

# Load game dictionary
dGameVars = pickle.loads(game.getScriptData())

# Set values independent of current values
dGameVars["key1"] = someFunction()
dGameVars["key2"] = anotherFunction()

# Get values with defined defaults
iName = dGameVars.get("key1",key1Default)
szName = dGameVars.get("key2",key2Default)

# Get values with default to None
iName = dGameVars.get("key1")
szName = dGameVars.get("key2")

# Set values dependent on current values only (not recommended unless None is valid state)
dGameVars["key1"] = someFunction(dGameVars.get("key1"))
dGameVars["key2"] = anotherFunction(dGameVars.get("key2"))

# Set values dependent on its current values or defaults (recommended)
dGameVars["key1"] = someFunction(dGameVars.get("key1",key1Default))
dGameVars["key2"] = anotherFunction(dGameVars.get("key2",key2Default))

# Save game dictionary
game.setScriptData(pickle.dumps(dGameVars))

Spoiler Specific Example :

This is the relevant code from CvEventManager.py I used for solving the infinite tech bug. Note that only a single variable "freeTechsAllowed" is currently being mapped to each player dictionary, but more can easily be added now.

import pickle

class CvEventManager:
 
    def onGameStart(self, argsList):
 
        for iPlayer in range(gc.getMAX_PLAYERS()):
            player = gc.getPlayer(iPlayer)
            player.setScriptData(pickle.dumps({}))
 
    def onBuildingBuilt(self, argsList):
 
        iPlayer = pCity.getOwner()
        player = gc.getPlayer(iPlayer)
 
        if (gc.getBuildingInfo(iBuildingType).getFreeTechs() > 0):
            dPlayerVars = pickle.loads(player.getScriptData())
            dPlayerVars["freeTechsAllowed"] = dPlayerVars.get("freeTechsAllowed",0) + gc.getBuildingInfo(iBuildingType).getFreeTechs()
            player.setScriptData(pickle.dumps(dPlayerVars))
 
    def onProjectBuilt(self, argsList):
 
        iPlayer = pCity.getOwner()
        player = gc.getPlayer(iPlayer)
 
        if (gc.getProjectInfo(iProjectType).getTechShare() > 0):
            dPlayerVars = pickle.loads(player.getScriptData())
            dPlayerVars["freeTechsAllowed"] = dPlayerVars.get("freeTechsAllowed",0) + gc.getNumTechInfos() # Placeholder for techs due
            player.setScriptData(pickle.dumps(dPlayerVars))
 
    def onTechAcquired(self, argsList):
 
        team = gc.getTeam(iTeam)
        player = gc.getPlayer(iPlayer)
 
        if (gc.getGame().countKnownTechNumTeams(iTechType) == 0 and iPlayer == team.getLeaderID()):
            dPlayerVars = pickle.loads(player.getScriptData())
            dPlayerVars["freeTechsAllowed"] = dPlayerVars.get("freeTechsAllowed",0) + gc.getTechInfo(iTechType).getFirstFreeTechs()
            player.setScriptData(pickle.dumps(dPlayerVars))
 
        if (gc.getGame().isFinalInitialized() and not gc.getGame().GetWorldBuilderMode()):
    
            if (team.getResearchLeft(iTechType) > 0):
                dPlayerVars = pickle.loads(player.getScriptData())
                if (dPlayerVars.get("freeTechsAllowed",0) > 0):
                    dPlayerVars["freeTechsAllowed"] = dPlayerVars.get("freeTechsAllowed",0) - 1
                    player.setScriptData(pickle.dumps(dPlayerVars))
                else:
                    team.setHasTech(iTechType,False,iPlayer,False,False)
                    CyInterface().addImmediateMessage("UNAUTHORIZED FREE TECHNOLOGY REMOVED", "")
                    playerEra=0
                    for jTechType in range(gc.getNumTechInfos()):
                        if (team.isHasTech(jTechType)):
                            techEra = gc.getTechInfo(jTechType).getEra()
                            if (techEra > playerEra):
                                playerEra = techEra
                    player.setCurrentEra(playerEra)
                    return

 

[Leoreth]

Thanks for writing this guide!

I think it's nice that the game offers to assign script data to each individual game object, but often that makes handling it very tedious, because it needs to be serialized and deserialized individually every time you want to access or modify it.

I am using a pattern originally implemented by Rhye and significantly improved by embryodead, that I made minor improvements to over time. The idea is to have a singleton object containing all the relevant persistent data in Python that you can interact with just like any other Python object (read/write members, add functions, nested structures etc.). This singleton is stored only as the script data of the CvGame object (which is itself a singleton). The object is initialised from deserialized script data when the game is started/loaded and dumped as serialized script data when the game is saved.

Here's a sketch of the idea:

class GameData:

    def __init__(self):
        self.setup()

    # data is a Python dict coming from script data
    # we can dump it into this object's __dict__ to reconstruct the object
    def update(self, data):
        self.__dict__.update(data)

    # define all the variables you want to persistently track here
    def setup(self):
        self.dPlayerFreeTechsAllowed = {}  # or dict((iPlayer, 0) for iPlayer in range(gc.getNumPlayers())) if you prefer

    # this can also have methods to make your life easier
    def changePlayerFreeTechsAllowed(self, iPlayer, iChange):
        self.dPlayerFreeTechsAllowed[iPlayer] = self.dPlayerFreeTechsAllowed.get(iPlayer, 0) + iChange

data = GameData()

Then similar to your example you only need to set up the state management once:

from StoredData import data
import pickle

gc = CyGlobalContext()

class CvEventManager:

    def onGameStart(self, argsList):
        data = GameData()

    def onGameLoad(self, argsList):
        data.update(pickle.loads(gc.getGame().getScriptData()))

    def onGameSaved(self, argsList):
        gc.getGame().setScriptData(pickle.dumps(data.__dict__)))

    # adapting part of your example above
    def onBuildingBuilt(self, argsList):
        # ...

        if gc.getBuildingInfo(iBuilding).getFreeTechs() > 0:
            data.changePlayerFreeTechsAllowed(iPlayer, gc.getBuildingInfo(iBuilding).getFreeTechs())

The nice thing about this approach is that once it is set up, you never need to worry about script data and pickling or persistence in general at all anymore. Just remember that any value you want to be persisted is tracked in the GameData singleton and you're good. It can also be imported in any Python module that is part of your mod, which makes it easy to pass variables around as well.

As you can see here I chose to represent the tracking of multiple players using a dictionary with their ID as keys - I found that generally sufficient and actually preferable to letting each player object hold its own script data. If you track a lot of player specific data, you can do so in a nested PlayerData object.

An example of that (and a lot of other stuff that would be distracting, which is why I did not use it as my initial example) can be found in my own version of the pattern here.

 

[CSD]

Thanks @Leoreth for sharing your ideas on how this can be improved!

I wish persistent variables could rely on events like onGameSave for state saving, but unfortunately some critical methods (e.g. onBuildingBuilt) fail to execute again after loading an autosave, resulting in the same outcome as no persistent state (e.g. Oracle free tech is denied on load).

It is also possible to pass a tuple (or any hashable type) as the variable name argument, which would allow a single ScriptData to handle all persistent variables (e.g. pickle.loads(game.getScriptData()).get((iPlayer, "freeTechsAllowed"), 0) ).

I agree the wrapping lines of code needed for data serialization increases tedium significantly, and found the dictionary methods could be improved aesthetically, which is why I've written a simple class to act as an on-demand variable interface for ScriptData:

import pickle

class CustomScriptData:
    def __init__(self):
        return
        
    def load(self, cyObject):
        if (not cyObject.getScriptData()):
            cyObject.setScriptData(pickle.dumps({}))
        dVars = pickle.loads(cyObject.getScriptData())
        return dVars
        
    def save(self, cyObject, dVars):
        cyObject.setScriptData(pickle.dumps(dVars))
        return
    
    def get(self, cyObject, varName, default=0):
        dVars = self.load(cyObject)
        return dVars.get(varName, default)

    def set(self, cyObject, varName, newValue):
        dVars = self.load(cyObject)
        dVars[varName] = newValue
        self.save(cyObject, dVars)
        return
    
    def add(self, cyObject, varName, addValue, default=0):
        newValue = self.get(cyObject, varName, default) + addValue
        self.set(cyObject, varName, newValue)
        return

csd = CustomScriptData()

Spoiler Side-note to the observant reader :

I honestly couldn't think of a better class name than CustomScriptData, so its instance abbreviation (csd) matching my username (CSD) was a funny coincidence. Feel free to rename it as I expect no attribution. [IMG alt=""]https://forums.civfanatics.com/images/smilies/crazyeyes.gif[/IMG]

And here is an updated specific example for the infinite tech fix using the new class:

from CustomScriptData import csd
#No longer need import pickle

class CvEventManager:

    #No longer need onGameStart or any initialization
    
    def onBuildingBuilt(self, argsList):
 
        if (building.getFreeTechs() > 0):
            csd.add(player, "freeTechsAllowed", building.getFreeTechs())
 
    def onProjectBuilt(self, argsList):
 
        if (project.getTechShare() > 0):
            csd.add(player, "freeTechsAllowed", techDue)
 
    def onTechAcquired(self, argsList):
 
        if (game.countKnownTechNumTeams(iTechType) == 0 and iPlayer == team.getLeaderID()):
            csd.add(player, "freeTechsAllowed", tech.getFirstFreeTechs())
 
        if (game.isFinalInitialized() and not gc.getGame().GetWorldBuilderMode()):
    
            if (team.getResearchLeft(iTechType) > 0):
                if (csd.get(player, "freeTechsAllowed") > 0):
                    csd.add(player, "freeTechsAllowed", -1)
                else:
                    team.setHasTech(iTechType,False,iPlayer,False,False)
                    CyInterface().addImmediateMessage("UNAUTHORIZED FREE TECHNOLOGY REMOVED", "")
                    player.setCurrentEra(playerEra)
                    return

It is still possible to pass a tuple (or any hashable type) as the variable name argument (e.g. csd.get(game, (iPlayer,"freeTechsAllowed")) ).

 

[Leoreth]

I wish persistent variables could rely on events like onGameSave for state saving, but unfortunately some critical methods (e.g. onBuildingBuilt) fail to execute again after loading an autosave, resulting in the same outcome as no persistent state (e.g. Oracle free tech is denied on load).

That's a good point - I am not entirely sure what the issue you are describing is, but I distantly remember fixing/creating an extra event for autosaving to also save the game data. Which obviously is not an option if you are going for a pure Python module. It's difficult to remember which minor DLL alteration your Python code relies on.

 

[CSD]

The bug being fixed in the example is mostly relevant for multiplayer, since one could easily get free tech from the worldbuilder instead.

Looks like this commit was where you fixed it in DoC.

Another peculiar thing about the onBuildingBuilt event is that it only gets called for AI after they've received free tech from say the Oracle, so I had to move the relevant code into onTechAcquired and add another persistent variable ("freeTechBuildingsBuilt", iBuildingType) (which uses a tuple key per city) to keep track of any buildings that have already granted their free tech(s):

    def onTechAcquired(self, argsList):
        
        if (game.countKnownTechNumTeams(iTechType) == 0 and iPlayer == team.getLeaderID()):
            csd.add(player, "freeTechsAllowed", tech.getFirstFreeTechs())
        
        if (game.isFinalInitialized() and not game.GetWorldBuilderMode()):
            
            if (team.getResearchLeft(iTechType) > 0 and (game.getGameTurn() > 0 or not game.isOption(GameOptionTypes.GAMEOPTION_ADVANCED_START))):
                
                for iBuildingType in range(gc.getNumBuildingInfos()):
                    building = gc.getBuildingInfo(iBuildingType)
                    if (player.getBuildingClassCount(building.getBuildingClassType()) > 0):
                        freeTechsPerBuilding = building.getFreeTechs()
                        if (freeTechsPerBuilding > 0):
                            buildingFreeTechsToAllow = 0
                            for iCity in range(player.getNumCities()):
                                pCity = player.getCity(iCity)
                                buildingsInCity = pCity.getNumBuilding(iBuildingType)
                                if (buildingsInCity > 0):
                                    if (pCity.getBuildingOriginalOwner(iBuildingType) == iPlayer):
                                        buildingFreeTechsToAllow += (buildingsInCity - csd.get(pCity, ("freeTechBuildingsBuilt", iBuildingType))) * freeTechsPerBuilding
                                        csd.set(pCity, ("freeTechBuildingsBuilt", iBuildingType), buildingsInCity)
                            csd.add(player, "freeTechsAllowed", buildingFreeTechsToAllow)
                
                if (csd.get(player, "freeTechsAllowed") > 0):
                    csd.add(player, "freeTechsAllowed", -1)
                else:
                    team.setHasTech(iTechType,False,iPlayer,False,False)
                    CyInterface().addImmediateMessage("UNAUTHORIZED FREE TECHNOLOGY REMOVED", "")
                    playerEra=0
                    player.setCurrentEra(playerEra)
                    return