robk
Warlord
When I originally wrote this, I only knew about one method of saving data but, as you can see in the thread discussion, there are actually two different methods of saving data for your mod. Both have their advantages and disadvantages and I'll try to outline them below.
Note (2011-05-23): see this reply about recent additions that allow us to save game data in a SQL database that's embedded in the save file.
Summary
The first method, which I figured out first and was the original content of this post, involves saving data to the modUserData SQLite database. Even though it's a sql database underneath the covers, you can only access it through a pair of methods that, essentially, turn it into a key-value pair store. One method, SetValue(key, value), saves a value with a key while the other, GetValue(key), retrieves the data from the store. key should be a string while value is typed as variant in the SQLite table, so it can accept both integers and strings.
For details, check the modUserData section below.
modUserData Pros
modUserData Cons
The second method for saving data involves using the ScriptData functions associated with various elements in the game. If you look at the Player, Plot and Unit tables in the Tuner, you'll notice that each has a function called SetScriptData(string) and GetScriptData(). The first takes a simple string as an argument while the second returns that last string that was set. So, if you call Players[0].SetScriptData("test") and then immediately call Players[0].GetScriptData(), you'll get "test" returned back to you.
For details, check the ScriptData section below.
ScriptData Pros
ScriptData Cons
modUserData Method
While searching around trying to figure out how to create graphs for historical data, I came across these functions in the tutorial mods.
When you create the modUserData object, civ V creates a sqlite file called modid-modver.db in Documents\My Games\Sid Meier's Civilization 5\ModUserData. It only has one table in it, SimpleValues, with two columns: name & Value. When you use SetValue(), rowname is inserted as the Name and data is inserted as the value. Essentially, it acts as a key-value store for your mod.
Here's an example showing how you can store an retrieve a hypothetical option for a mod that describes how many units to show in some context:
Unfortunately, or fortunately depending on what you're trying to do, the domain or context of the data in that file is global. That is, any mod from any instance of the game can access it. So if you call SetValue("stuff", "things") from one game and then start up another game, GetValue("stuff") will return the string "things".
For my Info Addict mod, this isn't good because I needed a way to save data within the context of a single game. That is, if I want the military power of player2 at turn 3, I want to make sure it comes from the current game and not some other game that I just finished playing (nor do I want games overwriting each other). Originally, I came up with the idea of counting up resources on the map to identify a game but that makes Info Addict incompatible with any mods that change resources during the course of a game. It also means that a player who likes static maps wouldn't be able to go back to an old save nor bounce back and forth between different games without causing data corruption.
Currently, I'm using a two pronged approach. My main data is saved in the modUserDatabase while I'm saving a unique identifier in the save file using the Scriptdata method, outlined next.
ScriptData Method
Each plot, player and unit element in the game has two functions associated with it that allow you to save data to the saved game file itself: GetScriptData() and SetScriptData(). At its most basic form, using it looks like this:
Using it in this manner, this method of saving data has three huge issues:
To mitigate all of these problems, a library has been developed by modders (primarily Whys with contributions from killmeplease, Thalassicus, & Afforess) called SaveUtils. It allows you to save your data to one of those in game objects without deleting data that has been previously stored. Further, it allows you to save complex data structures without having to do the string conversions (serialization) yourself.
Briefly, using the library looks like this:
The third arg to save() can be most LUA data structures and conversions take place in the library to save it as a string. When load() is called, the reverse process is done to take the string representation of your data and return it to its original form: be it a string, integer, table, etc. As long as all modders use this library to store their data, there should be no data corruption because the library saves data in a namespace based on the MY_MOD_NAME variable. If a mod uses save(), the existing data for other mods will be preserved and everyone can use the guaranteed to exist objects together, such as Players[0] or Map.GetPloyByIndex(0).
For further reading and much more detail, check out the main thread and a tutorial written by Whys:
SaveUtils.lua -- Itemized Data Storage Across Mods
SaveUtils Tutorial
An example using both methods
In my Info Addict mod, I'm using the modUserData method to store the bulk of my data and a single game identifier using SaveUtils. This way, I can store large amounts of data in the SQL database but still access it through successive saves without relying on an inconsistent method (i.e. a hack).
To store the game identifier, I'm generating a unique ID (a timestamp, actually) and saving it in the saved game file like so:
To save data, I have a function that basically does this at the end of each turn (the table alldata is set up prior to this running):
.. and retrieving the data looks like this:
Note: this isn't the exact code but a simplified version for illustration.
This isn't the only way to save data but just an example that incorporates both methods. When developing your mod, you should examine what you're saving and determine the best place to save your data. If you decide to use the ScriptData method though, please use SaveUtils so your mod and everyone else's don't clobber each other.
Note (2011-05-23): see this reply about recent additions that allow us to save game data in a SQL database that's embedded in the save file.
Summary
The first method, which I figured out first and was the original content of this post, involves saving data to the modUserData SQLite database. Even though it's a sql database underneath the covers, you can only access it through a pair of methods that, essentially, turn it into a key-value pair store. One method, SetValue(key, value), saves a value with a key while the other, GetValue(key), retrieves the data from the store. key should be a string while value is typed as variant in the SQLite table, so it can accept both integers and strings.
For details, check the modUserData section below.
modUserData Pros
- Easily accessible from outside of civ V by using standard sql lite tools. Really nice for debugging.
- Includes built-in controls to make sure data from different mods don't intefere with each other. It's possible to circumvent this but that would have be done on purpose or through carelessness.
- Grabbing or saving discrete chunks of data is simple and efficient.
- When SetValue() is called, the data is immediately written to the SQL database so there are no sync'ing issues or missing data in the event of a crash
modUserData Cons
- The scope of the key-value store is global. That is, accessing key from one game is the same as accessing that from another instance of civ V on the same computer. If you're saving something like user options for your mod, this is great but data associated with the actual play of a game will have to include some identifier for that particular instance. (The section on modUserData goes into detail about this)
- Data is saved in a separate file from the standard saved game file. If you need to do some bug squishing and want someone to send you their game files, you'll have to ask for both the save game file and the modUserData related to your mod.
- I don't have strong data supporting this but, in my experience, grabbing large amounts of data from the SQL store appears to be somewhat slow. Asking for or saving a single key-value pair takes a trivial amount of time while doing the same for 6000+ pairs takes a significant amount of time (up to 5 seconds or so). That makes sense to me because each call to GetValue() should generate a single SQL statement for one row and initiating 6000 successive SQL calls is fairly expensive.
The second method for saving data involves using the ScriptData functions associated with various elements in the game. If you look at the Player, Plot and Unit tables in the Tuner, you'll notice that each has a function called SetScriptData(string) and GetScriptData(). The first takes a simple string as an argument while the second returns that last string that was set. So, if you call Players[0].SetScriptData("test") and then immediately call Players[0].GetScriptData(), you'll get "test" returned back to you.
For details, check the ScriptData section below.
ScriptData Pros
- Data is actually saved in the game's save file so there is no need to include identifiers for a particular instance of the game.
- Community built libraries already exist to make this a useful place to store your data.
ScriptData Cons
- Without using a library (which, thankfully, is available), you can only save simple strings.
- As above, without using a library, there is no built in facility to tie data to a particular mod. If your mod uses ScriptData to save data and another mod does the same, you'll both overwrite each other's data, which makes it pretty useless.
- Data is tied to a particular object so you have to be careful to keep persistent data in a object that always exists, like Players[0] or Map.GetPlotByIndex(0).
modUserData Method
While searching around trying to figure out how to create graphs for historical data, I came across these functions in the tutorial mods.
PHP:
modUserData = Modding.OpenUserData(modid, modver);
modUserData.SetValue(rowname, data);
modUserData.GetValue(rowname);When you create the modUserData object, civ V creates a sqlite file called modid-modver.db in Documents\My Games\Sid Meier's Civilization 5\ModUserData. It only has one table in it, SimpleValues, with two columns: name & Value. When you use SetValue(), rowname is inserted as the Name and data is inserted as the value. Essentially, it acts as a key-value store for your mod.
Here's an example showing how you can store an retrieve a hypothetical option for a mod that describes how many units to show in some context:
PHP:
modver = 1;
modid = "awesomeMod";
modUserData = Modding.OpenUserData(modid, modver);
modUserData.SetValue("options-unitsToShow", "32");
-- Later in your code
myOptions.unitsToShow = modUserData.GetValue("options-unitsToShow");Unfortunately, or fortunately depending on what you're trying to do, the domain or context of the data in that file is global. That is, any mod from any instance of the game can access it. So if you call SetValue("stuff", "things") from one game and then start up another game, GetValue("stuff") will return the string "things".
For my Info Addict mod, this isn't good because I needed a way to save data within the context of a single game. That is, if I want the military power of player2 at turn 3, I want to make sure it comes from the current game and not some other game that I just finished playing (nor do I want games overwriting each other). Originally, I came up with the idea of counting up resources on the map to identify a game but that makes Info Addict incompatible with any mods that change resources during the course of a game. It also means that a player who likes static maps wouldn't be able to go back to an old save nor bounce back and forth between different games without causing data corruption.
Currently, I'm using a two pronged approach. My main data is saved in the modUserDatabase while I'm saving a unique identifier in the save file using the Scriptdata method, outlined next.
ScriptData Method
Each plot, player and unit element in the game has two functions associated with it that allow you to save data to the saved game file itself: GetScriptData() and SetScriptData(). At its most basic form, using it looks like this:
PHP:
pPlayer = Players[0];
pPlayer:SetScriptData("Here is my saved data");
data = pPlayer:GetScriptData();Using it in this manner, this method of saving data has three huge issues:
- You can only save one string per object.
- Only one mod can use a saved area without causing data corruption for every mod that wants to save data.
- There are only have a finite number of objects that are guaranteed to exist in the game. You can be pretty sure that Players[0] always exists, but Map.GetPlotByIndex(4096) probably only exists on the largest maps.
To mitigate all of these problems, a library has been developed by modders (primarily Whys with contributions from killmeplease, Thalassicus, & Afforess) called SaveUtils. It allows you to save your data to one of those in game objects without deleting data that has been previously stored. Further, it allows you to save complex data structures without having to do the string conversions (serialization) yourself.
Briefly, using the library looks like this:
PHP:
include("SaveUtils"); MY_MOD_NAME="awesomeMod";
pPlayer = Players[0];
dataKey = "datakey";
save(pPlayer, dataKey, "Here is my saved data");
data = load(pPlayer, dataKey);The third arg to save() can be most LUA data structures and conversions take place in the library to save it as a string. When load() is called, the reverse process is done to take the string representation of your data and return it to its original form: be it a string, integer, table, etc. As long as all modders use this library to store their data, there should be no data corruption because the library saves data in a namespace based on the MY_MOD_NAME variable. If a mod uses save(), the existing data for other mods will be preserved and everyone can use the guaranteed to exist objects together, such as Players[0] or Map.GetPloyByIndex(0).
For further reading and much more detail, check out the main thread and a tutorial written by Whys:
SaveUtils.lua -- Itemized Data Storage Across Mods
SaveUtils Tutorial
An example using both methods
In my Info Addict mod, I'm using the modUserData method to store the bulk of my data and a single game identifier using SaveUtils. This way, I can store large amounts of data in the SQL database but still access it through successive saves without relying on an inconsistent method (i.e. a hack).
To store the game identifier, I'm generating a unique ID (a timestamp, actually) and saving it in the saved game file like so:
PHP:
function GetGameIdent()
local timeKey = "timekey";
local plot = Map.GetPlotByIndex(1);
local gameStartTime = load(plot, timeKey);
if (type(gameStartTime) == "table" or gameStartTime == nil) then
gameStartTime = os.time();
logger:info("Saving new gameStartTime: " .. gameStartTime);
save(plot, timeKey, gameStartTime);
end;
return gameStartTime;
endTo save data, I have a function that basically does this at the end of each turn (the table alldata is set up prior to this running):
PHP:
local modver = 1;
local modid = "InfoAddict";
local modUserData = Modding.OpenUserData(modid, modver);
for pid, data in ipairs(alldata) do
local rowname = GetGameIdent() .. "-turn" .. turn .. "-player" .. pid
modUserData.SetValue(rowname, data)
end.. and retrieving the data looks like this:
PHP:
function getSavedData(turn, pid)
local GameIdent = GetGameIdent();
local modver = 1;
local modid = "InfoAddict";
local modUserData = Modding.OpenUserData(modid, modver);
local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
local data = modUserData.GetValue(rowname);
return data;
end;This isn't the only way to save data but just an example that incorporates both methods. When developing your mod, you should examine what you're saving and determine the best place to save your data. If you decide to use the ScriptData method though, please use SaveUtils so your mod and everyone else's don't clobber each other.
.
goodjob
What we could do is to agree on policy where each mods only modifies it's own data within [mod_id] [/mod_id] tag and get/set all other data in there as they were?
