View Full Version : Storing data across saved games from LUA


robk
Oct 12, 2010, 05:38 PM
When I originally wrote this, I only knew about one method of saving data but, as you can see in the thread discussion, there are actually two different methods of saving data for your mod. Both have their advantages and disadvantages and I'll try to outline them below.

Note (2011-05-23): see this reply (http://forums.civfanatics.com/showthread.php?p=10523135&postcount=51) about recent additions that allow us to save game data in a SQL database that's embedded in the save file.

Summary

The first method, which I figured out first and was the original content of this post, involves saving data to the modUserData SQLite database. Even though it's a sql database underneath the covers, you can only access it through a pair of methods that, essentially, turn it into a key-value pair store. One method, SetValue(key, value), saves a value with a key while the other, GetValue(key), retrieves the data from the store. key should be a string while value is typed as variant in the SQLite table, so it can accept both integers and strings.

For details, check the modUserData section below.

modUserData Pros

Easily accessible from outside of civ V by using standard sql lite tools. Really nice for debugging.
Includes built-in controls to make sure data from different mods don't intefere with each other. It's possible to circumvent this but that would have be done on purpose or through carelessness.
Grabbing or saving discrete chunks of data is simple and efficient.
When SetValue() is called, the data is immediately written to the SQL database so there are no sync'ing issues or missing data in the event of a crash


modUserData Cons

The scope of the key-value store is global. That is, accessing key from one game is the same as accessing that from another instance of civ V on the same computer. If you're saving something like user options for your mod, this is great but data associated with the actual play of a game will have to include some identifier for that particular instance. (The section on modUserData goes into detail about this)
Data is saved in a separate file from the standard saved game file. If you need to do some bug squishing and want someone to send you their game files, you'll have to ask for both the save game file and the modUserData related to your mod.
I don't have strong data supporting this but, in my experience, grabbing large amounts of data from the SQL store appears to be somewhat slow. Asking for or saving a single key-value pair takes a trivial amount of time while doing the same for 6000+ pairs takes a significant amount of time (up to 5 seconds or so). That makes sense to me because each call to GetValue() should generate a single SQL statement for one row and initiating 6000 successive SQL calls is fairly expensive.


The second method for saving data involves using the ScriptData functions associated with various elements in the game. If you look at the Player, Plot and Unit tables in the Tuner, you'll notice that each has a function called SetScriptData(string) and GetScriptData(). The first takes a simple string as an argument while the second returns that last string that was set. So, if you call Players[0].SetScriptData("test") and then immediately call Players[0].GetScriptData(), you'll get "test" returned back to you.

For details, check the ScriptData section below.

ScriptData Pros

Data is actually saved in the game's save file so there is no need to include identifiers for a particular instance of the game.
Community built libraries already exist to make this a useful place to store your data.


ScriptData Cons

Without using a library (which, thankfully, is available (http://forums.civfanatics.com/showthread.php?p=9799583)), you can only save simple strings.
As above, without using a library, there is no built in facility to tie data to a particular mod. If your mod uses ScriptData to save data and another mod does the same, you'll both overwrite each other's data, which makes it pretty useless.
Data is tied to a particular object so you have to be careful to keep persistent data in a object that always exists, like Players[0] or Map.GetPlotByIndex(0).




modUserData Method

While searching around trying to figure out how to create graphs for historical data, I came across these functions in the tutorial mods.


modUserData = Modding.OpenUserData(modid, modver);
modUserData.SetValue(rowname, data);
modUserData.GetValue(rowname);


When you create the modUserData object, civ V creates a sqlite file called modid-modver.db in Documents\My Games\Sid Meier's Civilization 5\ModUserData. It only has one table in it, SimpleValues, with two columns: name & Value. When you use SetValue(), rowname is inserted as the Name and data is inserted as the value. Essentially, it acts as a key-value store for your mod.

Here's an example showing how you can store an retrieve a hypothetical option for a mod that describes how many units to show in some context:


modver = 1;
modid = "awesomeMod";
modUserData = Modding.OpenUserData(modid, modver);
modUserData.SetValue("options-unitsToShow", "32");

-- Later in your code
myOptions.unitsToShow = modUserData.GetValue("options-unitsToShow");



Unfortunately, or fortunately depending on what you're trying to do, the domain or context of the data in that file is global. That is, any mod from any instance of the game can access it. So if you call SetValue("stuff", "things") from one game and then start up another game, GetValue("stuff") will return the string "things".

For my Info Addict mod, this isn't good because I needed a way to save data within the context of a single game. That is, if I want the military power of player2 at turn 3, I want to make sure it comes from the current game and not some other game that I just finished playing (nor do I want games overwriting each other). Originally, I came up with the idea of counting up resources on the map (http://forums.civfanatics.com/showthread.php?p=9915102&postcount=47) to identify a game but that makes Info Addict incompatible with any mods that change resources during the course of a game. It also means that a player who likes static maps wouldn't be able to go back to an old save nor bounce back and forth between different games without causing data corruption.

Currently, I'm using a two pronged approach. My main data is saved in the modUserDatabase while I'm saving a unique identifier in the save file using the Scriptdata method, outlined next.



ScriptData Method

Each plot, player and unit element in the game has two functions associated with it that allow you to save data to the saved game file itself: GetScriptData() and SetScriptData(). At its most basic form, using it looks like this:


pPlayer = Players[0];
pPlayer:SetScriptData("Here is my saved data");
data = pPlayer:GetScriptData();


Using it in this manner, this method of saving data has three huge issues:

You can only save one string per object.
Only one mod can use a saved area without causing data corruption for every mod that wants to save data.
There are only have a finite number of objects that are guaranteed to exist in the game. You can be pretty sure that Players[0] always exists, but Map.GetPlotByIndex(4096) probably only exists on the largest maps.


To mitigate all of these problems, a library has been developed by modders (primarily Whys with contributions from killmeplease, Thalassicus, & Afforess) called SaveUtils (http://forums.civfanatics.com/showthread.php?p=9799583). It allows you to save your data to one of those in game objects without deleting data that has been previously stored. Further, it allows you to save complex data structures without having to do the string conversions (serialization) yourself.

Briefly, using the library looks like this:


include("SaveUtils"); MY_MOD_NAME="awesomeMod";
pPlayer = Players[0];
dataKey = "datakey";
save(pPlayer, dataKey, "Here is my saved data");
data = load(pPlayer, dataKey);


The third arg to save() can be most LUA data structures and conversions take place in the library to save it as a string. When load() is called, the reverse process is done to take the string representation of your data and return it to its original form: be it a string, integer, table, etc. As long as all modders use this library to store their data, there should be no data corruption because the library saves data in a namespace based on the MY_MOD_NAME variable. If a mod uses save(), the existing data for other mods will be preserved and everyone can use the guaranteed to exist objects together, such as Players[0] or Map.GetPloyByIndex(0).

For further reading and much more detail, check out the main thread and a tutorial written by Whys:

SaveUtils.lua -- Itemized Data Storage Across Mods (http://forums.civfanatics.com/showthread.php?p=9799583)

SaveUtils Tutorial (http://forums.civfanatics.com/showthread.php?p=9868011&postcount=46)



An example using both methods

In my Info Addict mod, I'm using the modUserData method to store the bulk of my data and a single game identifier using SaveUtils. This way, I can store large amounts of data in the SQL database but still access it through successive saves without relying on an inconsistent method (i.e. a hack).

To store the game identifier, I'm generating a unique ID (a timestamp, actually) and saving it in the saved game file like so:


function GetGameIdent()

local timeKey = "timekey";
local plot = Map.GetPlotByIndex(1);
local gameStartTime = load(plot, timeKey);

if (type(gameStartTime) == "table" or gameStartTime == nil) then
gameStartTime = os.time();
logger:info("Saving new gameStartTime: " .. gameStartTime);
save(plot, timeKey, gameStartTime);
end;

return gameStartTime;
end


To save data, I have a function that basically does this at the end of each turn (the table alldata is set up prior to this running):


local modver = 1;
local modid = "InfoAddict";
local modUserData = Modding.OpenUserData(modid, modver);

for pid, data in ipairs(alldata) do
local rowname = GetGameIdent() .. "-turn" .. turn .. "-player" .. pid
modUserData.SetValue(rowname, data)
end


.. and retrieving the data looks like this:


function getSavedData(turn, pid)
local GameIdent = GetGameIdent();
local modver = 1;
local modid = "InfoAddict";
local modUserData = Modding.OpenUserData(modid, modver);
local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
local data = modUserData.GetValue(rowname);
return data;
end;

Note: this isn't the exact code but a simplified version for illustration.

This isn't the only way to save data but just an example that incorporates both methods. When developing your mod, you should examine what you're saving and determine the best place to save your data. If you decide to use the ScriptData method though, please use SaveUtils so your mod and everyone else's don't clobber each other.

The_J
Oct 12, 2010, 05:50 PM
I guess there are some parts missing, right?
Because if that stays like this, i'll move it to the main forum.

robk
Oct 12, 2010, 05:56 PM
Yeah, sorry about that J. I hit the summit button a little prematurely. :blush:

Pazyryk
Oct 12, 2010, 08:13 PM
I sure wish there was a better solution. The problem here is that the data isn't in the gamesave file. If someone sends you their gamesave (say, for bug squashing), then it won't find the proper table entry on your computer. Afforess seems to know a solution over here (http://forums.civfanatics.com/showthread.php?t=388892), but he hasn't volunteered any details yet.

robk
Oct 12, 2010, 08:32 PM
I sure wish there was a better solution. The problem here is that the data isn't in the gamesave file. If someone sends you their gamesave (say, for bug squashing), then it won't find the proper table entry on your computer. Afforess seems to know a solution over here (http://forums.civfanatics.com/showthread.php?t=388892), but he hasn't volunteered any details yet.

It would be really nice if we could register tables to be saved. Something like:

local tableData = {a, b, c, d};
local tableName = "myPreciousData";
registerSaveData(tableData, tableName);


Actually, looking at Afforess' code, he's attempting to do just that, except it looks like you can only save one table and you have to convert it to a string before you can save it.

Given what we have to work with, it's pretty easy to look into that sqlite db file. We would just have to ask people to send us both their save file and the db file associated with the mod. I'm totally with you, however - it would be nice if we had a easy facility to save data from our lua scripts.

Pazyryk
Oct 13, 2010, 08:37 AM
You can save data to 3 objects in the game, Players, Units, and Plots. I have tested two of these, and Players and Plots work fine. The function is the same for all of them. Use Players[iPlayerID]:GetScriptData() to retrieve the string, and layers[iPlayerID]:SetScriptData() to save it. The large part of my code is for serializing tables into savable strings and back.

The SVN is public, FYI.

This is from the revolutions / dynamic history mod thread, but the comment is relevant here. If I understand correctly, this script data will be saved with the gamesave file. Even more, each individual unit, plot and player can have a separate stored string. Afforess is expanding the method so that it can store tables. But it is quite powerful even as a string. Just use a "|" or somesuch to separate items and you can store as many things as you want about any particular plot, unit or player.

The db method in OP still has a usefulness for other things. I'll especially be using this for function troubleshooting in my modding.

The_J
Oct 13, 2010, 02:22 PM
Yeah, sorry about that J. I hit the summit button a little prematurely. :blush:

No problem ;).
Nice to see that the people are already doing tutorials (:goodjob:). It will sure help all the new people in modding and will speed it a bit up :).

alpaca
Oct 13, 2010, 02:47 PM
Wouldn't it be easier to give each game a numeric ID and store a "highest used ID" counter value instead?

Not that saving in the savegame wouldn't be better anyways :)

robk
Oct 13, 2010, 03:54 PM
Wouldn't it be easier to give each game a numeric ID and store a "highest used ID" counter value instead?

Not that saving in the savegame wouldn't be better anyways :)

Before I knew about SetScriptData, there wouldn't be a way to do that because the game wouldn't store that numeric ID in any way. I picked the counting resources thing because that would be a constant value across saved games. Now that I know about the ScriptData thing, yes, it would be way better to generate a numeric ID and just save it in the game file (or even use something like os.time() since that's unique to a single system and should rarely collide even if people are sharing save files).

Earlier, I thought about doing a hybrid thing, where I would generate that resource string, grab a nextID field from the modUserData and then associate the resource string with the gameID to cut down on the data size for each row. But, I was lazy and didn't get to that.

In a related note, I saw a Game.SetScriptData() function but it returns NYI, "not yet implemented". That's lame because that would be the perfect place to save game wide data. Right now, it looks like you have to piggy back off of player0 or something.

I'll have to add the ScriptData stuff to the OP since it seems to be a better place to store some info.

robk
Oct 13, 2010, 04:10 PM
I'll have to add the ScriptData stuff to the OP since it seems to be a better place to store some info.

Just had a thought: wouldn't any mod that uses Get/SetScriptData() be incompatible with any other mod that uses those same functions? If I wrote a simple gameID to Player[0]:SetScriptData(), I'd completely wipe out any other data in there.

Onni
Oct 13, 2010, 09:58 PM
Good find robk! :king:

Just had a thought: wouldn't any mod that uses Get/SetScriptData() be incompatible with any other mod that uses those same functions? If I wrote a simple gameID to Player[0]:SetScriptData(), I'd completely wipe out any other data in there.

For me that also seems to be the downside using Get/SetScriptData(). :( What we could do is to agree on policy where each mods only modifies it's own data within tag and get/set all other data in there as they were?

There is also one loop-hole using that non-strategic-resource-id. It only identifies a single unique game, not save games within it. What if a player decides to reload a game from an earlier save instead of the latest one?

To me the unique id which would fit for every purpose would be one that is tied to, both the game, and also to the last loaded save game. This could propably be accomplish by modifying core files "LoadMenu.lua" and "SaveMenu.lua". Currently the last loaded game info is erased in LoadMenu.lua. What we could do is to store a unique id's for both the game and save game which are tied to the save game creation time(unique). Like this:
[CURRENT] = {datetime = "2010-10-24-14-50", gameId = 1, saveId = 2}
[1] = {datetime = "2010-10-24-13-45", gameId = 1, saveId = 1}
[2] = {datetime = "2010-10-24-14-50", gameId = 1, saveId = 2}


So after that any game that needs to get the current games unique id would just use this command:
local modUserData = Modding.OpenUserData("LoadSaveMenuModId", 1);
local gameId = modUserData.GetValue("CURRENT");

Onni
Oct 13, 2010, 10:25 PM
Just figured that there is a unique id for each map(=game): Network.GetMapRandSeed()

Using that and perhaps turn counter we could make a compromise solution for even the save game files? Then we wouldn't need any core files changes nor Get/SetScriptData policies. ...Unless we are playing on a fixed map? ;)

robk
Oct 14, 2010, 06:02 PM
For me that also seems to be the downside using Get/SetScriptData(). :( What we could do is to agree on policy where each mods only modifies it's own data within tag and get/set all other data in there as they were?

Even better, if someone wrote a common library to do just that we'd be sitting pretty. 'Course, I could see massive amounts of data being stored in a single string causing problems.

There is also one loop-hole using that non-strategic-resource-id. It only identifies a single unique game, not save games within it. What if a player decides to reload a game from an earlier save instead of the latest one?

For me, that wasn't a big deal. If someone restarts from an earlier save, the "future" data gets overwritten as the new turns get played. Of course, it causes a problem if someone goes back to an earlier save and then skips back to a save that is in the future of the same game. I'm just going to say that behavior is unsupported right now :mischief:

robk
Oct 14, 2010, 06:12 PM
Just figured that there is a unique id for each map(=game): Network.GetMapRandSeed()

Neat!

Unless we are playing on a fixed map? ;)

Aw nuts, I didn't think of that. That kinda removes the possibility of playing more than one game using a fixed map at a time with my mod.

Afforess
Oct 14, 2010, 08:30 PM
I see my named mentioned several times in the thread, and people seem to want my expertise (such as it is...). Please PM me, so I notice quicker than happenstance searching. ;)

Even better, if someone wrote a common library to do just that we'd be sitting pretty. 'Course, I could see massive amounts of data being stored in a single string causing problems.

That sounds like an excellent idea, and I am up to the challenge. I'll see what I can do this weekend. Ideally, you would should be able to call a function with an ID of your choosing, unique to your mod (or save data, or whatever), and a table or string, and save it, and then retrieve it with just your ID; Without clashing with everyone else's mod.

Duha
Oct 14, 2010, 11:59 PM
That sounds like an excellent idea, and I am up to the challenge. I'll see what I can do this weekend. Ideally, you would should be able to call a function with an ID of your choosing, unique to your mod (or save data, or whatever), and a table or string, and save it, and then retrieve it with just your ID; Without clashing with everyone else's mod.

IMHO game has such fucntion:
modUserData = Modding.OpenUserData(modid, modver);
As I can sugest it creates new table for each modid, modver.

Afforess
Oct 15, 2010, 12:42 AM
The OP said it didn't save the data in-between saves. It just put's it in the SQL table.

Onni
Oct 15, 2010, 04:37 AM
The OP said it didn't save the data in-between saves. It just put's it in the SQL table.
It saves the data into a global SQL table that can be accessed from any save game. You can also save and retrieve your data using row keys as opposite to a single string.

Afforess
Oct 15, 2010, 11:35 AM
It saves the data into a global SQL table that can be accessed from any save game. You can also save and retrieve your data using row keys as opposite to a single string.

But it doesn't store it in the save. It's basically the same as storing it in a text file on the computer. It's fine and dandy if you only have 1 computer, and never share saves...

Duha
Oct 15, 2010, 11:18 PM
But it doesn't store it in the save. It's basically the same as storing it in a text file on the computer. It's fine and dandy if you only have 1 computer, and never share saves...

And how you want to resolve it?
I dont think that appending such data to save file is good idea.

You can save additional data to same dir as civ5 saves with same name and different extension. Some games stored their saves in that way.

Afforess
Oct 15, 2010, 11:56 PM
And how you want to resolve it?
I dont think that appending such data to save file is good idea.

You can save additional data to same dir as civ5 saves with same name and different extension. Some games stored their saves in that way.

You can't append data to saves anyway, however, Civilization 5 has some ways of allowing modders to save data inside of saves like normal data. If you don't know what I'm talking about, I recommend you read a few of the threads here about saving data in the Lua forum. ;)

killmeplease
Oct 18, 2010, 02:52 PM
hello. have just discovered this thread :)
i made a simple table serialization script few days ago (no spaces in string keys and values). so it could be handy i think or provide some foundation to build a data saving system upon. it also can be improved if it will be needed, in terms of performance or universality.

-- Serialization
-- Author: killmeplese
-- DateCreated: 10/17/2010 2:40:28 PM
--------------------------------------------------------------
-- With these functions you can save and load tables to/from string variable.
-- Tables might contain numbers, boolean values, string values WITH NO SPACES,
-- and nested tables. The code was tested in WinQLua program.
--
-- Test:
-- tbl = { 1, 2, 3, t={6,7,8}, a=55, b=56, zz={"alpha", "beta", false} };
-- str = SerializeTable(tbl);
-- print(str);
-- tbl = DeserializeTable(str);
-- str = SerializeTable(tbl);
-- print(str);
--
-- Output:
-- { 1=1 2=2 3=3 a=55 t={ 1=6 2=7 3=8 } zz={ 1=alpha 2=beta 3=false } b=56 }
-- { 1=1 2=2 3=3 a=55 t={ 1=6 2=7 3=8 } zz={ 1=alpha 2=beta 3=false } b=56 }
--------------------------------------------------------------

function SerializeTable(tbl)
local res = "{ ";
local num = 0;
for k, v in pairs(tbl) do
if num > 0 then
res = res.." ";
end
res = res..k.."=";
if type(v) == "table" then
res = res..SerializeTable(v);
else
res = res..tostring(v); -- tostring is necessary to serialize boolean values
end
num = num + 1;
end
return res.." }";
end

--------------------------------------------------------------
-- turned out to be much harder than to serialize, lol
function DeserializeTable(str) -- no spaces in keys and values allowed
local res = {}; -- result table

local _,_,str = string.find(str,"{%s(.*)%s}"); -- trim { } brackets
-- save nested table values to a temp table
local nested = {};
local tableNum = 1;
for t in string.gmatch(str,"{[^{}]*}") do -- match nested tables
table.insert(nested, t);
end
-- cut nested tables from string and replace them with "{}"
local simple = string.gsub(str, "{([^{}]*)}", "{}");
-- parse key=value pairs
for k, v in string.gmatch(simple,"(%w+)=(%S+)") do -- no spaces in keys or values !
-- parse value
if v == "{}" then
v = DeserializeTable(nested[tableNum]); -- recursively call deserialization
tableNum = tableNum + 1;
else
-- parse simple value, it can be a number, a boolean or a word (string with no spaces)
local n = tonumber(v);
if n ~= nil then
v = n;
elseif v == "true" then
v = true;
elseif v == "false" then
v = false;
end
end
-- parse key and add data to table:
local kn = tonumber(k);
if kn ~= nil then
table.insert(res, kn, v);
else
res[k] = v;
end
end
return res; -- return result table
end
-------------------------------------------------------------

it was reported in this thread (http://forums.civfanatics.com/showthread.php?t=391231) that loadstring can be used for low-level deserialization so we can deserialize tables of any complexity with 1 line of code and with greater-than-parsing performance, but i experienced problems with loadstring and havent get it working yet.
Here is a loadstring-variant (no restrictions on keys and values):
function SerializeTable(tbl)
local res = "{ ";
local num = 0;
for k, v in pairs(tbl) do
if num > 0 then
res = res..", ";
end
if type(k) == "number" then
res = res..k.."=";
else
res = res.."["..k.."]".."=";
end
if type(v) == "table" then
res = res..SerializeTable(v);
else
if v == false or v == true or type(v) == "number" then
v = tostring(v);
else
v = "\""..v.."\""; -- this is a string
end
res = res..v;
end
num = num + 1;
end
return res.." }";
end

function DeserializeTable(str)
return assert(loadstring(""..str))();
end

data for different mods can be saved into one table:
data["mod1"] = {...}
data["mod2"] = {...}
etc
so you just deserialize saved script data, put your values in your section and save it back. the downside is poor performance i think as each mod have to serialize/deserialize each time it writes something in this common table, and there might be many of such operations during one turn conducted by several mods. but i cant think of anything better.

it would be great if we had a BeforeGameSaved event so we could store common table in a memory and write it to the script data only on demand. but we have not.

once i browsed civ5luaApi i spotted SaveGame and confused it with an event (it turned out to be a method of Game object). in result i have this non-functioning code:

-- ScriptDataManager
-- Author: killmeplease
-- DateCreated: 10/16/2010 8:37:12 AM
--------------------------------------------------------------

-- its like a 3-dimensional array [category, id, mod] with each element containing mod data (which is a table)

local scriptData = { Players = {}, Plots = {}, Units = {} };

function ScriptData(category, pItem, mod)
if scriptData[category] == nil then
-- unknown category
return nil;
end

local catData = scriptData[category];
local itemID = GetID(category, pItem);

print("Accessing "..category.." id: "..itemID);

local itemData = catData[itemID];
if itemData == nil then -- data for this ID was not loaded from ScriptData
local serializedData = pItem:GetScriptData(); -- load data
if serializedData == "" or serializedData == nil then -- is no data?
-- no data for this ID
catData[itemID] = { [mod] = {} }; -- create empty data for ID and for mod
print("Category entry and mod data were created");
else
-- there is data, deserialize it:
catData[itemID] = assert(loadstring(serializedData))();
if catData[itemID][mod] == nil then -- no data for given mod though
catData[itemID][mod] = {}; -- create empty mod data
print("Mod data was created");
else
print("Mod data was loaded");
end
end
end
return catData[itemID][mod]; -- return cat.item.mod data, loaded or newly created (empty)
end

function GetID(category, pItem)
if category == "Players" then
return pItem:GetID();
elseif category == "Plots" then
return pItem:GetX() * 1000 + pItem:GetY();
elseif category == "Units" then
return pItem:GetOwner() * 1000 + pItem:GetID(); -- GetOwner returns player ID.
end
end

function SaveScriptData()
print("saving script data...");
-- here we have to serialize all data for all changed items and call SetScriptData:
-- set players script data if changed:
for id, v in pairs(scriptData.Players) do
if v ~= nil and v.changed then
Players[id]:SetScriptData(serialize(v));
end
end
-- set plots script data if changed
for id, v in pairs(scriptData.Plots) do
if v ~= nil and v.changed then
local plotX = math.floor(id / 1000);
local plotY = id % 1000;
Map.GetPlotXY(plotX, plotY, 0, 0):SetScriptData(serialize(v));
end
end
-- set units script data if changed
for id, v in pairs(scriptData.Units) do
if v ~= nil and v.changed then
local playerID = math.floor(id / 1000);
local unitID = id % 1000;
Players[playerID]:GetUnit(unitID):SetScriptData(serialize(v));
end
end
print("script data saved");
end

Events.SaveGame.Add( SaveScriptData );

its also buggy. just want to illustrate my idea of in-memory cache.

Afforess
Oct 18, 2010, 04:48 PM
hello. have just discovered this thread :)
i made a simple table serialization script few days ago (no spaces in string keys and values). so it could be handy i think or provide some foundation to build a data saving system upon. it also can be improved if it will be needed, in terms of performance or universality.

Since spaces are obviously important in your method, couldn't you strip all the spaces out beforehand and replace them with something like "[SPACE]", so that it doesn't break when a modder uses a space?

I also noticed the lack of a save game event, which would be handy. Without it, we need to save on the beginning of the players turn.

One thing, to be clear, if I wanted to retrieve my mod's data, I would use ScriptData like this?

ScriptData("Players", pPlayer, 42); With 42 being an arbitrary number?

Two changes I would make:

1.) Map.GetPlotXY(plotX, plotY, 0, 0) can be shortened to Map.GetPlot(plotX, plotY);

2.) check that Players[playerID]:GetUnit(unitID) isn't nil. Units can die.

Afforess
Oct 18, 2010, 11:37 PM
Killtech Loadstring does not work. Firaxis must have disabled it.

Further analysis also revealed you forgot a way to set the data so it could be saved. What's the point of saving data if you can never alter it. ;)

I think I almost have it working. It saves the data correctly, and sets it correctly with a new function I added. But it fails to restore the saved data, and gives back garbage. Hopefully you can see what I'm missing. I've uploaded the changes the SVN, please take a look at them.

killmeplease
Oct 19, 2010, 01:25 AM
now i will look to svn. have not installed tortoise yet, have to do it

spaces can be introduced, i had not bothered with them because i had no need to save strings with spaces. the most easy way is to replace them with "_".

Saving on begin of turn is a weak solution (i thought of it too) because if user will save in the middle all the changes to script data since turn start will be lost.


One thing, to be clear, if I wanted to retrieve my mod's data, I would use ScriptData like this?

ScriptData("Players", pPlayer, 42); With 42 being an arbitrary number?


local playerData = ScriptData("Players", pPlayer, "Revolutions");
playerData.Cities["Berlin"].Loyality = x; -- write
local empireStability = playerData.EmpireStability; -- read
etc

ScriptData returns a table of your data that can be modified as you like. You can use 42 OFC but its intended to be a mod name :)

Data is saved when SaveScriptData is called. It is saved for all mods, players, plots and units (it is being pushed from global scriptData{} cache to ScriptData fields of corresponding objects) and the best place to call it is SaveGame event (if it was present). I dropped this mod as there is no SaveGame event (as it turned out).

killmeplease
Oct 19, 2010, 06:25 AM
The next version for serialization:


function Serialize(tbl)
local res = "{ ";
local num = 0;
for k, v in pairs(tbl) do
if num > 0 then
res = res.." ";
end
res = res..k.."=";
if type(v) == "table" then
res = res..SerializeTable(v);
else
if v == false or v == true or type(v) == "number" then
v = tostring(v);
else
v = v:gsub('"', "\[QUOTE\]");
v = v:gsub('{', "\[LCB\]");
v = v:gsub('}', "\[RCB\]");
v = "\""..v.."\""; -- string
end
res = res..v;
end
num = num + 1;
end
return res.." }";
end

function Deserialize(str)
local tbls = {}; -- nested tables
local topTbl = {};
repeat
topTbl = {}; -- clear variable for top tbl
-- find first top-level table position in str, its values list and key
local strStart, strEnd, topTblKey, topTblVals = str:find("([%w]*)={%s([^{}]*)%s}");
if topTblKey == nil then
strStart, strEnd, topTblVals = str:find("{%s([^{}]*)%s}");
end
--print(topTblVals);

-- parse values:
-- save string values of top table to a temp table
local strings = {};
for vstr in topTblVals:gmatch('"([^"]*)"') do -- match "..." string values
vstr = vstr:gsub("%[QUOTE%]", '"');
vstr = vstr:gsub("%[LCB%]", '{');
vstr = vstr:gsub("%[RCB%]", '}');
table.insert(strings, vstr);
end
-- cut string values from top table string and replace them with "@" symbol
topTblVals = topTblVals:gsub('"[^"]*"', "@");

-- parse key=value pairs
stringNum = 1;
for k, v in topTblVals:gmatch("(%S+)=(%S+)") do
-- parse value
if v == "#" then
v = tbls[k];
-- tbls[k] = nil;
elseif v == "@" then
v = strings[stringNum];
stringNum = stringNum + 1;
else
-- parse simple value, a number or a boolean
local n = tonumber(v);
if n ~= nil then
v = n;
elseif v == "true" then
v = true;
elseif v == "false" then
v = false;
end
end
-- parse key:
local n = tonumber(k);
if n ~= nil then
k = n;
end
-- add value to the table:
topTbl[k] = v;
end
if topTblKey ~= nil then -- key is nil when it is a base table
tbls[topTblKey] = topTbl;
str = str:sub(1, strStart - 1)..topTblKey.."=#"..str:sub(strEnd + 1);
else
str = "#";
end
--print(str);
until str == "#";
return topTbl;
end



test:


>> tbl = { 1, 2, 3, t={6, {7, 7.5}, 8}, a=55, b=56, zz={"alpha", "beta \"or\" {gamma}", false} };
>> str = SerializeTable(tbl);
>> print(str);
>> tbl = Deserialize(str);
>> str = SerializeTable(tbl);
>> print(str);
{ 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 }
{ 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 }



restrictions:
do not use string keys comprised of numbers only (e.g. "234")
do not use spaces in string keys

examples:


tbl = {
[1] = 123, -- ok
["1"] = 453, -- wrong!
["123xz"], -- ok
["abc"] = 64, -- ok
abc2 = 23, -- ok
["a b c"] = 231 -- wrong!
}
local str = Serialize(tbl);



technique that allows multiple mods to access script data: use mod identifier after deserializing script data to access your data.
read:

local str = pPlayer:GetScriptData();
data = Deserialize(str);
revdata = data["Revolutions"];

write:

revdata.somefield = somevalue;
local str = pPlayer:GetScriptData();
data = Deserialize(str);
data["Revolutions"] = revdata;
local str = serialize(data);
pPlayer:SetScriptData(str);

all modders have to use this approach for thier mods be compatible with each other.
otherwise one change for a some item by some mod will erase all other mod's data for this item.

killmeplease
Oct 19, 2010, 07:17 AM
so we can write a wrapper for these routines.
and it will look like

local playerData = GetPlayerData(pPlayer, "Revolutions"); -- read
SetPlayerData(pPlayer, playerData, "Revolutions"); -- write

where playerData is some table with mod's values

internal realization:


function GetPlayerData(pPlayer, modName)
local sdata = pPlayer:GetScriptData();
if sdata ~= nil and sdata ~= "" then
return Deserialize(sdata)[modName];
end
return nil;
end

function SetPlayerData(pPlayer, playerData, modName)
local sdata = pPlayer:GetScriptData();
local data = nil;
if sdata ~= nil and sdata ~= "" then
data = Deserialize(sdata);
data[modName] = playerData;
else
data = { [modName] = playerData };
end
sdata = Serialize(data);
pPlayer:SetScriptData(sdata);
end

function UsePlayerData(pPlayer, modName) -- this increases performance a bit but to prevent conflicts with other mods it has only be used locally
local sdata = pPlayer:GetScriptData();
local data = nil;
if sdata ~= nil and sdata ~= "" then
data = Deserialize(sdata);
else
data = {};
end

local modData = nil;
if data[modName] == nil then
data[modName] = {};
end
modData = data[modName];

local function Save()
pPlayer:SetScriptData(Serialize(data));
end

return modData, Save;
end

-- UsePlayerData example:
local revdata, save = UsePlayerData(pPlayer, "Revolutions");
local somevalue1 = revdata.somefield1;
revdata.somefield2 = somevalue2;
save();

killmeplease
Oct 19, 2010, 08:44 AM
ScriptDataManager reincarnation:


local scriptData = { Players = {}, Plots = {}, Units = {} };

function AccessData(category, pItem, mod)
if not IsValidCategory(category) then
-- unknown category
return nil;
end
local catData = scriptData[category];
local itemData = catData[pItem];
if itemData == nil then -- was not initialized yet (accessing for first time)
local sdata = pItem:GetScriptData(); -- load data
if sdata == "" or sdata == nil then -- is no data?
catData[pItem] = { [mod] = {} }; -- create empty data for item and for mod
else
-- there is data, deserialize it:
catData[pItem] = Deserialize(sdata);
if catData[pItem][mod] == nil then -- no data for given mod though
catData[pItem][mod] = {}; -- create empty
end
end
end

local function Save()
SaveData(category, pItem);
end

return catData[pItem][mod], Save; -- return cat.item.mod data, loaded or newly created (empty) and function to save changes
end

function SaveData(category, pItem) -- save function works for all mods data for an item so no need to specify a mod
if not IsValidCategory(category) or scriptData[category][pItem] == nil then
return;
end
sdata = Serialize(scriptData[category][pItem]);
pItem:SetScriptData(sdata);
end

function IsValidCategory(category)
if category == "Players" or category == "Plots" or category == "Units" then
return true;
end
return false;
end

it loads data on demand and saves it in cache so we have not to deserialize it each time we access it. deserialization occures only once after game is loaded, on first demand.
but we have to save state each time we change something, thus launching serialization:

local revdata, s = AccessData("Players", pPlayer, "Revolutions");
revdata.somefield = somevalue;
s();

Whys
Oct 19, 2010, 06:38 PM
killmeplease,

As you know, we've been working on some of the same functionality. I'm impressed by what you've created here, but disappointed by the lack of class structure. I think our efforts could nicely compliment one another, but I'm having trouble with your latest iteration of deserialize.

This line of code:
for vstr in topTblVals:gmatch('"([^"]*)"') do -- match "..." string values

Produces this error:
...attempt to index local 'topTblVals' (a nil value)

I'd like to suggest the following change so as to handle a nil value:

if topTblVals == nil then
topTblVals = "";
end


I have a good reason for this, really. :)

killmeplease
Oct 19, 2010, 10:01 PM
As you know, we've been working on some of the same functionality. I'm impressed by what you've created here, but disappointed by the lack of class structure.

i do not see a need in making class for 2 global functions.
you can modify this code as you like though, making a class wrapper or such ;)

thanks for bug report. i havent foresaw the situation where a table is empty. so here is the fixed code:


function Deserialize(str)
local tbls = {}; -- nested tables
local topTbl = {};
repeat
topTbl = {}; -- clear variable for top tbl
-- find first top-level table position in str, its values list and key
local strStart, strEnd, topTblKey, topTblVals = str:find("([%w]*)={%s([^{}]*)%s}");
if topTblKey == nil then
strStart, strEnd, topTblVals = str:find("{%s([^{}]*)%s}");
end
topTblVals = topTblVals or ""; -- empty table
...

Whys
Oct 20, 2010, 01:30 AM
Please consider the following code. It allows for ITEMIZED data storage across mods. It does not currently load to cache, but I'm working on it. I assume it works across saves, but I haven't tested that.

Currently, the load function will return the entire save table if no mod name is designated, and additional class functions for retrieving different kinds of data groupings can be easily implemented.

SaveUtils.lua

-- vymdt.01.2010.10.19.0000
-- Created by: Whys -Open source
--================================================== =========================
-- SaveUtils
--================================================== =========================
--[[
Credits: killmeplease -- serialize() and deserialize().
Special Thanks: Thalassicus, Afforess.
]]

-----------------------------------------------------------------------------
-- Save Class
-----------------------------------------------------------------------------
--[[
Allows SetScriptData() and GetScriptData() to behave as a hash-table of
tables while maintaining seperation of individual mod data.
ie:
save( "myMod", pPlayer, key1, table );
save( "myMod", pPlayer, key2, table );
save( "someMod", pPlayer, key1, table );
save( "myMod", pPlot, ...., ..... ); --same as above.
save( "myMod", pUnit, ...., ..... ); --same as above.

load( "myMod", pPlayer, key1 );
load( "myMod", pPlayer, key2 );
load( "someMod", pPlayer, key1 );
load( "myMod", pPlot, .... ); --same as above.
load( "myMod", pUnit, .... ); --same as above.
load( nil, pPlayer ); --all data for pPlayer.
load( nil, pPlot ); --all data for pPlot.
load( nil, pUnit ); --all data for pUnit.
]]
Save = {};

function Save:new( o )
o = o or {};
setmetatable( o, self );
self.__index = self;
self:initKeys( o );
return o;
end

-------------------------------------------------
-- Save:initKeys()
-------------------------------------------------
--[[
Deep assignment of keyed values. Required for
duplication of table data assigned by key.
]]
function Save:initKeys( o )
for k,v in pairs( o ) do
self[k] = v;
if type( v ) == "table" then
self:initKeys( v );
end
end
end

-------------------------------------------------
-- Save:set()
-------------------------------------------------
--[[
Sets the given value to the given key for the
given mod.
]]
function Save:set( mod, key, value )
self[mod] = self[mod] or {};
self[mod][key] = value;
end

-------------------------------------------------
-- Save:get()
-------------------------------------------------
--[[
Returns value for given mod and key. Returns
self when mod not given.
]]
function Save:get( mod, key )
local r = self;
if mod ~= nil and self[mod] ~= nil and key ~= nil then
r = self[mod][key];
end
return r;
end

-----------------------------------------------------------------------------
--END Save Class
-----------------------------------------------------------------------------

-------------------------------------------------
-- save()
-------------------------------------------------
--[[
Saves the given value to the given key for the
given target of the given mod.
]]
function save( mod, target, key, value )
local pSave = load( mod, target );
pSave:set( mod, key, value );
target:SetScriptData(serialize( pSave ));
end

-------------------------------------------------
-- load()
-------------------------------------------------
--[[
Loads the value for the given key for the given
target of the given mod.
]]
function load( mod, target, key )
local pSave = Save:new(deserialize( target:GetScriptData() ));
return pSave:get( mod, key );
end

-------------------------------------------------
-- serialize()
-------------------------------------------------
--[[
Created by: killmeplease.
]]
function serialize( tbl )
local r = "{ ";
local num = 0;
for k,v in pairs( tbl ) do
if num > 0 then
r = r.." ";
end
r = r..k.."=";
if type( v ) == "table" then
r = r..serialize( v );
else
if v == false or v == true or type( v ) == "number" then
v = tostring( v );
else
v = v:gsub('"', "\[QUOTE\]");
v = v:gsub('{', "\[LCB\]");
v = v:gsub('}', "\[RCB\]");
v = "\""..v.."\""; -- string
end
r = r..v;
end
num = num +1;
end
return r.." }";
end

-------------------------------------------------
-- deserialize()
-------------------------------------------------
--[[
Created by: killmeplease.
]]
function deserialize( str )
local tbls = {}; -- nested tables
local topTbl = {};
repeat
topTbl = {}; -- clear variable for top tbl
-- find first top-level table position in str, its values list and key.
local strStart, strEnd, topTblKey, topTblVals = str:find("([%w]*)={%s([^{}]*)%s}");
if topTblKey == nil then
strStart, strEnd, topTblVals = str:find("{%s([^{}]*)%s}");
end
if topTblVals == nil then
topTblVals = "";
end
--print(topTblVals);

-- parse values:
-- save string values of top table to a temp table
local strings = {};
for vstr in topTblVals:gmatch('"([^"]*)"') do -- match "..." string values
vstr = vstr:gsub("%[QUOTE%]", '"');
vstr = vstr:gsub("%[LCB%]", '{');
vstr = vstr:gsub("%[RCB%]", '}');
table.insert(strings, vstr);
end
-- cut string values from top table string and replace them with "@" symbol
topTblVals = topTblVals:gsub('"[^"]*"', "@");

-- parse key=value pairs
stringNum = 1;
for k,v in topTblVals:gmatch("(%S+)=(%S+)") do
-- parse value
if v == "#" then
v = tbls[k];
-- tbls[k] = nil;
elseif v == "@" then
v = strings[stringNum];
stringNum = stringNum +1;
else
-- parse simple value, a number or a boolean
local n = tonumber(v);
if n ~= nil then
v = n;
elseif v == "true" then
v = true;
elseif v == "false" then
v = false;
end
end
-- parse key:
local n = tonumber(k);
if n ~= nil then
k = n;
end
-- add value to the table:
topTbl[k] = v;
end
if topTblKey ~= nil then -- key is nil when it is a base table
tbls[topTblKey] = topTbl;
str = str:sub(1, strStart -1)..topTblKey.."=#"..str:sub(strEnd +1);
else
str = "#";
end
--print(str);
until str == "#";
return topTbl;
end

--================================================== =========================
--END SaveUtils
--================================================== =========================
-- Created by: Whys -Open source



include( "SaveUtils" );

function testThis()

local pPlayer = GetPlayer();
local t = { 1, 2, 3, t={6, {7, 7.5}, 8}, a=55, b=56, zz={"alpha", "beta \"or\" {gamma}", false} };

save( "myMod", pPlayer, "key1", t );
save( "myMod", pPlayer, "key2", {"some value"} );
save( "someMod", pPlayer, "key1", { 3, 2, 1, true } );

print( "test1: "..serialize(load( "myMod", pPlayer, "key1" )) );
print( "test2: "..serialize(load( "myMod", pPlayer, "key2" )) );
print( "test3: "..serialize(load( "someMod", pPlayer, "key1" )) );
print( "test4: "..serialize(load( nil, pPlayer )) );
end
Events.ActivePlayerTurnStart.Add( testThis );

Output:

test1: { 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 }

test2: { 1="some value" }

test3: { 1=3 2=2 3=1 4=true }

test4: { myMod={ key1={ 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 } key2={ 1="some value" } } someMod={ key1={ 1=3 2=2 3=1 4=true } } }

Whys
Oct 20, 2010, 01:36 AM
i do not see a need in making class for 2 global functions.

It's a matter of modularity, efficiency, and readablity. :)

killmeplease
Oct 20, 2010, 02:51 AM
Good code Whys.
What I've learned is passing category parameter being redundant in my code. It remained from old ScriptDataManager realization and should be removed.

Whys
Oct 20, 2010, 11:22 AM
Thanks. :)

By request, I've created another thread for this, but I would like us to remain highly collaborate on it, for SaveUtils.lua would not exist without the contribution of you and others.

http://forums.civfanatics.com/showthread.php?p=9799583#post9799583

Whys
Oct 20, 2010, 11:25 AM
If you're curious, handling that nil value in deserialize() makes it possible to write the following line of code inline.

local pSave = Save:new(deserialize( target:GetScriptData() ));

robk
Oct 20, 2010, 11:42 AM
Thanks. :)

By request, I've created another thread for this, but I would like us to remain highly collaborate on it, for SaveUtils.lua would not exist without the contribution of you and others.

http://forums.civfanatics.com/showthread.php?p=9799583#post9799583

I added that post link to the original post. I've been meaning to summarize everything we've talked about to add to the OP but just haven't found the time.

killmeplease
Oct 20, 2010, 01:15 PM
ScriptDataManager, final (i believe) version:

local scriptData = { }; -- cache table

function AccessData(pItem, modName)
local itemData = sriptData[pItem];
if itemData == nil then
-- was not initialized yet (being accessing for first time)
local sdata = pItem:GetScriptData(); -- load data
if sdata == "" or sdata == nil then -- is no data?
scriptData[pItem] = { [modName] = {} }; -- create empty data for item and for mod
else
-- there is data for a given item, deserialize it:
scriptData[pItem] = Deserialize(sdata);
if scriptData[pItem][modName] == nil then
scriptData[pItem][modName] = {}; -- if no data for a given mod - create empty
end
end
end

-- return mod data, loaded or newly created (empty)
return scriptData[pItem][modName];
end

function SaveData(pItem) -- save function works for all mods data for an item so no need to specify a mod
sdata = Serialize(scriptData[pItem]);
pItem:SetScriptData(sdata);
end

--serialization omitted as there are no changes since post #30


usage:

local pData = AccessData(pPlayer, "MyMod");
local x2 = pData[2].x;
pData.y = x2 * 42;
SaveData(pPlayer);


i feel its good enough for me and i leave it as that.

Whys
Oct 21, 2010, 03:59 AM
I'd like to suggest another change, this time to the serialize function. In it's current form, I can only itemize tables, which kind of defeats the whole point of itemization. The following changes to serialize solves the issue and allows me to itemize all data types.

I've created a local sub routine to avoid duplicating logic within the function, without cluttering up the global scope. It all stays nicely packaged. :)


function serialize( data )

local r = "";
local sub = function( v )
if v == false or v == true or type( v ) == "number" then
v = tostring( v );
else
v = v:gsub('"', "\[QUOTE\]");
v = v:gsub('{', "\[LCB\]");
v = v:gsub('}', "\[RCB\]");
v = "\""..v.."\""; -- string
end
return v;
end

if type(data ) ~= "table" then
r = sub( data );
else
r = "{ ";
local num = 0;
for k,v in pairs( data ) do
if num > 0 then
r = r.." ";
end
r = r..k.."=";
if type( v ) == "table" then
r = r..serialize( v );
else
v = sub( v );
r = r..v;
end
num = num +1;
end
r = r.." }";
end
return r;
end


Test:

include( "SaveUtils" );

function testThis()

local pPlayer = GetPlayer();
local t = { 1, 2, 3, t={6, {7, 7.5}, 8}, a=55, b=56, zz={"alpha", "beta \"or\" {gamma}", false} };

save( "myMod", pPlayer, "key1", "test" );
save( "myMod", pPlayer, "key2", 100 );
save( "someMod", pPlayer, "key1", true );
save( "myMod", pPlayer, "key3", t );

print( "test1: "..serialize(load( "myMod", pPlayer, "key1" )) );
print( "test2: "..serialize(load( "myMod", pPlayer, "key2" )) );
print( "test3: "..serialize(load( "someMod", pPlayer, "key1" )) );
print( "test4: "..serialize(load( "myMod", pPlayer, "key3" )) );
print( "test5: "..serialize(load( nil, pPlayer )) );
end
Events.ActivePlayerTurnStart.Add( testThis );


Output:

test1: "test"
test2: 100
test3: true
test4: { 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 }
test5: { myMod={ key1="test" key3={ 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 } key2=100 } someMod={ key1=true } }

killmeplease
Oct 21, 2010, 04:18 AM
Nice addition

Whys
Oct 21, 2010, 04:19 AM
To be clear, I want my shell to remain cross-compatible so that any mod using either of our implementations will not conflict with one another while saving and loading data. With the above changes in place, compatibility should not be an issue.

Whys
Oct 21, 2010, 01:52 PM
Minor edit. By allowing serialize() to also handle nil as an empty string, I can now itemize the nil value, which does not store the value, but rather removes both the key and value from the table for complete deletion. This appears to be automatic for Lua.

Empty strings are still handled normally.


function serialize( data )

local r = "";
local sub = function( v )
if v == nil then
v = "";
end
if v == false or v == true or type( v ) == "number" then
v = tostring( v );
else
v = v:gsub('"', "\[QUOTE\]");
v = v:gsub('{', "\[LCB\]");
v = v:gsub('}', "\[RCB\]");
v = "\""..v.."\""; -- string
end
return v;
end

if type( data ) ~= "table" then
r = sub( data );
else
r = "{ ";
local num = 0;
for k,v in pairs( data ) do
if num > 0 then
r = r.." ";
end
r = r..k.."=";
if type( v ) == "table" then
r = r..serialize( v );
else
r = r..sub( v );
end
num = num +1;
end
r = r.." }";
end
return r;
end



Test:

include( "SaveUtils" );

function testThis()

local pPlayer = GetPlayer();

save( "myMod", pPlayer, "key1", "test" );
save( "myMod", pPlayer, "key1", nil );
save( "myMod", pPlayer, "key2", "" );

print( "test1: "..serialize(load( "myMod", pPlayer, "key1" )) );
print( "test2: "..serialize(load( nil, pPlayer )) );
end
Events.ActivePlayerTurnStart.Add( testThis );


Output:

test1: ""
test2: { myMod={ key2="" } }

Whys
Oct 21, 2010, 02:09 PM
Just made a small change to the nil check above because a logic error was causing itemized false values to be lost.

Wrong:

v = v or "";


Correct:

if type( v ) == "nil" then
v = "";
end


More Correct:


if v == nil then
v = "";
end

Whys
Oct 27, 2010, 07:41 PM
Version 3 of SaveUtils.lua (http://forums.civfanatics.com/showthread.php?t=392958) is now available.

I've rewritten the serialize() and deserialize() functions from scratch. True to Lua, you can now use any data type except nil for a key.
It is not backwards compatible.

Be aware that while it is technically possible to use any data type except nil for a key, using a table for reference will produce undesirable results. This is not a limitation of the functions, but rather is an inherent limitation of all serialized data. Example below.

Warning:

local key = {"table key"};
local data = {key="value"};
data = deserialize(serialize( data ));
notfound = data[key]; --present but not returned.
print( notfound ); --prints nothing.


Functions: serialize(), deserialize().

--================================================== =========================
--[[
Serializes given data and returns result string.
]]
function serialize( p )

local r = "";
if p ~= nil then
if type( p ) ~= "table" then
if p == nil or p == true or p == false
or type( p ) == "number" then r = tostring( p );
elseif type( p ) == "string" then
if p:lower() == "true" or p:lower() == "false"
or tonumber( p ) ~= nil then r = '"'..p..'"';
else r = p;
end
end
r = r:gsub( "{", "\[LCB\]" );
r = r:gsub( "}", "\[RCB\]" );
r = r:gsub( "=", "\[EQL\]" );
r = r:gsub( ",", "\[COM\]" );
else
r = "{"; local b = false;
for k,v in pairs( p ) do
if b then r = r..","; end
r = r..serialize( k ).."="..serialize( v );
b = true;
end
r = r.."}"
end
end
return r;
end
--================================================== =========================
--[[
Deserializes given string and returns result data.
]]
function deserialize( str )

local findToken = function( str, int )
if int == nil then int = 1; end
local s, e, c = str:find( "({)" ,int);
if s == int then --table.
local len = str:len();
local i = 1; --open brace.
while i > 0 and s ~= nil and e <= len do --find close.
s, e, c = str:find( "([{}])" ,e+1);
if c == "{" then i = i+1;
elseif c == "}" then i = i-1;
end
end
if i == 0 then c = str:sub(int,e);
else print( "deserialize(): Malformed table." ); --error.
end
else s, e, c = str:find( "([^=,]*)" ,int); --primitive.
end
return s, e, c, str:sub( e+1, e+1 );
end

local r = nil; local s, c, d;
if str ~= nil then
local sT, eT, cT = str:find( "{(.*)}" );
if sT == 1 then
r = {}; local len = cT:len(); local e = 1;
if cT ~= "" then
repeat
local t1, t2; local more = false;
s, e, c, d = findToken( cT, e );
if s ~= nil then t1 = deserialize( c ); end
if d == "=" then --key.
s, e, c, d = findToken( cT, e+2 );
if s ~= nil then t2 = deserialize( c ); end
end
if d == "," then e = e+2; more = true; end --one more.
if t2 ~= nil then r[t1] = t2;
else table.insert( r, t1 );
end
until e >= len and not more;
end
elseif tonumber(str) ~= nil then r = tonumber(str);
elseif str == "true" then r = true;
elseif str == "false" then r = false;
else
s, e, c = str:find( '"(.*)"' );
if s == 1 and e == str:len() then
if c == "true" or c == "false" or tonumber( c ) ~= nil then
str = c;
end
end
r = str;
r = r:gsub( "%[LCB%]", "{" );
r = r:gsub( "%[RCB%]", "}" );
r = r:gsub( "%[EQL%]", "=" );
r = r:gsub( "%[COM%]", "," );
end
end
return r;
end


Development Tests:

These tests use the following function for inspecting a table:


--================================================== =========================
--[[
Returns a string representation of any given data type.
]]
function out( p )

local r = "";
if p ~= nil then
if type( p ) ~= "table" then
if p == true or p == false
or type( p ) == "number" then r = tostring( p );
else r = '"'..p..'"';
end
else
r = "{"; local b = false;
for k,v in pairs( p ) do
if b then r = r..","; end
r = r..out( k ).."="..out( v );
b = true;
end
r = r.."}"
end
end
return r;
end


serialize():

print("test1: " ..serialize( "test" ));
print("test2: " ..serialize( 0 ));
print("test3: " ..serialize( 1 ));
print("test4: " ..serialize( 2 ));
print("test5: " ..serialize( "0" ));
print("test6: " ..serialize( "1" ));
print("test7: " ..serialize( "2" ));
print("test8: " ..serialize( true ));
print("test9: " ..serialize( false ));
print("test10: "..serialize( "true" ));
print("test11: "..serialize( "false" ));
print("test12: "..serialize( nil ));
print("test13: "..serialize( {} ));
print("test14: "..serialize( {"test"} ));
print("test15: "..serialize( {"test this"} ));
print("test16: "..serialize( {["test"]="this"} ));
print("test17: "..serialize( {{["test"]="this"}} ));
print("test18: "..serialize( {[{"test"}]={"this"}} ));
print("test19: "..serialize( {"test","this"} ));
print("test20: "..serialize( {["that"]="test","this"} ));
print("test21: "..serialize( {[{}]={}} ));
print("test22: "..serialize( {[{"one"}]={"two"},"three",{"four"}} ));
print("test23: "..serialize( {["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ));
print("test24: "..serialize( {"one",["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ));
print("test25: "..serialize( {"one",["two"]="three","four"} ));
print("test26: "..serialize( {["t"]={6,{7,7.5},8},["a"]=55} ));
print("test27: "..serialize( {1,2,3,["zz"]={"alpha={beta},gamma"},["t"]={6,{7,7.5},8},["zz2"]={"alpha2={beta2},gamma2"},["a"]=55,["b"]=56,false} ));
print("test28: "..serialize( {1,{2},[""]=""} ));
print("test29: "..serialize( {1,{[""]=2},{[3]=""}} ));
print("test30: "..serialize( {"",{},{""},""} ));
print("test31: "..serialize( "{" ));
print("test32: "..serialize( "}" ));
print("test33: "..serialize( "=" ));
print("test34: "..serialize( "," ));


Output:

test1: test
test2: 0
test3: 1
test4: 2
test5: "0"
test6: "1"
test7: "2"
test8: true
test9: false
test10: "true"
test11: "false"
test12:
test13: {}
test14: {1=test}
test15: {1=test this}
test16: {test=this}
test17: {1={test=this}}
test18: {{1=test}={1=this}}
test19: {1=test,2=this}
test20: {1=this,that=test}
test21: {{}={}}
test22: {1=three,2={1=four},{1=one}={1=two}}
test23: {four={1=five,six=seven,eight={}},two=three}
test24: {1=one,four={1=five,six=seven,eight={}},two=three}
test25: {1=one,2=four,two=three}
test26: {a=55,t={1=6,2={1=7,2=7.5},3=8}}
test27: {1=1,2=2,3=3,4=false,zz={1=alpha[EQL][LCB]beta[RCB][COM]gamma},zz2={1=alpha2[EQL][LCB]beta2[RCB][COM]gamma2},b=56,t={1=6,2={1=7,2=7.5},3=8},a=55}
test28: {1=1,2={1=2},=}
test29: {1=1,2={=2},3={3=}}
test30: {1=,2={},3={1=},4=}
test31: [LCB]
test32: [RCB]
test33: [EQL]
test34: [COM]


deserialize():

print("test1: " ..out(deserialize(serialize( "test" ))));
print("test2: " ..out(deserialize(serialize( 0 ))));
print("test3: " ..out(deserialize(serialize( 1 ))));
print("test4: " ..out(deserialize(serialize( 2 ))));
print("test5: " ..out(deserialize(serialize( "0" ))));
print("test6: " ..out(deserialize(serialize( "1" ))));
print("test7: " ..out(deserialize(serialize( "2" ))));
print("test8: " ..out(deserialize(serialize( true ))));
print("test9: " ..out(deserialize(serialize( false ))));
print("test10: "..out(deserialize(serialize( "true" ))));
print("test11: "..out(deserialize(serialize( "false" ))));
print("test12: "..out(deserialize(serialize( nil ))));
print("test13: "..out(deserialize(serialize( {} ))));
print("test14: "..out(deserialize(serialize( {"test"} ))));
print("test15: "..out(deserialize(serialize( {"test this"} ))));
print("test16: "..out(deserialize(serialize( {["test"]="this"} ))));
print("test17: "..out(deserialize(serialize( {{["test"]="this"}} ))));
print("test18: "..out(deserialize(serialize( {[{"test"}]={"this"}} ))));
print("test19: "..out(deserialize(serialize( {"test","this"} ))));
print("test20: "..out(deserialize(serialize( {["that"]="test","this"} ))));
print("test21: "..out(deserialize(serialize( {[{}]={}} ))));
print("test22: "..out(deserialize(serialize( {[{"one"}]={"two"},"three",{"four"}} ))));
print("test23: "..out(deserialize(serialize( {["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ))));
print("test24: "..out(deserialize(serialize( {"one",["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ))));
print("test25: "..out(deserialize(serialize( {"one",["two"]="three","four"} ))));
print("test26: "..out(deserialize(serialize( {["t"]={6,{7,7.5},8},["a"]=55} ))));
print("test27: "..out(deserialize(serialize( {1,2,3,["zz"]={"alpha={beta},gamma"},["t"]={6,{7,7.5},8},["zz2"]={"alpha2={beta2},gamma2"},["a"]=55,["b"]=56,false} ))));
print("test28: "..out(deserialize(serialize( {1,{2},[""]=""} ))));
print("test29: "..out(deserialize(serialize( {1,{[""]=2},{[3]=""}} ))));
print("test30: "..out(deserialize(serialize( {"",{},{""},""} ))));
print("test31: "..out(deserialize(serialize( "{" ))));
print("test32: "..out(deserialize(serialize( "}" ))));
print("test33: "..out(deserialize(serialize( "=" ))));
print("test34: "..out(deserialize(serialize( "," ))));

Output:

test1: "test"
test2: 0
test3: 1
test4: 2
test5: "0"
test6: "1"
test7: "2"
test8: true
test9: false
test10: "true"
test11: "false"
test12: ""
test13: {}
test14: {1="test"}
test15: {1="test this"}
test16: {"test"="this"}
test17: {1={"test"="this"}}
test18: {{1="test"}={1="this"}}
test19: {1="test",2="this"}
test20: {1="this","that"="test"}
test21: {{}={}}
test22: {1="three",2={1="four"},{1="one"}={1="two"}}
test23: {"four"={1="five","six"="seven","eight"={}},"two"="three"}
test24: {1="one","four"={1="five","six"="seven","eight"={}},"two"="three"}
test25: {1="one",2="four","two"="three"}
test26: {"a"=55,"t"={1=6,2={1=7,2=7.5},3=8}}
test27: {1=1,2=2,3=3,4=false,"zz"={1="alpha={beta},gamma"},"zz2"={1="alpha2={beta2},gamma2"},"b"=56,"t"={1=6,2={1=7,2=7.5},3=8},"a"=55}
test28: {1=1,2={1=2},""=""}
test29: {1=1,2={""=2},3={3=""}}
test30: {1="",2={},3={1=""},4=""}
test31: "{"
test32: "}"
test33: "="
test34: ","

Onni
Nov 01, 2010, 09:28 PM
Thank you Afforess, killmeplease and Whys! :goodjob:

I hope you don't mind that I used your serialize and deserialize functions in CityWillard (and possibly in DiploWillard later)? Although I'm not using "SetScriptData", but "Modding.OpenUserData", to store my settings I still needed this serialization. When that "modUserData.SetValue()" is saving data it takes about 0.1 seconds for a row. And even if I tried to be very conservative with the saving it normally took me about 3 seconds to save anything useful. :( But with the serialization I could save all my settings in about 0,5 seconds.

I hope this mod-data-saving is something that Firaxis will change for a proper solution in the future. :rolleyes:

Whys
Nov 02, 2010, 02:50 AM
I recently made some minor tweeks to the serialize() and deserialize(). I don't think it's anything significant, but it's always best to use the latest build.

--================================================== =========================
--[[
Serializes given data and returns result string.
INVALID data types: function, userdata, thread.
]]
function serialize( p )

local r = ""; local t = type( p );
if t == "function" or t == "userdata" or t == "thread" then
print( "serialize(): Invalid type: "..t ); --error.
elseif p ~= nil then
if t ~= "table" then
if p == nil or p == true or p == false
or t == "number" then r = tostring( p );
elseif t == "string" then
if p:lower() == "true" or p:lower() == "false"
or tonumber( p ) ~= nil then r = '"'..p..'"';
else r = p;
end
end
r = r:gsub( "{", "\[LCB\]" );
r = r:gsub( "}", "\[RCB\]" );
r = r:gsub( "=", "\[EQL\]" );
r = r:gsub( ",", "\[COM\]" );
else
r = "{"; local b = false;
for k,v in pairs( p ) do
if b then r = r..","; end
r = r..serialize( k ).."="..serialize( v );
b = true;
end
r = r.."}"
end
end
return r;
end
--================================================== =========================
--[[
Deserializes given string and returns result data.
]]
function deserialize( str )

local findToken = function( str, int )
if int == nil then int = 1; end
local s, e, c = str:find( "({)" ,int);
if s == int then --table.
local len = str:len();
local i = 1; --open brace.
while i > 0 and s ~= nil and e <= len do --find close.
s, e, c = str:find( "([{}])" ,e+1);
if c == "{" then i = i+1;
elseif c == "}" then i = i-1;
end
end
if i == 0 then c = str:sub(int,e);
else print( "deserialize(): Malformed table." ); --error.
end
else s, e, c = str:find( "([^=,]*)" ,int); --primitive.
end
return s, e, c, str:sub( e+1, e+1 );
end

local r = nil; local s, c, d;
if str ~= nil then
local sT, eT, cT = str:find( "{(.*)}" );
if sT == 1 then
r = {}; local len = cT:len(); local e = 1;
if cT ~= "" then
repeat
local t1, t2; local more = false;
s, e, c, d = findToken( cT, e );
if s ~= nil then t1 = deserialize( c ); end
if d == "=" then --key.
s, e, c, d = findToken( cT, e+2 );
if s ~= nil then t2 = deserialize( c ); end
end
if d == "," then e = e+2; more = true; end --one more.
if t2 ~= nil then r[t1] = t2;
else table.insert( r, t1 );
end
until e >= len and not more;
end
elseif tonumber(str) ~= nil then r = tonumber(str);
elseif str == "true" then r = true;
elseif str == "false" then r = false;
else
s, e, c = str:find( '"(.*)"' );
if s == 1 and e == str:len() then
if c == "true" or c == "false" or tonumber( c ) ~= nil then
str = c;
end
end
r = str;
r = r:gsub( "%[LCB%]", "{" );
r = r:gsub( "%[RCB%]", "}" );
r = r:gsub( "%[EQL%]", "=" );
r = r:gsub( "%[COM%]", "," );
end
end
return r;
end

robk
Nov 13, 2010, 01:50 PM
Although it's still unfinished, I've updated the OP to include some info and comparisons between using the ScriptData and modUserData methods for saving data. If anyone has comments to add, especially for the pros and cons sections, please post it and I'll include them the main post.

Still to come some time this weekend: detailed section on ScriptData usage, which will basically say "use SaveUtils or else!". :p

robk
Nov 17, 2010, 01:56 PM
Saving my old method of identifying a saved game when using the modUserDatabase. I don't recommend using this but am keeping it around for reference.

For my Info Addict mod, this wasn't good because I needed a way to save data within the context of a single game. That is, if I wanted the military power of player2 at turn 3, I wanted to make sure it came from the current game and not some other game that I just finished playing (nor did I want games overwriting each other). So, I came up with a sort of hack to uniquely identify a game. I count up the number of non-strategic resources in the game and generate a long string that should, most of the time, act as a unique identifier for that particular game. An example looks like this:

b3c13f67ge7go0i5m6p4s18s6wh25wi7

Now, I have a way to identify a unique game so, to record the military power of player2 at turn 3, I can construct the rowname like so and record the data:

rowname = "b3c13f67ge7go0i5m6p4s18s6wh25wi7-player2-turn3-militarypower";
modUserData.SetValue(rowname, "23");

In practice, I'm actually saving more than just military power on one row but the above example should illustrate how easy it is to save data.

For reference, here's the function I use to generate that long identifier. It's not perfect but, suffices for now.


-- The game ident cache is a global that will hold the game ident
-- info so we don't have to recreate it every time. Should help
-- a smidge with performance.

local g_GameIdentCache = nil;

-- GetGameIdent attempts to generate a string that will uniquely identify
-- a game through save states. It does this by counting resources that I
-- hope don't ever change throughout the duration of a single game. Since
-- there's a random element in generating those resources, I also hope that
-- there is not much chance that two games will generate the exact same number
-- of resources. Just in case someone codes up a resource depletion mod, I'm
-- only counting up non-strategic resources.

-- The g_GameIdentCache variable gets filled the first time this function is
-- called and that string is returned on every subsequent call.

-- Also note that there's a GameInfo.GetName() function provided by Firaxis
-- that may do this same thing. As of right now, that function is reported
-- NYI ("not yet implemented").

function GetGameIdent()

-- Set up a table of resource types we want to include as part of the unique
-- identifier. The value in the table denotes the letter that corresponds to
-- that type in the return string.

if (g_GameIdentCache ~= nil) then
return g_GameIdentCache;
end;

dprint("GetGameIdent() processing")

local validTypes = {}
validTypes.RESOURCE_GOLD = "go"
validTypes.RESOURCE_SILVER = "s"
validTypes.RESOURCE_GEMS = "ge"
validTypes.RESOURCE_MARBLE = "m"
validTypes.RESOURCE_FISH = "f"
validTypes.RESOURCE_WINE = "wi"
validTypes.RESOURCE_INCENSE = "i"
validTypes.RESOURCE_WHEAT = "wh"
validTypes.RESOURCE_COW = "c"
validTypes.RESOURCE_BANANA = "b"
validTypes.RESOURCE_SHEEP = "s"
validTypes.RESOURCE_PEARLS = "p"


-- Build a sorted resources table to make sure the order is always
-- consistent.

function resourceSort(a, b)
return Locale.Compare(a.Type, b.Type) == -1;
end

dprint("Building resource table")
local resources = {}
for resource in GameInfo.Resources() do
table.insert(resources, resource)
end
table.sort(resources, resourceSort)


-- Iterate over the resources that we just sorted and generate the unique
-- string from the resources present in validTypes.

dprint("Building unique string")
local countstr = ""
for i, resource in ipairs(resources) do
rtype = resource.Type
if (validTypes[rtype] ~= nil) then
count = Map.GetNumResources(resource.ID)
countstr = countstr .. validTypes[rtype] .. count
end
end

dprint("Ident = " .. countstr);
g_GameIdentCache = countstr;
dprint("GetGameIdent() finished");
return countstr;

end;


To wrap it all up, here's how you would use that GetGameIdent() to both save and retrieve data:



--saving some data

local modid = "myAwesomeMod";
local modver = 1;

local GameIdent = GetGameIdent();
local modUserData = Modding.OpenUserData(modid, modver);
local turn = Game.GetGameTurn();

-- Loop through active players to record their military power

for iPlayerLoop = 0, GameDefines.MAX_MAJOR_CIVS-1, 1 do
local pPlayer = Players

if(not pPlayer:IsMinorCiv() and pPlayer:IsEverAlive()) then
local pid = pPlayer:GetID();
local data = pPlayer:GetMilitaryMight();
local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
modUserData.SetValue(rowname, data);
end;
end;


-- A little later, in some other function, we need to get player 5's power

local modid = "myAwesomeMod";
local modver = 1;
local GameIdent = GetGameIdent();
local modUserData = Modding.OpenUserData(modid, modver);
local pid = 5;
local turn = Game.GetGameTurn();
local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
local power = modUserData.GetValue(rowname);



[I]Note: Identifying unique games like this may cause problems with fixed map games. Read on in the thread for more details.

robk
Nov 17, 2010, 03:48 PM
OP updated with information on the script data methods and using SaveUtils, along with an example that uses both methods of saving data.

MouseyPounds
May 01, 2011, 09:25 AM
Bumping this with info from Valkrionn (http://forums.civfanatics.com/showpost.php?p=10450746&postcount=43) on the new Civ5SavedGameDatabase.db system.
Looks like it did make it into the February patch.

Essentially, all Civ5 saves now have an SQL database attached, which is accessible by mods. This database is extracted and stored in the cache folder with the file name "Civ5SavedGameDatabase.db". Think of it as SaveUtils, but cleanly incorporated.

The cached file is the file that is actually used (this means external tools can access it as well).

Accessing the database from within Lua is simple.


local savedData = Modding.OpenSaveData();
if(savedData ~= nil) then --It might be nil if there was an error and the file doesn't exist/is corrupted.
savedData.SetValue(name, someValue); --Assign a simple value into the SimpleValues table.

local value = savedData.GetValue(name); --Retrieve the value in SimpleValues that matches the name.

local query = savedData.Query("SELECT Value from SimpleValues"); -- Full SQL access to the db :)
for row in query do
print(row.Value);
end
end

Whys
May 01, 2011, 01:35 PM
Some additional food for thought.

http://forums.civfanatics.com/showthread.php?p=10452065#post10452065

MouseyPounds
May 23, 2011, 02:01 PM
In an effort to keep this reference up-to-date, I'm reposting recent findings regarding the Civ5SavedGameDatabase.db system. We have confirmed that the Query method gives full access to the database so that you can create and manipulate custom tables if the SimpleValues table is not robust enough for your needs. A simple example:


local saveDB = Modding.OpenSaveData()
-- Create a table
for row in saveDB:Query('CREATE TABLE MyTest("ID" NOT NULL PRIMARY KEY, "A" INTEGER, "B" TEXT)') do end
-- Insert something
for row in saveDB:Query('INSERT INTO MyTest("ID","A","B") VALUES(1,2,"three")') do end
-- Retrieve everything
for row in saveDB:Query("SELECT * FROM MyTest") do
print(row.ID, row.A, row.B)
end


Whys is looking at updating SaveUtils to provide easy access to do all this; see the recent discussion in the SaveUtils topic. (http://forums.civfanatics.com/showthread.php?p=10514925#post10514925)

robk
May 23, 2011, 03:07 PM
Thanks Mousey. I added a quick note at the top of the original post pointing to your reply so that it's not lost in the shuffle.