Storing data across saved games from LUA

Minor edit. By allowing serialize() to also handle nil as an empty string, I can now itemize the nil value, which does not store the value, but rather removes both the key and value from the table for complete deletion. This appears to be automatic for Lua.

Empty strings are still handled normally.

Code:
function serialize( data )
	
	local r = "";
	local sub = function( v )
		[COLOR="Blue"]if v == nil then
			v = "";
		end[/COLOR]
		if v == false or v == true or type( v ) == "number" then
			v = tostring( v );
		else
			v = v:gsub('"', "\[QUOTE\]");
			v = v:gsub('{', "\[LCB\]");
			v = v:gsub('}', "\[RCB\]");
			v = "\""..v.."\""; -- string
		end
		return v;
	end

	if type( data ) ~= "table" then
		r = sub( data );
	else
		r = "{ ";
		local num = 0;
		for k,v in pairs( data ) do
			if num > 0 then
				r = r.." ";
			end
			r = r..k.."=";
			if type( v ) == "table" then
				r = r..serialize( v );
			else
				r = r..sub( v );
			end
			num = num +1;
		end
		r = r.." }";
	end
	return r;
end


Test:
Code:
include( "SaveUtils" );

function testThis()
	
	local pPlayer = GetPlayer();
	
	save( "myMod", pPlayer, "key1", "test" );
	save( "myMod", pPlayer, "key1", nil );
	save( "myMod", pPlayer, "key2", "" );

	print( "test1: "..serialize(load( "myMod",   pPlayer, "key1" )) );
	print( "test2: "..serialize(load( nil, pPlayer )) );
end
Events.ActivePlayerTurnStart.Add( testThis );

Output:
Code:
test1: ""
test2: { myMod={ key2="" } }
 
Just made a small change to the nil check above because a logic error was causing itemized false values to be lost.

Wrong:
Code:
v = v or "";

Correct:
Code:
if type( v ) == "nil" then
			v = "";
		end

More Correct:

Code:
if v == nil then
			v = "";
		end
 
Version 3 of SaveUtils.lua is now available.

I've rewritten the serialize() and deserialize() functions from scratch. True to Lua, you can now use any data type except nil for a key.
It is not backwards compatible.

Be aware that while it is technically possible to use any data type except nil for a key, using a table for reference will produce undesirable results. This is not a limitation of the functions, but rather is an inherent limitation of all serialized data. Example below.

Warning:
Code:
local key = {"table key"};
local data = {key="value"};
data = deserialize(serialize( data ));
notfound = data[key]; --present but not returned.
print( notfound ); --prints nothing.

Functions: serialize(), deserialize().
Code:
--===========================================================================
--[[
Serializes given data and returns result string.
]]
function serialize( p )
	
	local r = "";
	if p ~= nil then
		if type( p ) ~= "table" then
			if p == nil or p == true or p == false
				or type( p ) == "number" then r = tostring( p );
			elseif type( p ) == "string" then
				if p:lower() == "true" or p:lower() == "false"
						or tonumber( p ) ~= nil then r = '"'..p..'"';
				else r = p;
				end
			end
			r = r:gsub( "{", "\[LCB\]" );
			r = r:gsub( "}", "\[RCB\]" );
			r = r:gsub( "=", "\[EQL\]" );
			r = r:gsub( ",", "\[COM\]" );
		else
			r = "{"; local b = false;
			for k,v in pairs( p ) do
				if b then r = r..","; end
				r = r..serialize( k ).."="..serialize( v );
				b = true;
			end
			r = r.."}"
		end
	end
	return r;
end
--===========================================================================
--[[
Deserializes given string and returns result data.
]]
function deserialize( str )

	local findToken = function( str, int )
		if int == nil then int = 1; end
		local s, e, c = str:find( "({)" ,int);
		if s == int then --table.
			local len = str:len();
			local i = 1; --open brace.
			while i > 0 and s ~= nil and e <= len do --find close.
				s, e, c = str:find( "([{}])" ,e+1);
				if     c == "{" then i = i+1;
				elseif c == "}" then i = i-1;
				end
			end
			if i == 0 then c = str:sub(int,e);
			else print( "deserialize(): Malformed table." ); --error.
			end
		else s, e, c = str:find( "([^=,]*)" ,int); --primitive.
		end
		return s, e, c, str:sub( e+1, e+1 );
	end

	local r = nil; local s, c, d;
	if str ~= nil then
		local sT, eT, cT = str:find( "{(.*)}" );
		if sT == 1 then
			r = {}; local len = cT:len(); local e = 1;
			if cT ~= "" then
				repeat
					local t1, t2; local more = false;
					s, e, c, d = findToken( cT, e );
					if s ~= nil then t1 = deserialize( c ); end
					if d == "=" then --key.
						s, e, c, d = findToken( cT, e+2 );
						if s ~= nil then t2 = deserialize( c ); end
					end
					if d == "," then e = e+2; more = true; end --one more.
					if t2 ~= nil then r[t1] = t2;
					else table.insert( r, t1 );
					end
				until e >= len and not more;
			end
		elseif tonumber(str) ~= nil then r = tonumber(str);
		elseif str == "true"  then r = true;
		elseif str == "false" then r = false;
		else
			s, e, c = str:find( '"(.*)"' );
			if s == 1 and e == str:len() then
				if c == "true" or c == "false" or tonumber( c ) ~= nil then
					str = c;
				end
			end
			r = str;
			r = r:gsub( "%[LCB%]", "{" );
			r = r:gsub( "%[RCB%]", "}" );
			r = r:gsub( "%[EQL%]", "=" );
			r = r:gsub( "%[COM%]", "," );
		end
	end
	return r;
end

Development Tests:
Spoiler :

These tests use the following function for inspecting a table:
Spoiler :

Code:
--===========================================================================
--[[
Returns a string representation of any given data type.
]]
function out( p )

	local r = "";
	if p ~= nil then
		if type( p ) ~= "table" then
			if p == true or p == false
				or type( p ) == "number" then r = tostring( p );
			else r = '"'..p..'"';
			end
		else
			r = "{"; local b = false;
			for k,v in pairs( p ) do
				if b then r = r..","; end
				r = r..out( k ).."="..out( v );
				b = true;
			end
			r = r.."}"
		end
	end
	return r;
end

serialize():
Code:
print("test1: " ..serialize( "test" ));
print("test2: " ..serialize( 0 ));
print("test3: " ..serialize( 1 ));
print("test4: " ..serialize( 2 ));
print("test5: " ..serialize( "0" ));
print("test6: " ..serialize( "1" ));
print("test7: " ..serialize( "2" ));
print("test8: " ..serialize( true ));
print("test9: " ..serialize( false ));
print("test10: "..serialize( "true" ));
print("test11: "..serialize( "false" ));
print("test12: "..serialize( nil ));
print("test13: "..serialize( {} ));
print("test14: "..serialize( {"test"} ));
print("test15: "..serialize( {"test this"} ));
print("test16: "..serialize( {["test"]="this"} ));
print("test17: "..serialize( {{["test"]="this"}} ));
print("test18: "..serialize( {[{"test"}]={"this"}} ));
print("test19: "..serialize( {"test","this"} ));
print("test20: "..serialize( {["that"]="test","this"} ));
print("test21: "..serialize( {[{}]={}} ));
print("test22: "..serialize( {[{"one"}]={"two"},"three",{"four"}} ));
print("test23: "..serialize( {["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ));
print("test24: "..serialize( {"one",["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ));
print("test25: "..serialize( {"one",["two"]="three","four"} ));
print("test26: "..serialize( {["t"]={6,{7,7.5},8},["a"]=55} ));
print("test27: "..serialize( {1,2,3,["zz"]={"alpha={beta},gamma"},["t"]={6,{7,7.5},8},["zz2"]={"alpha2={beta2},gamma2"},["a"]=55,["b"]=56,false} ));
print("test28: "..serialize( {1,{2},[""]=""} ));
print("test29: "..serialize( {1,{[""]=2},{[3]=""}} ));
print("test30: "..serialize( {"",{},{""},""} ));
print("test31: "..serialize( "{" ));
print("test32: "..serialize( "}" ));
print("test33: "..serialize( "=" ));
print("test34: "..serialize( "," ));

Output:
Code:
test1: test
test2: 0
test3: 1
test4: 2
test5: "0"
test6: "1"
test7: "2"
test8: true
test9: false
test10: "true"
test11: "false"
test12: 
test13: {}
test14: {1=test}
test15: {1=test this}
test16: {test=this}
test17: {1={test=this}}
test18: {{1=test}={1=this}}
test19: {1=test,2=this}
test20: {1=this,that=test}
test21: {{}={}}
test22: {1=three,2={1=four},{1=one}={1=two}}
test23: {four={1=five,six=seven,eight={}},two=three}
test24: {1=one,four={1=five,six=seven,eight={}},two=three}
test25: {1=one,2=four,two=three}
test26: {a=55,t={1=6,2={1=7,2=7.5},3=8}}
test27: {1=1,2=2,3=3,4=false,zz={1=alpha[EQL][LCB]beta[RCB][COM]gamma},zz2={1=alpha2[EQL][LCB]beta2[RCB][COM]gamma2},b=56,t={1=6,2={1=7,2=7.5},3=8},a=55}
test28: {1=1,2={1=2},=}
test29: {1=1,2={=2},3={3=}}
test30: {1=,2={},3={1=},4=}
test31: [LCB]
test32: [RCB]
test33: [EQL]
test34: [COM]

deserialize():
Code:
print("test1: " ..out(deserialize(serialize( "test" ))));
print("test2: " ..out(deserialize(serialize( 0 ))));
print("test3: " ..out(deserialize(serialize( 1 ))));
print("test4: " ..out(deserialize(serialize( 2 ))));
print("test5: " ..out(deserialize(serialize( "0" ))));
print("test6: " ..out(deserialize(serialize( "1" ))));
print("test7: " ..out(deserialize(serialize( "2" ))));
print("test8: " ..out(deserialize(serialize( true ))));
print("test9: " ..out(deserialize(serialize( false ))));
print("test10: "..out(deserialize(serialize( "true" ))));
print("test11: "..out(deserialize(serialize( "false" ))));
print("test12: "..out(deserialize(serialize( nil ))));
print("test13: "..out(deserialize(serialize( {} ))));
print("test14: "..out(deserialize(serialize( {"test"} ))));
print("test15: "..out(deserialize(serialize( {"test this"} ))));
print("test16: "..out(deserialize(serialize( {["test"]="this"} ))));
print("test17: "..out(deserialize(serialize( {{["test"]="this"}} ))));
print("test18: "..out(deserialize(serialize( {[{"test"}]={"this"}} ))));
print("test19: "..out(deserialize(serialize( {"test","this"} ))));
print("test20: "..out(deserialize(serialize( {["that"]="test","this"} ))));
print("test21: "..out(deserialize(serialize( {[{}]={}} ))));
print("test22: "..out(deserialize(serialize( {[{"one"}]={"two"},"three",{"four"}} ))));
print("test23: "..out(deserialize(serialize( {["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ))));
print("test24: "..out(deserialize(serialize( {"one",["two"]="three",["four"]={"five",["six"]="seven",["eight"]={}}} ))));
print("test25: "..out(deserialize(serialize( {"one",["two"]="three","four"} ))));
print("test26: "..out(deserialize(serialize( {["t"]={6,{7,7.5},8},["a"]=55} ))));
print("test27: "..out(deserialize(serialize( {1,2,3,["zz"]={"alpha={beta},gamma"},["t"]={6,{7,7.5},8},["zz2"]={"alpha2={beta2},gamma2"},["a"]=55,["b"]=56,false} ))));
print("test28: "..out(deserialize(serialize( {1,{2},[""]=""} ))));
print("test29: "..out(deserialize(serialize( {1,{[""]=2},{[3]=""}} ))));
print("test30: "..out(deserialize(serialize( {"",{},{""},""} ))));
print("test31: "..out(deserialize(serialize( "{" ))));
print("test32: "..out(deserialize(serialize( "}" ))));
print("test33: "..out(deserialize(serialize( "=" ))));
print("test34: "..out(deserialize(serialize( "," ))));

Output:
Code:
test1: "test"
test2: 0
test3: 1
test4: 2
test5: "0"
test6: "1"
test7: "2"
test8: true
test9: false
test10: "true"
test11: "false"
test12: ""
test13: {}
test14: {1="test"}
test15: {1="test this"}
test16: {"test"="this"}
test17: {1={"test"="this"}}
test18: {{1="test"}={1="this"}}
test19: {1="test",2="this"}
test20: {1="this","that"="test"}
test21: {{}={}}
test22: {1="three",2={1="four"},{1="one"}={1="two"}}
test23: {"four"={1="five","six"="seven","eight"={}},"two"="three"}
test24: {1="one","four"={1="five","six"="seven","eight"={}},"two"="three"}
test25: {1="one",2="four","two"="three"}
test26: {"a"=55,"t"={1=6,2={1=7,2=7.5},3=8}}
test27: {1=1,2=2,3=3,4=false,"zz"={1="alpha={beta},gamma"},"zz2"={1="alpha2={beta2},gamma2"},"b"=56,"t"={1=6,2={1=7,2=7.5},3=8},"a"=55}
test28: {1=1,2={1=2},""=""}
test29: {1=1,2={""=2},3={3=""}}
test30: {1="",2={},3={1=""},4=""}
test31: "{"
test32: "}"
test33: "="
test34: ","
 
Thank you Afforess, killmeplease and Whys! :goodjob:

I hope you don't mind that I used your serialize and deserialize functions in CityWillard (and possibly in DiploWillard later)? Although I'm not using "SetScriptData", but "Modding.OpenUserData", to store my settings I still needed this serialization. When that "modUserData.SetValue()" is saving data it takes about 0.1 seconds for a row. And even if I tried to be very conservative with the saving it normally took me about 3 seconds to save anything useful. :( But with the serialization I could save all my settings in about 0,5 seconds.

I hope this mod-data-saving is something that Firaxis will change for a proper solution in the future. :rolleyes:
 
I recently made some minor tweeks to the serialize() and deserialize(). I don't think it's anything significant, but it's always best to use the latest build.

Code:
--===========================================================================
--[[
Serializes given data and returns result string.
INVALID data types: function, userdata, thread.
]]
function serialize( p )
	
	local r = ""; local t = type( p );
	if t == "function" or t == "userdata" or t == "thread" then
		print( "serialize(): Invalid type: "..t ); --error.
	elseif p ~= nil then
		if t ~= "table" then
			if p == nil or p == true or p == false
				or t == "number" then r = tostring( p );
			elseif t == "string" then
				if p:lower() == "true" or p:lower() == "false"
						or tonumber( p ) ~= nil then r = '"'..p..'"';
				else r = p;
				end
			end
			r = r:gsub( "{", "\[LCB\]" );
			r = r:gsub( "}", "\[RCB\]" );
			r = r:gsub( "=", "\[EQL\]" );
			r = r:gsub( ",", "\[COM\]" );
		else
			r = "{"; local b = false;
			for k,v in pairs( p ) do
				if b then r = r..","; end
				r = r..serialize( k ).."="..serialize( v );
				b = true;
			end
			r = r.."}"
		end
	end
	return r;
end
--===========================================================================
--[[
Deserializes given string and returns result data.
]]
function deserialize( str )

	local findToken = function( str, int )
		if int == nil then int = 1; end
		local s, e, c = str:find( "({)" ,int);
		if s == int then --table.
			local len = str:len();
			local i = 1; --open brace.
			while i > 0 and s ~= nil and e <= len do --find close.
				s, e, c = str:find( "([{}])" ,e+1);
				if     c == "{" then i = i+1;
				elseif c == "}" then i = i-1;
				end
			end
			if i == 0 then c = str:sub(int,e);
			else print( "deserialize(): Malformed table." ); --error.
			end
		else s, e, c = str:find( "([^=,]*)" ,int); --primitive.
		end
		return s, e, c, str:sub( e+1, e+1 );
	end

	local r = nil; local s, c, d;
	if str ~= nil then
		local sT, eT, cT = str:find( "{(.*)}" );
		if sT == 1 then
			r = {}; local len = cT:len(); local e = 1;
			if cT ~= "" then
				repeat
					local t1, t2; local more = false;
					s, e, c, d = findToken( cT, e );
					if s ~= nil then t1 = deserialize( c ); end
					if d == "=" then --key.
						s, e, c, d = findToken( cT, e+2 );
						if s ~= nil then t2 = deserialize( c ); end
					end
					if d == "," then e = e+2; more = true; end --one more.
					if t2 ~= nil then r[t1] = t2;
					else table.insert( r, t1 );
					end
				until e >= len and not more;
			end
		elseif tonumber(str) ~= nil then r = tonumber(str);
		elseif str == "true"  then r = true;
		elseif str == "false" then r = false;
		else
			s, e, c = str:find( '"(.*)"' );
			if s == 1 and e == str:len() then
				if c == "true" or c == "false" or tonumber( c ) ~= nil then
					str = c;
				end
			end
			r = str;
			r = r:gsub( "%[LCB%]", "{" );
			r = r:gsub( "%[RCB%]", "}" );
			r = r:gsub( "%[EQL%]", "=" );
			r = r:gsub( "%[COM%]", "," );
		end
	end
	return r;
end
 
Although it's still unfinished, I've updated the OP to include some info and comparisons between using the ScriptData and modUserData methods for saving data. If anyone has comments to add, especially for the pros and cons sections, please post it and I'll include them the main post.

Still to come some time this weekend: detailed section on ScriptData usage, which will basically say "use SaveUtils or else!". :p
 
Saving my old method of identifying a saved game when using the modUserDatabase. I don't recommend using this but am keeping it around for reference.

Spoiler :
For my Info Addict mod, this wasn't good because I needed a way to save data within the context of a single game. That is, if I wanted the military power of player2 at turn 3, I wanted to make sure it came from the current game and not some other game that I just finished playing (nor did I want games overwriting each other). So, I came up with a sort of hack to uniquely identify a game. I count up the number of non-strategic resources in the game and generate a long string that should, most of the time, act as a unique identifier for that particular game. An example looks like this:

Code:
b3c13f67ge7go0i5m6p4s18s6wh25wi7

Now, I have a way to identify a unique game so, to record the military power of player2 at turn 3, I can construct the rowname like so and record the data:

Code:
rowname = "b3c13f67ge7go0i5m6p4s18s6wh25wi7-player2-turn3-militarypower";
modUserData.SetValue(rowname, "23");

In practice, I'm actually saving more than just military power on one row but the above example should illustrate how easy it is to save data.

For reference, here's the function I use to generate that long identifier. It's not perfect but, suffices for now.

Code:
-- The game ident cache is a global that will hold the game ident
-- info so we don't have to recreate it every time. Should help
-- a smidge with performance.

local g_GameIdentCache = nil;

-- GetGameIdent attempts to generate a string that will uniquely identify
-- a game through save states. It does this by counting resources that I
-- hope don't ever change throughout the duration of a single game. Since
-- there's a random element in generating those resources, I also hope that
-- there is not much chance that two games will generate the exact same number
-- of resources. Just in case someone codes up a resource depletion mod, I'm
-- only counting up non-strategic resources.

-- The g_GameIdentCache variable gets filled the first time this function is
-- called and that string is returned on every subsequent call.

-- Also note that there's a GameInfo.GetName() function provided by Firaxis
-- that may do this same thing. As of right now, that function is reported
-- NYI ("not yet implemented").
  
function GetGameIdent()
  
  -- Set up a table of resource types we want to include as part of the unique
  -- identifier. The value in the table denotes the letter that corresponds to
  -- that type in the return string.

  if (g_GameIdentCache ~= nil) then
    return g_GameIdentCache;
  end;

  dprint("GetGameIdent() processing")

  local validTypes = {}
  validTypes.RESOURCE_GOLD = "go"
  validTypes.RESOURCE_SILVER = "s"
  validTypes.RESOURCE_GEMS = "ge"
  validTypes.RESOURCE_MARBLE = "m"
  validTypes.RESOURCE_FISH = "f"
  validTypes.RESOURCE_WINE = "wi"
  validTypes.RESOURCE_INCENSE = "i"
  validTypes.RESOURCE_WHEAT = "wh"
  validTypes.RESOURCE_COW = "c"
  validTypes.RESOURCE_BANANA = "b"
  validTypes.RESOURCE_SHEEP = "s"
  validTypes.RESOURCE_PEARLS = "p"
  

  -- Build a sorted resources table to make sure the order is always
  -- consistent. 

  function resourceSort(a, b)
    return Locale.Compare(a.Type, b.Type) == -1;
  end

  dprint("Building resource table")
  local resources = {}
  for resource in GameInfo.Resources() do
      table.insert(resources, resource)
  end
  table.sort(resources, resourceSort)


  -- Iterate over the resources that we just sorted and generate the unique
  -- string from the resources present in validTypes.

  dprint("Building unique string")
  local countstr = ""
  for i, resource in ipairs(resources) do
    rtype = resource.Type
      if (validTypes[rtype] ~= nil) then
      count = Map.GetNumResources(resource.ID)
        countstr = countstr .. validTypes[rtype] .. count
    end
  end

  dprint("Ident = " .. countstr);
  g_GameIdentCache = countstr;
  dprint("GetGameIdent() finished");
  return countstr;

end;


To wrap it all up, here's how you would use that GetGameIdent() to both save and retrieve data:

Code:
--saving some data

local modid = "myAwesomeMod";
local modver = 1;

local GameIdent = GetGameIdent();
local modUserData = Modding.OpenUserData(modid, modver);
local turn = Game.GetGameTurn();

-- Loop through active players to record their military power

for iPlayerLoop = 0, GameDefines.MAX_MAJOR_CIVS-1, 1 do
  local pPlayer = Players[iPlayerLoop]

    if(not pPlayer:IsMinorCiv() and pPlayer:IsEverAlive()) then
      local pid = pPlayer:GetID();
      local data = pPlayer:GetMilitaryMight();
      local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
      modUserData.SetValue(rowname, data);
    end;
end;


-- A little later, in some other function, we need to get player 5's power

local modid = "myAwesomeMod";
local modver = 1;
local GameIdent = GetGameIdent();
local modUserData = Modding.OpenUserData(modid, modver);
local pid = 5;
local turn = Game.GetGameTurn();
local rowname = GameIdent .. "-turn" .. turn .. "-player" .. pid;
local power = modUserData.GetValue(rowname);

Note: Identifying unique games like this may cause problems with fixed map games. Read on in the thread for more details.
 
OP updated with information on the script data methods and using SaveUtils, along with an example that uses both methods of saving data.
 
Bumping this with info from Valkrionn on the new Civ5SavedGameDatabase.db system.
Looks like it did make it into the February patch.

Essentially, all Civ5 saves now have an SQL database attached, which is accessible by mods. This database is extracted and stored in the cache folder with the file name "Civ5SavedGameDatabase.db". Think of it as SaveUtils, but cleanly incorporated.

The cached file is the file that is actually used (this means external tools can access it as well).

Accessing the database from within Lua is simple.

Code:
local savedData = Modding.OpenSaveData();
if(savedData ~= nil) then --It might be nil if there was an error and the file doesn't exist/is corrupted.
  savedData.SetValue(name, someValue); --Assign a simple value into the SimpleValues table.

  local value = savedData.GetValue(name); --Retrieve the value in SimpleValues that matches the name.

  local query = savedData.Query("SELECT Value from SimpleValues"); -- Full SQL access to the db :)
   for row in query do
        print(row.Value);
   end
end
 
In an effort to keep this reference up-to-date, I'm reposting recent findings regarding the Civ5SavedGameDatabase.db system. We have confirmed that the Query method gives full access to the database so that you can create and manipulate custom tables if the SimpleValues table is not robust enough for your needs. A simple example:

Code:
local saveDB = Modding.OpenSaveData()
-- Create a table
for row in saveDB:Query('CREATE TABLE MyTest("ID" NOT NULL PRIMARY KEY, "A" INTEGER, "B" TEXT)') do end
-- Insert something
for row in saveDB:Query('INSERT INTO MyTest("ID","A","B") VALUES(1,2,"three")') do end
-- Retrieve everything
for row in saveDB:Query("SELECT * FROM MyTest") do 
  print(row.ID, row.A, row.B)
end

Whys is looking at updating SaveUtils to provide easy access to do all this; see the recent discussion in the SaveUtils topic.
 
Thanks Mousey. I added a quick note at the top of the original post pointing to your reply so that it's not lost in the shuffle.
 
Back
Top Bottom