Pazyryk
Oct 06, 2011, 05:28 PM
Current version: 0.12
Changes:
0.11 (Sep 23, 2012) fixed a bug in 0.10 that caused a crash when a variable changed type.
0.12 (May 1, 2013) adds TableSave to LuaEvents so it can be called from another state.
The idea behind TableSaverLoader is that you do all game logic in Lua tables, then reference those tables in a master table for "preservation" through game exit/reload. TableSaverLoader (hooked up properly as shown in attached "ExampleMain.lua" below) will save tables of any arbitrary size* and nesting structure to SavedGameDB, then restore them on game reload. There is no issue about when you save/exit because it intercepts all 4 gamesave types: cntl-S, save from menu, F11 quicksave and endturn for autosave. The code for the gamesave interception is actually in the ExampleMain.lua file attached below.
Download here: http://forums.civfanatics.com/downloads.php?do=file&id=17824 (See How to Hook Up section below.)
(*No limit to size as far as I know, though there are always time considerations. For 16739 table elements, TableSave takes 22 sec to save the first time and about 13 seconds thereafter when all or most elements don't change, and TableLoad 0.175 seconds to load this data. That's a bit of a wait at turn end ... hopefully you won't have 16739 things to save. This is on my 5-yr-old computer running a Standard map.)
TableSave(gT, "MyMod") will save gT and all nested values and tables to a single table called "MyMod_Data" in SavedGameDB. A second table called "MyMod_Info" keeps save/load info including a checksum used to test data integrity on load.
TableLoad(gT, "MyMod") will rebuild gT after a game load.
I've included two files. One is "TableSaverLoader" with the two functions to call. The second is "ExampleMain" that calls TableSave before any manual game saves (it intercepts quicksave, cntr-s and save from menu; this code taken from Gedemon) and at turn end (so that autosave is updated), and calls TableLoad after loading a game.
My ExampleMain sets up a global table ("gT") and "preloads" it with two other tables:
gValues = {}
gPlots = {}
gT = { gValues, gPlots } --my global table that holds everything to be saved
To illustrate, here's an interactive FireTuner session using TableSaverLoader and ExampleMain to drive it:
START A NEW GAME, PRESS "Begin Journey"
--prints TableSave time: 0.36800000000005, checksum: 99, inserts: 2, updates: 0, unchanged: 0, deletes: 0
print(gT) -- prints table: 25EDF9C0
print(gValues) -- prints table: 25EDF858
print(gT[1]) -- prints table: 25EDF858
END TURN
--prints TableSave time: 0.095000000000027, checksum: 99, inserts: 0, updates: 0, unchanged: 2, deletes: 0
a_table = {x=1,y=2,z=3}
a_nested_table = {'a',{'b',{a_table,{{{{{'hello there'}}}}}}}}
gValues.bBoolean = true
gValues.a_table = a_table
gValues.a_nested_table = a_nested_table
print(gValues.bBoolean) --prints true
print(gValues.a_nested_table[2][2][2][1][1][1][1][1]) --prints hello there
print(a_table) --prints table: 266B1EF0
print(gValues.a_table) --prints table: 266B1EF0
print(gValues.a_nested_table[2][2][1]) --prints table: 266B1EF0
print(gValues.a_table.y) --prints 2
print(gValues.a_nested_table[2][2][1].y) --prints 2
F11 (QUICKSAVE)
--prints TableSave time: 0.14399999999978, checksum: 1243, inserts: 17, updates: 0, unchanged: 2, deletes: 0
EXIT TO MAIN MENU AND LOAD QUICKSAVE, PRESS "Continue your journey"
--prints TableLoad ran without error; checksum: 1243
print(gValues.bBoolean) --prints true
print(gValues.a_nested_table[2][2][2][1][1][1][1][1]) --prints hello there
print(a_table) --prints nil
print(gValues.a_table) --prints table: 155CB858
print(gValues.a_nested_table[2][2][1]) --prints table: 155CB858
--note that a_table wasn't saved (it isn't in gT)
--however, two "paths" to the same table in gT point to the same table (this was a devil to make work!)
print(gValues.a_table.y) --prints 2
print(gValues.a_nested_table[2][2][1].y) --prints 2
DebugFillgPlots() --this function gets plotyType, featureType and improvementType for each plot, and puts them in gPlots
--prints found 4160 plots
F11 (QUICKSAVE)
--prints Getting DB info for TableSave (this is the first save after a game load)...
--prints TableSave time: 21.983, checksum: 841967, inserts: 16720, updates: 0, unchanged: 19, deletes: 0
--(MyMod_Data now has 16739 rows)
print(gPlots[64][50].plotType) --prints 3
print(gPlots[64][50].featureType) --prints 0
print(gPlots[64][50].improvementType) --prints -1
NEXT TURN
--prints TableSave time: 13.491, checksum: 841967, inserts: 0, updates: 0, unchanged: 16739, deletes: 0
-- (its a bit faster now because it doesn't update unchanged values)
EXIT TO MAIN MENU AND LOAD AutoSave_0001, PRESS "Continue your journey"
--prints TableLoad ran without error; checksum: 841967
print(gPlots[64][50].plotType) --prints 3
print(gPlots[64][50].featureType) --prints 0
print(gPlots[64][50].improvementType) --prints -1
Below is MyMod_Info after the session above. One line is added for each TableSave. When TableLoad runs, it reads the last line (to get checksum at last save) and writes info to the last 3 columns of this line (if called >1 time, it will keep reading and writing on this same line).
http://forums.civfanatics.com/attachment.php?attachmentid=303664&stc=1&d=1317942815
And MyMod_Data (1st page only). You don't need to understand this, but it may be interesting if you want to know the underlying logic.
http://forums.civfanatics.com/attachment.php?attachmentid=303663&stc=1&d=1317942815
Compatibility between mods
There should be no problem as long as you use a unique string (your mod name for example) in your calls to TableSave and TableLoad.
Sharing between mods
I have not tried this, so I'm explaining from a purely theoretical perspective. In theory, two mods should be able to "read" the same DB table using TableLoad without any problems. However, you can't have two mods modifying the same table. Each one keeps its own representation of the DB table in memory (this is part of TableSave, not TableLoad). This is so TableSave knows which values to update, and which are unchanged and do not need an update. All hell will break loose if two mods TableSave to the same table.
Does this replace SaveUtils and ShareUtils?
I don't know. I built this for my own needs, which do not include sharing between mods. Primarily, I need to have many tables that maintain content through save/load. I don't want to write a separate save function for each table, and I don't want to have to worry about making a structure that is optimal for DB save (which is not necessarily optimal for complex Lua logic). Also, my tables are somewhat large (though not 16000 elements thank goodness!) so I needed a good understanding of speed and how many table elements I can safely work with (I don't know the speed or capacity of SaveUtils ... it may or may not be better than TableSaverLoader).
Note for initial release: don't try to save/load two different global tables with 2 calls to TableSave, e.g., TableSave(gT1, "MyMod1") and TableSave(gT2, "MyMod2"). This is intended to work in the next version, but won't work now.
!!!!Warning: if you keep the DB open in SQLite Manager through a game exit/load cycle, bad things will happen!!!! [Edit: I'm not 100% sure but I think that this warning is no longer an issue after one of the patches.]
----------------------------------------------------------------------------------
How to Hook Up
You could potentially run TableSave every time something changes. But this will be very slow because each save involves at least one database transaction (http://www.sqlite.org/faq.html#q19), and these have to wait for your hard drive to rotate around at least twice (assuming you have a rotating HD). TableSave is fast because it packs up to 600 updates or 300 inserts into a single transaction. It is also smart enough to know what has changed and what has not, so only updates DB data that has actually changed. The best way to use this is to run TableSave immediately before each game save. Unfortunately, Firaxis did not provide us with a gamesave event. So you will have to do a little coding to do this. In the example below, all of the tables I want to save are nested in gT, which is a field in MapModData (I create a local version of gT in all files that use it). "Ea" is the name of my mod so I use that as a unique string identifier.
To intercept saves from the normal Game Save menu, I added this bit of code in SaveMenu.lua:
--Paz add (somewhere near the top)
MapModData.gT = MapModData.gT or {}
local gT = MapModData.gT
--end Paz add
...snip...
function OnSave()
--Paz add
LuaEvents.TableSaverLoaderSave(gT, "Ea")
--end Paz add
if(g_SelectedEntry == nil) then
...
Note that I added TableSave to LuaEvents in version 0.12, so you will need the new version for this to work. The bit of code at the top exposes my target table to this file (now pointed to by the local variable gT). The LuaEvents call will fire TableSave when the player clicks Save (even if the player clicks No on the following dialog box -- but it doesn't matter if we have an extra DB save).
To intercept quicksaves from either player clicking F11 or Quick Save from the Main menu, add this code:
function InputHandler( uiMsg, wParam, lParam )
if uiMsg == KeyEvents.KeyDown then
if wParam == Keys.VK_F11 then
TableSave(gT, "Ea")
print("Quicksaving...")
UI.QuickSave()
return true
end
end
end
function OnQuickSaveClicked()
print("QuickSaveGame clicked")
TableSave(gT, "Ea")
UI.QuickSave()
end
function OnEnterGame() --Runs when Begin or Countinue Your Journey pressed
ContextPtr:SetInputHandler(InputHandler)
local QuickSaveButton = ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton")
QuickSaveButton:RegisterCallback( Mouse.eLClick, OnQuickSaveClicked )
Events.LoadScreenClose.Add(OnEnterGame)
Then you have to worry about autosaves. This is rather tricky due to timing of the autosave. You really want to run TableSave as close as possible (but before) the autosave. This is the best I think you can do:
local BARB_PLAYER_ID = GameDefines.MAX_PLAYERS - 1
local function OnEveryPlayerTurn(iPlayer)
if iPlayer == BARB_PLAYER_ID then
TableSave(gT, "Ea")
end
end
GameEvents.PlayerDoTurn.Add(OnEveryPlayerTurn)
There is a flaw here. If anything changes during the barb turn it will be reflected in the autosave but not the SaveGameDB. But that's the best you can do as far as I've been able to determine. (Hooking it up to OnActivePlayerTurnStart or OnActivePlayerTurnEnd is very far from the autosave.)
You only ever need TableLoad when loading a game from a save file. I have this code in my init function:
local bNewGame = true
local DBQuery = Modding.OpenSaveData().Query
for row in DBQuery("SELECT name FROM sqlite_master WHERE name='Ea_Info'") do
if row.name then bNewGame = false end -- presence of Ea_Info tells us that game already in session
end
if bNewGame then
print("Initializiing for new game...")
else
print("Initializing for loaded game...")
TableLoad(gT, "Ea")
endThe code above figures out whether this is a new game or not by the presence of Ea_Info in SavedGameDB (this is one of two tables created by TableSave). If you used TableSave(gT, "MyMod"), then this file would be MyMod_Info. If this file is present, then this must be a loaded game. If file not present, then this must be a new game.
What not to do:
Don't try to include TableSaverLoader.lua from multiple Lua states. The file creates a Lua representation of your SavedGameDB data and uses this to know what parts to update and what parts are unchanged. If you include from two states, you are basically duplicating this info; if something changes in state #1, then it will no longer be accurate in state #2. (This is why I use a LuaEvents above to call from SaveMenu.lua.
Changes:
0.11 (Sep 23, 2012) fixed a bug in 0.10 that caused a crash when a variable changed type.
0.12 (May 1, 2013) adds TableSave to LuaEvents so it can be called from another state.
The idea behind TableSaverLoader is that you do all game logic in Lua tables, then reference those tables in a master table for "preservation" through game exit/reload. TableSaverLoader (hooked up properly as shown in attached "ExampleMain.lua" below) will save tables of any arbitrary size* and nesting structure to SavedGameDB, then restore them on game reload. There is no issue about when you save/exit because it intercepts all 4 gamesave types: cntl-S, save from menu, F11 quicksave and endturn for autosave. The code for the gamesave interception is actually in the ExampleMain.lua file attached below.
Download here: http://forums.civfanatics.com/downloads.php?do=file&id=17824 (See How to Hook Up section below.)
(*No limit to size as far as I know, though there are always time considerations. For 16739 table elements, TableSave takes 22 sec to save the first time and about 13 seconds thereafter when all or most elements don't change, and TableLoad 0.175 seconds to load this data. That's a bit of a wait at turn end ... hopefully you won't have 16739 things to save. This is on my 5-yr-old computer running a Standard map.)
TableSave(gT, "MyMod") will save gT and all nested values and tables to a single table called "MyMod_Data" in SavedGameDB. A second table called "MyMod_Info" keeps save/load info including a checksum used to test data integrity on load.
TableLoad(gT, "MyMod") will rebuild gT after a game load.
I've included two files. One is "TableSaverLoader" with the two functions to call. The second is "ExampleMain" that calls TableSave before any manual game saves (it intercepts quicksave, cntr-s and save from menu; this code taken from Gedemon) and at turn end (so that autosave is updated), and calls TableLoad after loading a game.
My ExampleMain sets up a global table ("gT") and "preloads" it with two other tables:
gValues = {}
gPlots = {}
gT = { gValues, gPlots } --my global table that holds everything to be saved
To illustrate, here's an interactive FireTuner session using TableSaverLoader and ExampleMain to drive it:
START A NEW GAME, PRESS "Begin Journey"
--prints TableSave time: 0.36800000000005, checksum: 99, inserts: 2, updates: 0, unchanged: 0, deletes: 0
print(gT) -- prints table: 25EDF9C0
print(gValues) -- prints table: 25EDF858
print(gT[1]) -- prints table: 25EDF858
END TURN
--prints TableSave time: 0.095000000000027, checksum: 99, inserts: 0, updates: 0, unchanged: 2, deletes: 0
a_table = {x=1,y=2,z=3}
a_nested_table = {'a',{'b',{a_table,{{{{{'hello there'}}}}}}}}
gValues.bBoolean = true
gValues.a_table = a_table
gValues.a_nested_table = a_nested_table
print(gValues.bBoolean) --prints true
print(gValues.a_nested_table[2][2][2][1][1][1][1][1]) --prints hello there
print(a_table) --prints table: 266B1EF0
print(gValues.a_table) --prints table: 266B1EF0
print(gValues.a_nested_table[2][2][1]) --prints table: 266B1EF0
print(gValues.a_table.y) --prints 2
print(gValues.a_nested_table[2][2][1].y) --prints 2
F11 (QUICKSAVE)
--prints TableSave time: 0.14399999999978, checksum: 1243, inserts: 17, updates: 0, unchanged: 2, deletes: 0
EXIT TO MAIN MENU AND LOAD QUICKSAVE, PRESS "Continue your journey"
--prints TableLoad ran without error; checksum: 1243
print(gValues.bBoolean) --prints true
print(gValues.a_nested_table[2][2][2][1][1][1][1][1]) --prints hello there
print(a_table) --prints nil
print(gValues.a_table) --prints table: 155CB858
print(gValues.a_nested_table[2][2][1]) --prints table: 155CB858
--note that a_table wasn't saved (it isn't in gT)
--however, two "paths" to the same table in gT point to the same table (this was a devil to make work!)
print(gValues.a_table.y) --prints 2
print(gValues.a_nested_table[2][2][1].y) --prints 2
DebugFillgPlots() --this function gets plotyType, featureType and improvementType for each plot, and puts them in gPlots
--prints found 4160 plots
F11 (QUICKSAVE)
--prints Getting DB info for TableSave (this is the first save after a game load)...
--prints TableSave time: 21.983, checksum: 841967, inserts: 16720, updates: 0, unchanged: 19, deletes: 0
--(MyMod_Data now has 16739 rows)
print(gPlots[64][50].plotType) --prints 3
print(gPlots[64][50].featureType) --prints 0
print(gPlots[64][50].improvementType) --prints -1
NEXT TURN
--prints TableSave time: 13.491, checksum: 841967, inserts: 0, updates: 0, unchanged: 16739, deletes: 0
-- (its a bit faster now because it doesn't update unchanged values)
EXIT TO MAIN MENU AND LOAD AutoSave_0001, PRESS "Continue your journey"
--prints TableLoad ran without error; checksum: 841967
print(gPlots[64][50].plotType) --prints 3
print(gPlots[64][50].featureType) --prints 0
print(gPlots[64][50].improvementType) --prints -1
Below is MyMod_Info after the session above. One line is added for each TableSave. When TableLoad runs, it reads the last line (to get checksum at last save) and writes info to the last 3 columns of this line (if called >1 time, it will keep reading and writing on this same line).
http://forums.civfanatics.com/attachment.php?attachmentid=303664&stc=1&d=1317942815
And MyMod_Data (1st page only). You don't need to understand this, but it may be interesting if you want to know the underlying logic.
http://forums.civfanatics.com/attachment.php?attachmentid=303663&stc=1&d=1317942815
Compatibility between mods
There should be no problem as long as you use a unique string (your mod name for example) in your calls to TableSave and TableLoad.
Sharing between mods
I have not tried this, so I'm explaining from a purely theoretical perspective. In theory, two mods should be able to "read" the same DB table using TableLoad without any problems. However, you can't have two mods modifying the same table. Each one keeps its own representation of the DB table in memory (this is part of TableSave, not TableLoad). This is so TableSave knows which values to update, and which are unchanged and do not need an update. All hell will break loose if two mods TableSave to the same table.
Does this replace SaveUtils and ShareUtils?
I don't know. I built this for my own needs, which do not include sharing between mods. Primarily, I need to have many tables that maintain content through save/load. I don't want to write a separate save function for each table, and I don't want to have to worry about making a structure that is optimal for DB save (which is not necessarily optimal for complex Lua logic). Also, my tables are somewhat large (though not 16000 elements thank goodness!) so I needed a good understanding of speed and how many table elements I can safely work with (I don't know the speed or capacity of SaveUtils ... it may or may not be better than TableSaverLoader).
Note for initial release: don't try to save/load two different global tables with 2 calls to TableSave, e.g., TableSave(gT1, "MyMod1") and TableSave(gT2, "MyMod2"). This is intended to work in the next version, but won't work now.
!!!!Warning: if you keep the DB open in SQLite Manager through a game exit/load cycle, bad things will happen!!!! [Edit: I'm not 100% sure but I think that this warning is no longer an issue after one of the patches.]
----------------------------------------------------------------------------------
How to Hook Up
You could potentially run TableSave every time something changes. But this will be very slow because each save involves at least one database transaction (http://www.sqlite.org/faq.html#q19), and these have to wait for your hard drive to rotate around at least twice (assuming you have a rotating HD). TableSave is fast because it packs up to 600 updates or 300 inserts into a single transaction. It is also smart enough to know what has changed and what has not, so only updates DB data that has actually changed. The best way to use this is to run TableSave immediately before each game save. Unfortunately, Firaxis did not provide us with a gamesave event. So you will have to do a little coding to do this. In the example below, all of the tables I want to save are nested in gT, which is a field in MapModData (I create a local version of gT in all files that use it). "Ea" is the name of my mod so I use that as a unique string identifier.
To intercept saves from the normal Game Save menu, I added this bit of code in SaveMenu.lua:
--Paz add (somewhere near the top)
MapModData.gT = MapModData.gT or {}
local gT = MapModData.gT
--end Paz add
...snip...
function OnSave()
--Paz add
LuaEvents.TableSaverLoaderSave(gT, "Ea")
--end Paz add
if(g_SelectedEntry == nil) then
...
Note that I added TableSave to LuaEvents in version 0.12, so you will need the new version for this to work. The bit of code at the top exposes my target table to this file (now pointed to by the local variable gT). The LuaEvents call will fire TableSave when the player clicks Save (even if the player clicks No on the following dialog box -- but it doesn't matter if we have an extra DB save).
To intercept quicksaves from either player clicking F11 or Quick Save from the Main menu, add this code:
function InputHandler( uiMsg, wParam, lParam )
if uiMsg == KeyEvents.KeyDown then
if wParam == Keys.VK_F11 then
TableSave(gT, "Ea")
print("Quicksaving...")
UI.QuickSave()
return true
end
end
end
function OnQuickSaveClicked()
print("QuickSaveGame clicked")
TableSave(gT, "Ea")
UI.QuickSave()
end
function OnEnterGame() --Runs when Begin or Countinue Your Journey pressed
ContextPtr:SetInputHandler(InputHandler)
local QuickSaveButton = ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton")
QuickSaveButton:RegisterCallback( Mouse.eLClick, OnQuickSaveClicked )
Events.LoadScreenClose.Add(OnEnterGame)
Then you have to worry about autosaves. This is rather tricky due to timing of the autosave. You really want to run TableSave as close as possible (but before) the autosave. This is the best I think you can do:
local BARB_PLAYER_ID = GameDefines.MAX_PLAYERS - 1
local function OnEveryPlayerTurn(iPlayer)
if iPlayer == BARB_PLAYER_ID then
TableSave(gT, "Ea")
end
end
GameEvents.PlayerDoTurn.Add(OnEveryPlayerTurn)
There is a flaw here. If anything changes during the barb turn it will be reflected in the autosave but not the SaveGameDB. But that's the best you can do as far as I've been able to determine. (Hooking it up to OnActivePlayerTurnStart or OnActivePlayerTurnEnd is very far from the autosave.)
You only ever need TableLoad when loading a game from a save file. I have this code in my init function:
local bNewGame = true
local DBQuery = Modding.OpenSaveData().Query
for row in DBQuery("SELECT name FROM sqlite_master WHERE name='Ea_Info'") do
if row.name then bNewGame = false end -- presence of Ea_Info tells us that game already in session
end
if bNewGame then
print("Initializiing for new game...")
else
print("Initializing for loaded game...")
TableLoad(gT, "Ea")
endThe code above figures out whether this is a new game or not by the presence of Ea_Info in SavedGameDB (this is one of two tables created by TableSave). If you used TableSave(gT, "MyMod"), then this file would be MyMod_Info. If this file is present, then this must be a loaded game. If file not present, then this must be a new game.
What not to do:
Don't try to include TableSaverLoader.lua from multiple Lua states. The file creates a Lua representation of your SavedGameDB data and uses this to know what parts to update and what parts are unchanged. If you include from two states, you are basically duplicating this info; if something changes in state #1, then it will no longer be accurate in state #2. (This is why I use a LuaEvents above to call from SaveMenu.lua.