TableSaverLoader, for persisting Lua table data through game save/load

[LeeS]

I myself am not real worried about the extra time it takes for the save. I'm only saving about 1500 total pieces of info worst-case, and so far I haven't been able to percieve a difference in how long a game save requires.

One other question, though, is will TableSaverLoader work successfully if a table is empty ? I use one of my tables to keep track of which great general and admiral names have already been used, but in the early part of the game there won't have been any. Right now as a temp expedient because I wasn't sure whether an empty table would cause saving or reloading issues I just stick a stringed value "1" into the first "v" of the table I use to keep track of the names that have been used. [edit]I'd like to get rid of this extra clutter if it isn't necessary.

 

[Pazyryk]

Empty tables are handled correctly by TableSaverLoader: they are saved and rebuilt as such.

It isn't even a bad idea from a Lua perspective. Lua tables are inited with no storage space, so 1000 empty tables has the same memory footprint as 1000 numbers, if I understand Lua's inner workings correctly. (For those interested: They expand to hold 1 element, then double as needed. But never shrink.)

Edit:

Here's an interesting case where TableSaverLoader won't give you back exactly what you gave it due to differing number precisions:

> x = 1000000000001 ; print(x == tonumber(tostring(x)), x)
true 1000000000001
> x = 10000000000001 ; print(x == tonumber(tostring(x)), x)
true 10000000000001
> x = 100000000000001 ; print(x == tonumber(tostring(x)), x)
false 1e+014
> x = 1000000000000001 ; print(x == tonumber(tostring(x)), x)
false 1e+015
> x = 10000000000000001 ; print(x == tonumber(tostring(x)), x)
true 1e+016
> x = 100000000000000001 ; print(x == tonumber(tostring(x)), x)
true 1e+017

The reason the middle two give false is that Lua numbers are higher precision than the tostring method is able to handle (which is probably limited by some C++ variable type). The print statement itself implements tostring, so the result of that is what is printed. But the print is lying to you about the actual value of x. In the last two, I've exceeded the precision of Lua itself, so the round off becomes the same again.

This affects TableSaverLoader because it uses tostring to serialize numbers, and then tonumber to rebuild them. But I doubt that any modders need this precision so it doesn't really matter.

 

[Pazyryk]

Version 0.16 is released!

Even faster and it has a smaller Lua memory footprint than previous versions. It also fixes the apostrophe (') bug in string values - use as many as you like now!

Note however that I've added apostrophe to the list of illegal characters for string keys: #, $, ', international characters (more specifically, UTF-8 characters represented by >1 byte), and the empty string ("") itself. It would be possible to mod TableSave and TableLoad to allow each of these cases if someone really wanted to do that, at the cost of a tiny amount of overhead. But it didn't seem worth it to me for key values. TableSaverLoader is already more restrictive than Lua for keys anyway, since in Lua you are allowed to use anything other than nil as a key (even tables or functions).

I've also updated instructions in OP for catching game saves. For dll modders, I've posted the code I'm now using that adds a GameEvents.GameSave where the entire code block is skipped in the first autosave. There is an intermittent, system-dependent game hang that happens if you add a GameEvents hook at this initial save. It has nothing at all to do with TableSave itself! The hang comes from the the gDLL->GetScriptSystem() instruction that is part of setting up any GameEvents. All subsequent saves seem to be immune to this problem.

I also changed the OP instructions for Lua-only intercept of game saves to avoid using Events.LoadScreenClose to fire TableSave(). I don't know that there is a problem with this, but I'm very leery of that first autosave after my experience with GameEvents modding in dll above. It just seems like a bad time to fire off some DB I/O operation, with the possibility of inconsistent and system-specific problems. I think it is safer to run the first TableSave() at the end of your mod load.


Note on backward compatibility: I think 0.16 will load tables saved from 0.13/0.14. However, the checksum will be wrong. So to avoid a load error you would have to set one of the two "User Settings" bChecksum or bAssertError to false. It may be best to update to 0.16 when your mod is going through a "breaks saves" version update.

 

[DarkScythe]

Thanks for the update, Pazyryk.

I am looking at figuring out how to update my hookup code to work with your new "TableLoad returns boolean" function, but am slightly confused.

TableLoad() was previously used as a way of loading the data from the database back into Lua tables, or something along those lines. However, in your example code, you have:

local bNewGame = not TableLoad(gT, "MyMod")

If this returns a boolean (true or false), then how do you additionally go about loading the contents?

Looking at the updated code in version 0.16, am I correct in my interpretation that, within that bNewGame declaration which calls this function, if no data exists, it must be a new game, returns false, and I can then simply do something like:

if bNewGame then
     print("New game!")
end

However, if it is not a new game, and it finds existing data, it will return true, but, at the same time, has already loaded all the data, thus not requiring a separate 'else' clause in the above to do a TableLoad() which existed in previous versions of the hookup code?

Is that correct?

 

[Pazyryk]

Your last conjecture is correct.

It's is a very common coding convention to invoke a method to do something for you while giving some feedback to use in a conditional test. You could also do something like:

if TableLoad(gT, "MyMod") then
     print("Loaded game!")
     --do stuff
else
     print("New game!")
     --do stuff
end

Maybe I should use that in my OP for the benefit of new coders...

 

[DarkScythe]

Ah ha, okay, thanks.

That could potentially simplify a few things, though I'm still thinking about when to call that first TableSave().

You wanted to move it out of LoadScreenClose because of the hang issues, but I'm not sure if there's any discernible difference between calling it before and after the whole Lua file has loaded.

 

[Pazyryk]

You could call TableSave() before you've loaded all your mod. It just has to be after you change any data that is supposed to be persisted and before that first autosave (that runs immediately following Begin Your Journey). There's actually no need to do it after TableLoad(), unless (of course) you change something between that and the first autosave. Keep in mind that the game autosaves over an existing autosave when you load that autosave (that's very annoying and I don't see any reason for it, but that's what it does).

The very first TableSave() has more overhead than subsequent calls to it if you (for example) set up 1000 data points to be persisted in your init code. These are all db INSERTs on the first call (for a new game), whereas on subsequent TableSave() operations it is only the changed data that does anything to db. It should be very fast but db operations are weird, especially with a rotating hard drive. The hard drive has to rotate around twice to validate the db transaction before it will release the hard drive for other stuff. So it's the kind of thing that could be platform specific.

But I have never observed a "permanent/game-breaking hang" from using Events.LoadScreenClose. I'm just conjecturing a possible one based on my experience modding in a GameEvents that happened to fire at almost exactly the same time (which did hang the game in a very erratic and platform-specific way, irrespective of what was hooked to that GameEvents).

What I have observed though with LoadScreenClose (when I had a lot of code on it) was a noticeable "temp hang" after pressing the Begin your Journey button. Not a game-breaking hang. But still an aesthetically unpleasing experience for the player pressing that button. So I've moved all that code into my "mod loading" code so it is completely done already before the player presses that button. (The only thing I do on that event are some context code that depends on everything being loaded including UI Lua.)

 

[DarkScythe]

Hrm, okay..

So then, I might be okay with leaving the TableSave event hooked in to LoadScreenClosed for the time being, since I'm not using any DLL mods. However, in the event that it does hang under some weird circumstance, I should be able to move it elsewhere in the initialization phase. For the sake of cleanliness, I've moved the entire hookup code into its own file, so I'd like as much as possible to keep everything in one place (less headache down the line when I need to edit this stuff after not looking at it for a while.)

In that case, I suppose I can call TableSave() if TableLoad() returns false (meaning a new game.)
It's workable in my case, since I don't modify or load any new values until something actually happens, and this file is called after I've defined the main tables and their hierarchy.

In essence:

include("TableSaverLoader.lua")
MapModData.gT = MapModData.gT or {}
gT = MapModData.gT
gT.Table1 = {}
gT.Table2 = {}
gT.Table3 = {}
include("TableSaverLoaderInit.lua")

I'm sure if I ever wanted to have other functions verify/alter/update stuff in those tables during the course of my script's loading process, I'd stick a TableSave() manually there at the end as well.

 

[Pazyryk]

Yeah, that works.

Even if you do start adding init values to be persisted, you could just add them after gT.Table3 = {}. TableLoad(), if it runs, would simply overwrite the non-table values and add values to any nested tables if they happen to already exist with the correct key names (e.g., if you had gT.Table1 = {dogs = {}, cats = {}}, then TableLoad() will populate those tables rather than creating new dogs/cats tables).

 

[DarkScythe]

Excellent, thanks for the assistance again!

I'll work out the code I need to make all this happen and go test. I'll report back if I run into any issues.

 

[bane_]

Even thought the name says 'TableSaverLoader', I imagine that I can somehow store make variables persist through save/loads with this, right?

I'm still a bit confused on how to do this, is there any example file I can follow? Might be easier for me to understand.

 

[Pazyryk]

Yes, that's exactly what it's for. But the values you want to persist have to be in tables.

Then make these tables persistent by:

gT = {}
gT.table1 = table1
gT.table2 = table2
etc.

Then you should be able to copy paste code from the "How to Hook Up" section, mostly. You'll have to edit the "MyMod" string in the TableLoad and TableSave function calls.

 

[bane_]

The master table itself (gT) can contain non-table items if you want.

There is exactly ONE variable I need persisting through games, it is iNumFantasticGardensAI, that is increased each time the AI builds a certain improvement.

Spoiler :

GameEvents.PlayerBuilt.Add(
function(iPlayer, iUnit, iX, iY, iBuild) 
	if iBuild == GameInfoTypes["BUILD_FANTASTIC_GARDENS"] then
		iNumFantasticGardensAI = (iNumFantasticGardensAI or 0) + 1
	end
end)

So, if I'm understanding this right, all I need to do is to put "gT.table1 = iNumFantasticGardensAI" somewhere?

 

[Pazyryk]

No, it can only persist a value in a table. In your line above, you are just setting gT.table1 to 10 or 11 or whatever the value was when you run that line. iNumFantasticGardensAI won't be correct when the mod loads because it is just a standalone variable.

You need to modify a table value, not a standalone variable. Like this:

gT = {}
gT.iNumFantasticGardensAI = 0

GameEvents.PlayerBuilt.Add(
function(iPlayer, iUnit, iX, iY, iBuild) 
	if iBuild == GameInfoTypes["BUILD_FANTASTIC_GARDENS"] then
		gT.iNumFantasticGardensAI = gT.iNumFantasticGardensAI + 1
	end
end)

So now you are persisting a table (gT) and it contains the value you want in a key called iNumFantasticGardensAI.

 

[bane_]

Ohhh, ok. I believe I got it now.
Thank you.

 

[bane_]

Apparently I didn't.

I'm getting this error:

[30711.745] Runtime Error: C:\Users\bane_\Documents\My Games\Sid Meier's Civilization 5\MODS\Crane Clan (v 1)\Lua/Lib/CraneTableSaverLoader.lua:3: attempt to call global 'TableLoad' (a nil value)

Here is the code:

function OnModLoaded()

	local bNewGame = not TableLoad(gT, "L5RCraneClan")

	-- < mod-specific init code here depending on bNewGame >

	TableSave(gT, "L5RCraneClan")	--I talk about this more below
end

What should I put in the commented line, anyway?

Sorry for making so many questions.

 

[DarkScythe]

If there is anything specific you want to run if it is a new game, then you put that code in there, otherwise you can most likely get rid of it.

That section is only to allow you to call TableLoad() via OnModLoaded() once your main Lua file finishes loading.

If TableLoad() is returning nil, you probably don't have the TableSaverLoader.lua included()'d beforehand? (And set VFS=true on it?)

 

[Pazyryk]

You can always see if a function exists in your mod state by typing:

print(FunctionName)

in Fire Tuner. (Note that there is no () after the function!) If that gives you nil, then the code hasn't loaded for some reason. You can also see if code has loaded by putting print statements in the code itself. I always do that so I'm pretty sure there is something like "Loading TableSaverLoader..." at the top already.

 

[bane_]

I had forgotten to put the include.
I'm having errors still (hey, at least these are new):

[46529.482] CraneTableSaverLoader: Loading TableSaverLoader.lua...
[46529.482] CraneTableSaverLoader: Creating SavedGameDB tables for new game: L5RCraneClan_Data and L5RCraneClan_Info
[46529.497] Runtime Error: C:\Users\bane_\Documents\My Games\Sid Meier's Civilization 5\MODS\Crane Clan (v 1)\Lua\Lib\TableSaverLoader.lua:377: bad argument #1 to 'pairs' (table expected, got nil)
[46529.497] Runtime Error: Error loading C:\Users\bane_\Documents\My Games\Sid Meier's Civilization 5\MODS\Crane Clan (v 1)\Lua/Lib/CraneTableSaverLoader.lua.

Here's what I have in CraneTableSaverLoader.lua (the "Error loading" file):

Spoiler :

include("TableSaverLoader.lua")

function OnModLoaded()

	local bNewGame = not TableLoad(gT, "L5RCraneClan")

	-- < mod-specific init code here depending on bNewGame >

	TableSave(gT, "L5RCraneClan")	--I talk about this more below
end

function OnEnterGame()   --Runs when Begin or Continue Your Journey L5RCraneClan
	print("Player entering game ...")
	ContextPtr:LookUpControl("/InGame/GameMenu/SaveGameButton"):RegisterCallback(Mouse.eLClick, SaveGameIntercept)
	ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton"):RegisterCallback(Mouse.eLClick, QuickSaveIntercept)
end
Events.LoadScreenClose.Add(OnEnterGame)

function SaveGameIntercept()	--overrides Civ5 code when player presses Save Game from Game Menu or Cntr-s
	TableSave(gT, "L5RCraneClan")
	UIManager:QueuePopup(ContextPtr:LookUpControl("/InGame/GameMenu/SaveMenu"), PopupPriority.SaveMenu)
end

function QuickSaveIntercept()	--overrides Civ5 code when player presses Quick Save from Game Menu or F11
	TableSave(gT, "L5RCraneClan")
	UI.QuickSave()
end

local autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
function OnGameOptionsChanged()
	autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
end
Events.GameOptionsChanged.Add(OnGameOptionsChanged)

function OnAIProcessingEndedForPlayer(iPlayer)
	if iPlayer == 63 then					--runs on barb turn AFTER barb unit moves (very close to the regular autosave)
		if Game.GetGameTurn() % autoSaveFreq == 0 then	--only need to do on autosave turns
			TableSave(gT, "L5RCraneClan")
		end
	end
end
Events.AIProcessingEndedForPlayer.Add(OnAIProcessingEndedForPlayer)

function InputHandler(uiMsg, wParam, lParam)
	if uiMsg == KeyEvents.KeyDown then
		if wParam == Keys.VK_F11 then
			QuickSaveIntercept()		--F11 Quicksave
        		return true
		elseif wParam == Keys.S and UIManager:GetControl() then
			SaveGameIntercept()			--ctrl-s
			return true
		end
	end
end
ContextPtr:SetInputHandler(InputHandler)


OnModLoaded()

 

[Pazyryk]

It doesn't recognize the first arg as a table. To make gT into a table, do this:

gT = {}

Then you can put things in that table and they will be persisted.