TableSaverLoader, for persisting Lua table data through game save/load

[DarkScythe]

Thanks for the update!

And I didn't really contribute all that much, but thanks for the shout-out!

Anyway, I'm looking over and comparing your current "finalized" hookup code to previous iterations.

Is there a reason you pulled ContextPtr:SetInputHandler(InputHandler) out of the OnEnterGame() function where you define the other ContextPtr's? (Can a player even save before clicking through that loading screen?)

Otherwise, I just learned it wasn't a great idea to tie the first TableLoad() to LoadScreenClose; Breaks loading from an autosave, apparently. Thinking I can probably just wrap your entire init and hookup functions into a do() block, instead..

 

[Pazyryk]

ContextPtr by itself refers to whatever context the code is running from. So you can always run ContextPtr:SetInputHandler(InputHandler) right after InputHandler loads in any context. That's how you see it in all the base game UI Lua files.

However, ContextPtr:LookUpControl(< something >) refers to some external context or some Control element in that context, like the SaveGameButton in GameMenu UI context. Maybe that loads before your mod, but maybe after (if after, then it is undefined when your mod loads). So it's easier to assume neither and run it when you can be sure that all contexts have loaded, when the player enters the game.

 

[DarkScythe]

Hum.. I guess I have a ways to go.. I don't quite understand what the various Contexts mean. (Well, I kind of do understand the concept, but unable to really apply it to Civ, since I don't know what its various contexts are.)

That said, I modified the code slightly to fit into two blocks at the top of my file (I'm considering just moving it off entirely into another include() file at the top of my main Lua script.)

do
	local bNewGame = true
	local tableName = "TestTable"
	local DBQuery = Modding.OpenSaveData().Query
	local autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
	for row in DBQuery("SELECT name FROM sqlite_master WHERE name='" .. tableName .. "_Info'") do
		if row.name then
			bNewGame = false
			print("TableSaverLoader: Initializing to continue a loaded game...")
			TableLoad(gT, tableName)
			break
		end
	end

	if bNewGame then
		print("TableSaverLoader: Initializing for new game...")
		TableSave(gT, tableName)
	end

	function OnGameOptionsChanged()
		autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
	end

	Events.GameOptionsChanged.Add(OnGameOptionsChanged)

	function OnAIProcessingEndedForPlayer(iPlayer)
		if iPlayer == 63 then
			if Game.GetGameTurn() % autoSaveFreq == 0 then
				TableSave(gT, tableName)
			end
		end
	end

	Events.AIProcessingEndedForPlayer.Add(OnAIProcessingEndedForPlayer)
end

function TableSaverLoaderInit()
	print("Player entering game ...")
	local tableName = "TestTable"
	local function SaveGameIntercept()
		TableSave(gT, tableName)
		UIManager:QueuePopup(ContextPtr:LookUpControl("/InGame/GameMenu/SaveMenu"), PopupPriority.SaveMenu)
	end
	local function QuickSaveIntercept()
		TableSave(gT, tableName)
		UI.QuickSave()
	end
	local function InputHandler(uiMsg, wParam, lParam)
		if uiMsg == KeyEvents.KeyDown then
			if wParam == Keys.VK_F11 then
				QuickSaveIntercept()
	        	return true
			elseif wParam == Keys.S and UIManager:GetControl() then
				SaveGameIntercept()
				return true
			end
		end
	end
	ContextPtr:SetInputHandler(InputHandler)
	ContextPtr:LookUpControl("/InGame/GameMenu/SaveGameButton"):RegisterCallback(Mouse.eLClick, SaveGameIntercept)
	ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton"):RegisterCallback(Mouse.eLClick, QuickSaveIntercept)
	TableSave(gT, tableName)
end

Is there any error in the way this is structured?
I believe everything worked properly even if I placed the last function inside the do() block, but I only wanted to hook into LoadScreenClose event with one primary function, and I don't think it'd be able to see inside the block if I left it in there.

 

[Pazyryk]

I think it's OK and would work as coded. Just some comments:

1. gT has to be defined before this code (and visible in both scopes).
2. You have TableSave running twice in a new game, once at file load and once whenever you fire TableSaverLoaderInit
3. The do-end "container" doesn't serve any purpose here. OK, it isolates the 4 locals to that block, but so what? They would be isolated to the file anyway. Is it just to help you organize code?
4. You've squirreled away your mod's InputHandler in a specific function named TableSaverLoaderInit. There's nothing wrong with this operationally, but what if you want to do something with input not related to TableSaverLoader? For example, you decide to add the ability to press Return to go on to the next turn? Or some key stroke to bring up a mod-specific UI screen? It may be easier for you or someone else to find if it is front and center in your main mod code. (If you forget and use SetInputHandler somewhere else in this context, then the new function will replace the one above and cntr-S and F11 will suddenly stop working.)
5. I'm not sure why you localized the 3 functions in the function block but not the 2 in the do block. You could localize all of them or none of them as it is currently organized. Since they are running once it doesn't save you any time to localize. The only other reason to localize it to isolate some identifier name that you might want to use somewhere else. OK, I localize more than I need to by these criteria, but I'm just wondering if you saw some reason that doesn't exist. Of course, if you followed my advice in #4 an put InputHander somewhere more prominent, then you would have to make sure that both SaveGameIntercept and QuickSaveIntercept were visible. Since you are unlikely to want to use those particular names again, there is no reason they shouldn't be using global identifiers rather than local.

, and I don't think it'd be able to see inside the block if I left it in there.

It's not a question of "seeing into" anything. The only question is whether all the identifiers are properly defined when and where the code line runs. So if you add

Events.LoadScreenClose.Add(TableSaverLoaderInit)

in the do block, then it won't work because TableSaverLoaderInit is not defined yet. The do block runs as the file loads, but TableSaverLoaderInit is only defined after the "end" in its definition. In fact, the most sensible place to put that line of code (what you will see in other modder's and Firaxis code) is right after TableSaverLoaderInit's "end". That way you know that the function is defined and you can easily see what different Events, GameEvents or LuaEvents it is subscribed to (possibly >1 of these).

 

[DarkScythe]

Haha, I am still quite the novice coder, especially to Lua, so I'm bound to make a bunch of mistakes here and there.

To answer your points:

1) gT is defined beforehand (I only extracted the specific sections of code related to hooking up TSL)

2) This might be an artifact of your "previous" code, wherein you stuck a save upon "bNewGame" in order to catch the initial autosave. I noticed this, and will remedy it.

3) Yes, the do-end container is largely to help me organize the code. In shorter files (such as my Loader file) I have no problem omitting it and just running the code flat as the file loads. However, since this TSL hookup code is placed right inside my Trait Lua, I didn't want any potential issues from your code creeping outside of TSL-specific functions, so I wanted to encapsulate them. (Not saying your code might have problems, but my own code is a mess.)

4) I'm not incredibly clear on the InputHandler section, so I didn't quite know its significance. All I wanted to do was to keep the OnEnterGame() function (which I renamed to TableSaverLoaderInit()) in one piece, but it referenced other functions (SaveGameIntercept, etc.) so I figured I'd throw them together.

5) I'm not sure why I do things the way I do either, to be honest.. The pattern I kind of developed, and was following, is that functions that are called by other functions are localized, and the "non localized" functions are the ones that get added to the various game event hooks. It probably doesn't matter either way, but it kind of helps me figure out at a glance what type of function it is.

As for adding TableSaverLoaderInit into the LoadScreenClose event inside the do() block, that wasn't what I was doing.

I mentioned a couple posts back that I had all the functions that I hooked into LoadScreenClose into one primary function:

function GameInit()
	TableSaverLoaderInit()
	debugModeCheck()
	debugModList()
	debugPlayerList()
	GameStateInit()
end

Events.LoadScreenClose.Add(GameInit)

This was the reason why I was trying to extract your OnEnterGame() function.

If I'm supposed to be doing things differently, please let me know. I'm sort of guessing half the time, haha.

 

[Pazyryk]

Well, you're guessing right if it is only guesses. All of my complaints were cosmetic.

If you look at the base UI Lua files, you'll see an InputHandler function in each that is set with ContextPtr:SetInputHandler(InputHandler). Each of these UI elements (i.e., the paired lua/xml files) is added as a different context. Most of the UI files intercept Esc and/or Return to exit -- i.e., hide the context, or to do whatever else is appropriate for that UI. Popups don't exactly "go away", they just become hidden with ContextPtr:SetHide(true) (and their InputHandler stops working).

But everything I said above applies to your main mod code. It is most likely added as an InGameUIAddin, or possibly by modding InGame.xml. Either way, it is its own context and can have its own InputHander (but only one at a time). So if your main mod has one, it probably should be out in the open just for organizational considerations rather than tucked away where you will forget about it.

 

[DarkScythe]

True, they may be cosmetic, but I do want to do things properly, so if you're insisting that it's best to pull the InputHandlers out, I'll do that. Perhaps having 2 calls to LoadScreenClose event isn't too big a deal, I just don't want to get into the habit of hooking things from everywhere, as it's better for me to stay organized and have one central place where everything hooks in.

As for my mod code, my Loader file is added as an InGameUIAddin, and if my Civ is an active player in the game, it proceeds to include() my actual Lua script file which contains the TSL code along with everything else.

I still have a fairly strong desire to try and partition this stuff, but I don't know if moving the TSL code into an include() file would make any real difference, so for the moment I'll just leave everything in one gigantic, 1000+ line file.

 

[Pazyryk]

Including a file is pretty much exactly like putting code in a do block. It isolates the local identifiers to that file, but still allows sharing global identifiers. Which is very nice for organizing a mod.

There was a mistake in 0.13 (or call it a "limitation" to be generous). It would not reconstruct an empty table. So if a table index held an empty table on TableSave, it would be nil after TableLoad. I'm uploading 0.14 shortly which will fix that.

 

[Pazyryk]

Version 0.14 released!

Fixed error in 0.13 where it would not save an empty table (so that index would be nil on loading)

Note: It kind of slipped onto the last page, but I suggest all TableSaverLoader users re-read the OP and particularly the How To Hook Up sections. I rewrote it with 0.13 release. It should be easier for new users to figure out too. 0.13 itself provided a huge performance boost (but had this little error in it) so it is worth upgrading to 0.14 if you are using an older version.

Note2: 0.14 should be able to load a table saved with 0.13, but will see a checksum error. If you have bAssertError = true (in USER SETTINGS at the top of the file) then that is fatal - i.e., a game save from 0.13 is not compatible with 0.14. But you can change that to bAssertError = false to allow compatibility if you really need to (will still print checksum error, but can be ignored this time). [Never mind, should work just fine] 0.13/0.14 are both incompatible with earlier version saves.

 

[DarkScythe]

That sounds excellent, thanks for the tip.

If that's the case, I think I'll try to move stuff out into separate files (especially my own functions.. that script is scary long.)

I think after 2 months of development, I'm finally close to releasing this damn thing..

 

[Pazyryk]

I found a limitation for number representation. TableSaverLoader will crash if you try to persist 1.#INF or -1.#INF.

These are valid numbers in Lua, but Lua's own functions don't fully support them:

> 1/0
1.#INF
> -1/0
-1.#INF
> tostring(-1/0)
-1.#INF
> tonumber(tostring(-1/0))
nil

I'll post an update that fixes this in the next week or so. If it's urgent for you, let me know and I can do it sooner...

-------------------------------------------
Edit: Well, the fix is easy. Find this:

value = tonumber(valueString)	--numbers have no prefix byte
if not value then
	error("TableSaverLoader did not understand valueString from DB; row = " .. row.ID .. "; valueString = " .. tostring(valueString))
end

...and replace with this:

value = tonumber(valueString)	--numbers have no prefix byte
if not value then
	if valueString == "1.#INF" then
		value = 1/0	--yes, these are valid "numbers" in Lua
	elseif valueString == "-1.#INF" then
		value = -1/0
	elseif valueString == "-1.#IND" then
		value = 0/0
	else
		error("TableSaverLoader did not understand valueString from DB; row = " .. row.ID .. "; valueString = " .. tostring(valueString))
	end
end

I feel a little strange intentionally adding /0 into code, but so be it.


Edit2: Well, there also is -1.#IND (not to be confused with -1.#INF):

> 0/0
-1.#IND
> tostring(0/0)
-1.#IND
> tonumber(tostring(0/0))
nil

So I added this to code block above.

I'll release an updated 0.15 with this fix eventually. I have some memory optimization for the next version anyway...

 

[Pazyryk]

For anyone who uses the dll mod method to intercept game saves described in the OP, there's a problem that we've discovered in Éa. For reasons that are mysterious to me, the GameEvents code block sometimes (it's very inconsistent) causes a hang on the very first game save that happens when the player presses Begin Your Journey. It happens even if no Lua is attached to that GameEvents. I've established that it is this statement, gDLL->GetScriptSystem(), that causes the hang.

But it's only on the first save. Even players who experienced the hang almost 100% of the time on that first save never see it for subsequent saves. So the workaround is simply to bypass this code block for the first save only. You don't need the GameEvents to tell you it's going to happen when player presses Begin Your Journey, because you know that anyway.

It's a bit more complicated on the dll side now. I'll post the code to do it soon. [Someone post here to remind me if I forget...]

 

[S3rgeus]

For anyone who uses the dll mod method to intercept game saves described in the OP, there's a problem that we've discovered in Éa. For reasons that are mysterious to me, the GameEvents code block sometimes (it's very inconsistent) causes a hang on the very first game save that happens when the player presses Begin Your Journey. It happens even if no Lua is attached to that GameEvents. I've established that it is this statement, gDLL->GetScriptSystem(), that causes the hang.

But it's only on the first save. Even players who experienced the hang almost 100% of the time on that first save never see it for subsequent saves. So the workaround is simply to bypass this code block for the first save only. You don't need the GameEvents to tell you it's going to happen when player presses Begin Your Journey, because you know that anyway.

It's a bit more complicated on the dll side now. I'll post the code to do it soon. [Someone post here to remind me if I forget...]

I haven't read your updated OP yet, but I thought I'd chime in that I've encountered this hang issue. If you attach the VS debugger to CiV and open up the threads window, you'll find that there are two threads trying to grab a single lock/mutex (don't remember what they used - this was a few months ago) and it's resulted in a deadlock.

I'm not sure how to fix this exactly, the call stacks both went off into external code we don't have the source for quite quickly.

 

[Pazyryk]

Several months ago? But I hadn't posted any dll code back then. All I had were Lua-only methods for intercepting before game saves. Unless maybe you modded your own GameEvents.GameSave into dll?

But your explanation fits what we know. I'm quite sure now that the hang we've had lately in Éa is due to gDLL->GetScriptSystem() right at that specific time - i.e., right after player presses Begin Your Journey. It doesn't even matter if there is any Lua hooked up to the event. So it has nothing to do with TableSaverLoader per se. It's just a dangerous time to run that line in dll for some reason.

Or maybe you had TableSave() hooked up to Events.LoadScreenClose? This event fires right before the first autosave (perhaps it even triggers it in gDLL), which is exactly the criteria you want in theory, but perhaps this is a dangerous time for doing a lot of things. I isolated the problem above to running gDLL->GetScriptSystem() in dll at that time (even without calling TableSave). But it could be a bad time to write to DB as well. If so, I need to know that because that's how I have it currently shown in the OP for Lua-only hookup.

But, we haven't seen the problem at any other time. And I wouldn't want to consider gDLL->GetScriptSystem() to be generally dangerous, since it is used for all GameEvents. We've only had a tiny bit of testing now with new dll, but it seems like simply skipping the first save (so that gDLL->GetScriptSystem() doesn't run) gets us around the problem. You can run TableSave in init code somewhere before Events.LoadScreenClose, like right after all your mod code loads. No hang with that so far.

 

[hazel16]

Hi Pazyryk, thanks for posting this. Some of the users of my mod were having problems that I think were associated with SaveUtils so decided to switch to TableSaverLoader.

Unfortunately, I'm having trouble getting it to work. I have your lua save game intercepts set up.

Here is my code (g_ tables are defined previously):

function SaveData()
	--nest persistent tables in g_ML for saving purposes
	local g_ML = {g_Labels=g_Labels, 
			g_Options=g_Options, 
			g_UnitsKilled=g_UnitsKilled, 
			g_Cities=g_Cities, 
			g_IconsRecent=g_IconsRecent}
	TableSave(g_ML, "ML_ed1c8ceb81d3")
end

function LoadData()
	[COLOR="red"]TableLoad(g_ML,  "ML_ed1c8ceb81d3")[/COLOR]
	g_Labels=g_ML.g_Labels
	g_Options=g_ML.g_Options
	g_UnitsKilled=g_ML.g_UnitsKilled
	g_Cities=g_ML.g_Cities
	g_IconsRecent=g_ML.g_IconsRecent
end

function IsSaveGame()
	local bSaveGame = false
	local DBQuery = Modding.OpenSaveData().Query
	for row in DBQuery("SELECT name FROM sqlite_master WHERE name='ML_ed1c8ceb81d3_Info'") do
		bSaveGame = true	-- presence of 'ML_ed1c8ceb81d3_Info' tells us that game already in session
	end
	return bSaveGame
end

It appears to save without problem:

ML_Main: TableSave time: 0.057999999999993, checksum: 34571, inserts: 92, updates: 2, unchanged: 450, deletes: 0

But I get this error upon loading a game, and none of the tables populate:

ML_Main: +++++++++++++++MapLabelsMod: ML_Main.lua loaded
 ML_Main: Loading TableSaverLoader.lua...
 ML_Main: +++++++++++++++MapLabelsMod: ML_Labels.lua loaded
 ML_Main: +++++++++++++++MapLabelsMod: ML_Popup.lua loaded
 ML_Main: +++++++++++++++MapLabelsMod: ML_DynamicLabelsGenerator.lua loaded
 Runtime Error: C:\Users\...\Map Labels (v 4)\Lua\TableSaverLoader.lua:272: attempt to index local 'currentTable' (a nil value)
stack traceback:
	C:\Users\...\MODS\Map Labels (v 4)\Lua\TableSaverLoader.lua:272: in function 'TableLoad'
	C:\Users\...\Map Labels (v 4)\UI/ML_Main.lua:141: in function 'LoadData'
	C:\Users\...\Map Labels (v 4)\UI\ML_Labels.lua:409: in function 'InitLabels'
	C:\Users\...\Map Labels (v 4)\UI/ML_Main.lua:893: in main chunk
	=[C]: ?
 Runtime Error: Error loading C:\Users\...\Map Labels (v 4)\UI/ML_Main.lua.

My IsSaveGame() function appears to work just fine. The error occurs at LoadData()

	if (IsSaveGame()) then -- SAVED GAME
		dprint("Save Game")
		[COLOR="Red"]LoadData()[/COLOR]
		...

Appreciate your insight!

 

[hazel16]

Figured out what the problem was!

I was calling TableLoad with g_ML in LoadData() without having defined it (except locally in SaveData()). I added local g_ML = { } to LoadData() and it works!

Thanks again for providing this.

 

[LeeS]

I found a problem wherein if any of the string data within a saved table contains an apostrophe character ( ' ), TableSaverLoader is not able to handle such cases. The SaveGame and QuickSaveGame pushbuttons become locked while TableSaverLoader gets stuck in an error.

I was trying to save the In-Game names of Great Generals as players see them, and some of these (as well as some that my mod adds) have ' characters. Joan D'Arc for example. I made a test mod to confirm my suspicions, and got the same behavior when I allowed one table to have string data in a form of "EMPT'_TABLE" for the v in a k,v pair. In case I'm not explaining very well I've zipped the test mod and the lua error log into an attachment.

Changing that one occurance of "EMPT'_TABLE" to "EMPTY_TABLE" cures the problem.

 

[Pazyryk]

@LeeS,

Thanks for catching the apostrophe problem. I hadn't thought of this before although I should have. Apostrophe is used in sql to form strings when writing to DB, so it's quite obvious that this would crash the current code.

I don't want this restriction, so I'll update with a fix for this as fast as I can - which might be a day or two.

 

[LeeS]

OK. Good. Though I did redraft my lua to use TXT_KEYs within the saved tables, I still have the previous version archived which is a little cleaner code-bloat wise.

 

[Pazyryk]

More fun with Lua infinities and NaN (Not a Number), which will be supported in next version:

>x = 1/0
>print(x)
1.#INF
>print(x == x + 1)
true

>y = 0/0
>print(y)
-1.#IND
>print(y == y)
false

The effect of the last one is that TableSaverLoader will always see NaN as "changed" every time you use TableSave and update the DB for it (rather than skipping it as it does for any other unchanged value). The only effect is that if you (for some strange reason) maintain many 1000s of NaN's in your persisted data then you will have a bit of a slow down, although it will still be saved and loaded accurately.

Not as important as apostrophe support, but more interesting in a nerdy way.


Edit:
In the next version TableLoad will return true if it found existing data in the DB (created by TableSave) and false otherwise (exiting without error). So you will be able to use TableLoad now as a "new versus loaded game test" if you like. In other words:

this (in next version)...

local bNewGame = not TableLoad(gT, "MyMod")

is exactly equivalent to this existing code in OP...

local bNewGame = true
local DBQuery = Modding.OpenSaveData().Query
for row in DBQuery("SELECT name FROM sqlite_master WHERE name='MyMod_Info'") do
	bNewGame = false	-- presence of MyMod_Info tells us that game already in session
	break
end
if not bNewGame then
	TableLoad(gT, "MyMod")
end

But no need to change anything if you don't want to.