TableSaverLoader, for persisting Lua table data through game save/load

@FramedArchitect
Fortunately, there is a table called MapModData this is a "super global". It's seen by all Lua states. So in my mod I have this
Code:
MapModData.gT = MapModData.gT or {}
local gT = MapModData.gT
at the beginning of darn near every file including both UI and files in my mod's main state. It's written so that run order doesn't matter. Whichever file happens to run first creates the gT table and a pointer to it contained in MapModData. The second line creates a local variable in every file that has a pointer to gT. Then I put all my tables I want saved into gT.

Hi Pazyryk, thanks for this method. It has not worked for me, even though I've placed this pointer on every lua file in project (even CityBannerManager, and the main...). I suspect I am just loading up my project in the wrong way. I may look at your project for some ideas there.

Thanks again for TSL!
 
Is your mod not "seeing" MapModData? I can't imagine how this would happen, but you can test it easily in Live Tuner. Just try printing MapModData after selecting different Lua states. You should always see a table (the same table) from any state.
 
Is your mod not "seeing" MapModData? I can't imagine how this would happen, but you can test it easily in Live Tuner. Just try printing MapModData after selecting different Lua states. You should always see a table (the same table) from any state.
Interesting.... I had never really messed with Firetuner in this way....

So, while my mod is running this returns same table for all states EXCEPT the main state, which returns nil. My mods main and CityBannerManager return same table value... what's the issue there?
 
"Main" state doesn't matter. Basically, if you can print it in the fire tuner in a given state, then the name is a "global" that should be visible anywhere in that state (unless you do something like local MapModData = nil, which won't hurt MapModData at all but will effectively sandbox it).

Given that you can access MapModData, I can't see why the code above doesn't work for you.
 
Given that you can access MapModData, I can't see why the code above doesn't work for you.

*EDIT: Having given this a bit more thought, I realized I needed to set up a local in each context for table ... something like
Code:
local t = gT.g_PlagueCities

Which indeed was part of the problem. Also in CBM state I had to call local to define MapModData.

Overall I find TSL extremely easy to use, much more so that saveutils/sharedata. So thanks again.
 
Which indeed was part of the problem. Also in CBM state I had to call local to define MapModData.
I'm still not following you here. MapModData (like most everything in Lua including function names) is just a name. Names can be redefined in any scope (or undefined with name=nil). But MapModData should be a global in any Lua scope that already holds a pointer to one single table (the same table pointed to by all those different MapModData's). You shouldn't need to do anything with MapModData itself. You can (optionally) create a local variable in a given file (or even narrower scope) that points to this table with "local MapModData = MapModData". That should never be necessary but can speed things along if you have many accesses to that table.

If you type the line "MapModData.gT = {}" you are really doing three things: 1) Creating a table (the "{}" part) which, like any data in Lua, is fundamentally nameless. 2) Creating an element in the table MapModData called gT (table elements do have names). 3) Putting into that table element a pointer to the table you created. You can create a new pointer to the same table by "local bunnies = MapModData.gT". Neither "MapModData.gT" nor "bunnies" really is the table. Both hold pointers to the same table, though one happens to be a local variable and the other an element of a table. If you remove all pointers to a table, then Lua will quietly (without you telling it to) garbage collect that table since it is now eternally inaccessible to you.

The logic behind "MapModData.gT = MapModData.gT or {}" is as follows:
  1. If MapModData.gT does not exist yet, then MapModData.gT (the left side of the "or") evaluates to nil and Lua executes the stuff to the right of "or". I.e., it creates a new table and assigns a pointer to the new table to the just-now created element gT in MapModData.
  2. If MapModData.gT already exists (because the code ran from some other file or state before this) then nothing happens. The table element is unchanged and Lua does not even look at the right side of the "or" (so no new table is inited). (Due to some Lua optimization this kind of construction is faster than an "if then" construction.)
After this, MapModData.gT now holds a pointer to one single table (the same table) from any scope, regardless of file run order. After this, it is now safe to create additional (local) pointers to this table from many different scopes with something like "local gT = MapModData.gT".

Perhaps it's confusing to use the same name for a local variable and a table element or a global variable, but this is common in Lua. E.g., "local MapModData = MapModData" or "local print = print" or "local sort = table.sort". This is very convenient because you can do it at the top of any scope without having to change the code below (well, the 3rd example requires some re-coding). But it has a major impact on the compiled code and resulting execution time since local variables are easier to access than global.

The last piece of the puzzle that may be causing confusion here is to understand that "globals" in Lua aren't really global. They are themselves local to the particular state. In Civ5, each UIAddIn (with all its included files) is in it's own state. "Gobals" can be redefined in any state without affecting that name in other states. There are a bunch of globals already defined for you in all of these different states, including, for examples, "print" and "MapModData" (the former added by the base Lua package and the latter kindly provided by Firaxis). But you can redefine any of these in any state or undefine it with a nil assignment. You can use the Fire Tuner to see what particular globals are defined in each state. As you observed, "MapModData" is not defined in Main state, though it is in any "mod-accessible" state. Conversely, "_G" and "io" are defined in Main state but have been sandboxed by Firaxis in all other states so aren't available to modders. Even more tricky is a case like "os", which seems to exist in all states but actually (I suspect) points to a different table in Main versus other states, so modders can't do something like os.remove(<your harddrive>) from a mod but they can use os.clock() (or it could be the same table with restricted access to elements -- I'm not sure how they sandbox individual table elements like this).

Not sure if any of this helps, but your language makes me think that something is missing in your "Lua think" that is leading to an error.

Overall I find TSL extremely easy to use, much more so that saveutils/sharedata. So thanks again.
Yes, I agree, though I didn't want to blow my own horn too much :). saveutils/sharedata was created before we were given SaveGameDB. What's nice about TSL is that I don't have to think about it at all. Tables are just magically restored with their prior contents after game load. (The bad thing is the awkward way we have to intercept gamesaves since we still lack an Events for this.)
 
Hi Paz,

Since BNW release I've noticed some serious saved data corruption using TSL after multiple reloads from saved games. What I've noticed is that past table data seems to be reconstituted (seemingly at random) when loading from saved games, generally after the second or third load. The tuner error that precedes these corruptions is:

!!!! Error loading tables: checksumOld=679, checksumNew=1316 !!!!

I'm doing some more testing so I will provide more details as they become apparent. If you'd like to see the context for TSL, I'm using it in my Health and Plague mod.

best,
FA
 
Thanks for that info FA, I just converted to using this for my BNW relaunch of Corporations, so I will be sure to take a look once I get into testing later this week.

-Envoy
 
That's troubling. Unfortunately, I'm on the road for the next 3 weeks so won't be able to sort this out for a while.

The checksum test isn't totally foolproof. You can have a "false error" from international characters. They work fine, but the bytes get counted differently on the way in and out. I have occasionally seen wrong checksum in cases where I don't think the above applies (with G&K). I haven't tracked it down yet and I'm not sure whether it is an actual data error or an error in the checksum. I suspected the latter because I never saw a mod error from it. There could be something trivial like true/false being counted as 1/0 (for checksum) even though the data is restored in its proper type.

The key questions for you are: Is the data wrong? If so, what exactly isn't correct? I'd like to add some code to give more feedback so you know exactly what piece of data TableSaverLoader thinks is different going in versus out. But I won't be able to do that until the end of August.


Another issue to keep in mind is exactly when TableSaverLoader runs. I'm stating the obvious: but it only saves data when you run it. There is a big problem with autosaves since they happen at the end of the barb turn and there is no way to trigger Lua right before that. The best you can do is the beginning of the barb turn. So if some data changes during barb turn it won't be persisted. But in this case the checksum will be correct (data in was the same as data out) but the data itself will be wrong.
 
The key questions for you are: Is the data wrong? If so, what exactly isn't correct?
The data might be wrong, but more importantly it is old. As an example, I load a game from save and a persistent table now apparently holds data that were added and removed 200 turns previously. However, upon printing the table elements, the data seems to be current and accurate, even though my script will reference and try to process the older data (apparent from print statements). Manually deleting all table elements does not resolve this issue, and the script continues to reference and process old table data. It's almost as if there are multiple versions of the same table in the saved data. And reloading from the same saved game multiple times does not yield consistent corruption.

I could be manipulating my table data incorrectly, (I use the standard table library functions), or using your tool incorrectly, but again I hadn't noticed this issue until after BNW release. Previously I had tested the mod under GNK through upwards of 1500 turns, including multiple save/loads, without any apparent saved data errors. And the error is extremely easy to spot because after a corrupt reload there are nil value errors when referencing the saved table.

Another issue to keep in mind is exactly when TableSaverLoader runs.
I use your save handler, so there is an autosave. I was first able to replicate this issue when reloading multiple times from past autosaves. However, I can now confirm that data is sometimes corrupted from loading from manual saves.
 
It's hard for me to imagine any incompatibility with BNW. Unless they totally broke SaveGameDB.

Are you sure you aren't re-initing a table somewhere?

local a = {}
a[1] = "dog"
TableSave(a, "myMod")
local a = {}
a[1] = "cat"
TableSave(a, "myMod")

TableLoad(a, "myMod")
print(a[1])
--prints "dog", I think...

Remember that "a" is not a table. It is a label associated with a table. But here I've passed one table to TableSave and then assigned a totally different (new) table to a. It's not a good situation because TableSaverLoader keeps track of data passed to it (so it doesn't need to write DB for unchanged values). Here it is tracking one table from the first call, but I'm passing a totally different table in the second call. I'm not really sure how TableSaverLoader will handle this, but I expect it might get very confused. There is still a table in Lua that has a value "dog" at index 1. It may be checking this older table and saying "the table you first passed to me still has "dog" at index 1, so no need to update DB".

That's just a guess from your "old data" comment.
 
Pazyryk, would you be willing to spend 10 minutes look at my implementation of your utility in github to give it a gutcheck to see if I've done it right? It appears to be working just fine...

Envoy
 
It's hard for me to imagine any incompatibility with BNW. Unless they totally broke SaveGameDB.

Are you sure you aren't re-initing a table somewhere?
...
That's just a guess from your "old data" comment.

I don't think so. I initialize gT this way in defines (loads first)

Code:
MapModData.gT		= MapModData.gT or {}
gT			= MapModData.gT
gPlagueCities		= {}
gT.gPlagueCities	= gPlagueCities
I localize saved table (where needed) this way
Code:
local gPlagueCities	= gPlagueCities

Also, this is not a consistent error, and reloading from the same saved game sometimes results in corrupted data and sometimes does not. I have gone fully 20 saves without corruption, but sometimes no more than three.
 
Did we ever come to a conclusion as to whether or not things are working ok? I am getting ready to release Corporations for BNW and would like to use this as my save game mechanism. I don't seem to be having an issues, but I haven't tested as thoroughly loading and unloading as I should have yet.

Envoy
 
TableSaverLoader is advertised to work for "arbitrary nesting structure" and allow you to do whatever the heck you want with tables, as long as they are referenced by a master table. But I recently found a flaw.

Code:
gT = {}
local tableA = {}
local tableB = {}

gT.tableA = tableA
gT.tableB = tableB
Above code is the basic idea. After this you can save and load gT and everything nested comes along for the ride, and is reconstructed properly on load. For example, you could subsequently delete a table or create a new one (nesting it in the master) or add new strangely nested table to an already referenced table:
Code:
tableB = nil
tableC = {}
gT.tableC = tableC
tableA.newNestedTables = {{"cow", "dog"},{"car", "plane"}, {{{1}}}}
You can even have a table that is multiply referenced, as in this example:

Code:
gT = {}
local tableA = {}
local tableB = {}
local doubleRefTable = {}
tableA[1] = doubleRefTable 
tableB[1] = doubleRefTable
TableSaverLoader is smart enough to know that doubleRefTable is really only one table, and will rebuild it that way on load.

So everything above is OK.

However, this breaks TableSaverLoader (current 0.12 version):
Code:
tableA.a = {}
tableA.b = a
tableA.a = nil
It gets confused because the original table is still there. Contrary to appearance (at least to a non-Lua programmer), there is no table deletion in that code. TableSaverLoader knows the table is still there, but doesn't know that its internal reference to the table is wrong. I does give you a nice little warning "your data is corrupt" so you at least know something's wrong.

I have a fix for this (I'm currently using it for Éa). Unfortunately, the fix solves the problem above but does so by assuming you never have a table multiply referenced (the code example immediately above the last). I could probably do some fancy coding to make both scenarios work, but I really have other priorities now...

I think what I probably should do is release version 0.13 (what I'm currently using) and remove my "any arbitrary structure" claim. Specifically, your table nesting structure has to be "tree-like" -- i.e., the branches never re-connect. It's not a great limitation.
 
That sounds slightly confusing, but then again I'm still wrapping my head around all this stuff.
Thankfully, I think my table structures are fairly straight-forward, even if they are nested like crazy.

That said, I popped in here with a slightly different question.
I'm looking over my Lua script and trying to make things cleaner here and there, and I finally decided to take a look at the TabelSaverLoader initialization function at the top of the file.

In your OP, you suggested using this:
Code:
	local bNewGame = true
	local DBQuery = Modding.OpenSaveData().Query
	for row in DBQuery("SELECT name FROM sqlite_master WHERE name='Ea_Info'") do
		if row.name then bNewGame = false end	-- presence of Ea_Info tells us that game already in session
	end

	if bNewGame then
		print("Initializiing for new game...")
	else
		print("Initializing for loaded game...")
		TableLoad(gT, "Ea")
	end

But, I was wondering if I could clean that up; if that code would be identical to:
Code:
local DBQuery = Modding.OpenSaveData().Query
for row in DBQuery("SELECT name FROM sqlite_master WHERE name='Ea_Info'") do
	if row.name then
		print("Initializing for loaded game...")
		TableLoad(gT, "Ea")
		break -- presence of Ea_Info tells us that game already in session
	else
		print("Initializiing for new game...")
	end
end

Either way, thank you for your continued development of this tool!
 
@DarkScythe, the only problem above is that you will see "Initializiing for new game..." twice when loading a game. The iterator will cycle once for each table in Civ5SavedGameDatabase.db, which includes Ea_Data, Ea_Info and SimpleValues (so 3 times). It's only a problem if you put some code in that line with the print statement assuming that it is true. [No, that's not right] ... you will never see "Initializiing for new game...". The iterator only iterates if Ea_Info exists based on the "SELECT name FROM sqlite_master WHERE name='Ea_Info'" statement. So you can never get to the else block.

@All,

I've updated the How to Hook Up section in the OP with a helpful (I think) conceptual paragraph and a much much easier way to hook up if you are able to do a small dll modification (adding a GameSave event).

I also edited the Compatibility between mods and Sharing between mods sections, since I have a much better understanding of Lua states now then I did 3 yrs ago.
 
Ah! You are right.. For some reason I had overlooked that yesterday.
If the next item returns nil, it simply doesn't continue with the else condition.

Edit:
I think I will still try to compact that function a little bit, if only so that it looks a bit cleaner.

That said, I'm not sure if your red warning text about Civ5SavedGameDatabase.db was always up there, or if it was recently added with your OP update, but that does explain some weird issues I experienced a little while ago, and wound up creating a new function on game load to doublecheck if the tables were empty or not at the beginning of a game.
It seemed odd that simply assigning gT to {} in the beginning wouldn't clear it, and it makes sense (and I had suspected this back then) that the game wasn't able to write to the database.
Still, I'm not quite sure how that wound up reloading data from an old game, so my function just checks for any data, and replaces the table with a new {} if it finds any. I'm not sure if I should actually use a for loop to go and manually remove every element; I don't know if either method would actually clear the table if the database was not writable for some reason (open in a database viewer, for example.)
This whole thing of trying to create a more elegant method of searching for table data is partly what drove me to try and shorten your suggested init function a little bit.

I wonder though, can you add a warning message to output from TableSaverLoader if Civ5SavedGameDatabase.db is not writable for whatever reason?
 
Back
Top Bottom