TableSaverLoader, for persisting Lua table data through game save/load

As of Aug 26, 2014, I changed the download file name to add "016" to the end. I suggest you do this yourself if using it. The reason is that if two mods use TableSaverLoader but have different versions, it is random which of the (same-named) files will load. So if you are using new 0.16 functionality but 0.14 loads, then attempting to use that new functionality will be cause an error.

Current version is 0.16:

Download here

Changes:

• 0.11 (Sep 23, 2012) Fixed a bug in 0.10 that caused a crash when a persisted variable changed type.
• 0.12 (May 1, 2013) Adds TableSave to LuaEvents so it can be called from another state.
• 0.13 (Jul 13, 2014) Fixes a bug that occurred when a nested table was moved from one table index to another; recoded from ground-up for speed and robustness; there are some new (but easy) table rules you have to follow
• 0.14 (Jul 14, 2014) Fixed error in 0.13 where it would not save an empty table (so that index would be nil on loading)
• 0.15 (never released) More speed and memory optimizations: in particular, no longer stores a string representation of numbers; TableLoad now returns true (if save tables exist) or false, so can be used to detect a loaded game
• 0.16 (Aug 7, 2014) Fixed to allow apostrophe (') in string values (I'm fairly confident that any string value is OK now) and NaN, Inf and -Inf as numbers; more memory optimization

The idea behind TableSaverLoader is that you do all game logic in Lua tables, then reference those tables in a master table for "preservation" through game exit/reload. TableSaverLoader will save your master table and all nested tables (at any level) to Civ5SavedGameDatabase.db, then restore them on game load. You need to hook up TableSave and TableLoad to run with game save and load as described below. TableSave recursively saves a table and all nested tables (at any level) to Civ5SavedGameDatabase.db (which is packed away in the Civ5Save file during game save). TableLoad rebuilds the tables from Civ5SavedGameDatabase.db.

TableSave is fast, very fast...

Code:

TableSave time: 0.016000000000076, checksum: 102244, inserts: 5, updates: 97, unchanged: 1687, deletes: 2

That's a typical turn in Éa where I have about 1800 table items in total that are persisted. In the turn above, 97 of those items changed (updates), 5 new items were added (inserts) and 2 itmes were nilled (deletes). There are two reasons why TableSave is fast: 1) it knows what table items have changed since the last TableSave and only updates those items in the DB; and 2) it can pack up to 300 inserts or 600 updates into a single DB transaction. Each transaction requires a bit of time, so doing as many updates/inserts as possible in a single transaction is the way to go (note that changing a single value with DB:SetValue() is one transaction).

Table rules

• Keys must be strings or integers. If a string, it cannot be the empty string ("") or contain the characters #, $, ' (apostrophe), or international characters (specifically, UTF-8 characters encoded by >1 byte may screw up key parsing but I'm not 100% sure about that). To be complete, I should say that non-integer keys will work too (if you need that for some strange reason) so long as the condition x == tonumber(tostring(x)) is true.
• Values can be anything of type boolean, number, string or table (but NOT function, userdata or thread). There are no restricted strings or substrings of any kind.
• Table nesting structure must be "tree-like" relative to the master table provided to TableSave. In other words, the values in your nested tables (the "branch tips") can be reached by following only one path from the master table (the "trunk") with branches never reconnecting. However, it's perfectly OK if these tables are referenced from elsewhere.

Does TableLoad restore my tables and values exactly as they were?

To the best of my knowledge, you can save any string and get back exactly what you saved (v0.16 fixed a limitation with apostrophe, now OK). The new v0.16 even works for Lua "funny numbers" like 0/0, 1/0 and -1/0 (yes, these are valid numbers in Lua!). There is one case that I know where the data from TableLoad() isn't exactly what you had at TableSave(), which has to do with differing number precisions. It just so happens that the method tostring() has a lower precision than Lua numbers themselves (probably due to some C++ type declaration in tostring):

Code:

> x = 10000000000001 ; print(x == tonumber(tostring(x)), tostring(x))
true	10000000000001
> x = 100000000000001 ; print(x == tonumber(tostring(x)), tostring(x))
[B]false	[/B]1e+014
> x = 1000000000000001 ; print(x == tonumber(tostring(x)), tostring(x))
[B]false	[/B]1e+015
> x = 10000000000000001 ; print(x == tonumber(tostring(x)), tostring(x))
true	1e+016

In the middle two examples above, x does not equal tonumber(tostring(x)) due to round-off out at the 14th and 15th digits (the last example is equal again because I've exceeded Lua's own number precision). All numbers go through tostring in TableSave() and then tonumber in TableLoad() so you will have a bit of round-off change out at that 1e-14 level of precision. But it certainly won't affect any integers you would conceivably use (not even long integers) and we aren't simulating nuclear explosions here.

Tables loaded in a freshly started Civ5 session are (of course) not the same tables that you saved in TableSave. But they will have the same exact structure on load if you follow the "tree-like" rule above. In particular, empty tables are perfectly fine and you can nest to any level you want. My own mod Éa saves tables nested to level 12, but I would feel perfectly safe nesting to level 100 or 1000. (Anyone who tells you there is some limit to table nesting in Lua doesn't understand how tables are represented in Lua.)

I originally coded TableSaverLoader to handle tables "of arbitrary structure" including loopy structures and maybe even tables that referenced themselves. But this was rather a mess of code and the lack of rules cost a lot of overhead. I imposed the "tree-like" rule so I could streamline the code substantially in 0.13.

Usage

You should hook up TableLoad() to run when your mod inits from a loaded game, and TableSave() to run right before any game save (including quicksaves and autosaves). The latter is somewhat complicated but is covered in How to Hook Up section below.

Persisted tables can be set up as in the following example, where gT is the "master persisted table":

Code:

gT = {}
gValues = {}
gPlayers = {}
gReligions = {}
--these are global but they could just as well be local
							
gT = {	gValues = gValues,			
	gPlayers = gPlayers, 		
	gReligions = gReligions }
--I've given keys the same name as the tables, but that's not necessary; 
--the only thing that matters here is that the tables are nested in gT

These tables can be filled with whatever you want now (following rules above) including nested tables at any level. For example, gPlayers can be indexed to hold a table for each player, and those tables can hold more tables and/or values. Table fields containing values or tables can be created, altered or deleted at any time. The master table itself (gT) can contain non-table items if you want.

Then your code should run these:

Code:

TableSave(gT, "MyMod")  --before all game saves of any kind
TableLoad(gT, "MyMod")  --during game load to rebuild gT and nested tables as before

You should not add TableSave() in the logic of your mod except right before saves. Even if 10000 things change and you have not saved for 50 turns, no problem. TableSave() only needs to be called right before the next save. Running it at any other time serves no purpose! This means that you only need to fuss with TableSaverLoader code once. After it is hooked up so that it successfully runs on game loads and game saves (and you have referenced the tables you want to persist in your master table) then you can forget about TableSaverLoader forever.

Compatibility between mods

There should be no problem as long as you use a unique string (your mod name for example) in your calls to TableSave and TableLoad. It should be possible for two mods to use TableSaverLoader at the same time as long as they are each saving/loading their own tables (i.e., the "MyMod" text above has to be different for each mod).

Sharing between mods

Don't have two different mods (or two different states) including TableSaverLoader and saving/loading the same table. This will break since each mod (or state) is keeping its own Lua representation of what is in that DB table, and when one updates, inserts or deletes the other won't know it. You can have each save/loading their own global persisted table as explained above. Or... you can have one mod running TableSaverLoader and then share the "global persisted table" among mods. It's easy to create a table and pass it among all mods so that they share it. Just take advantage of MapModData table which is already shared and named in all Lua states:

Code:

MapModData.gT = MapModData.gT or {}
gT = MapModData.gT

Load order doesn't matter with this code. Whichever mod happens to load first will create a table. All subsequently loaded mods will use the already created table and name it "gT" within their respective environment. You can use the same exact code to share a table among states within a mod, for example to pass data between your mod state and UI states.

Does this replace SaveUtils and ShareUtils?

I don't know. I don't understand the need for ShareUtils since it is easy for mods to share tables. (But perhaps I don't understand the purpose of ShareUtils?) I built TableSaverLoader for my own needs: primarily, to have many tables of complex structure that maintain content through save/load. I didn't want to write a separate save/load function for each table, and I didn't want to have to worry about table structure (I do a lot of Lua logic and I would rather focus on that than how to save/load each table).

As far as speed goes, my guess is that SaveUtils will be faster than TableSaverLoader if you use its save function infrequently (specifically, less than your autosave frequency). Conversely, TableSaverLoader will be faster if some or all of your data changes much more frequently than that. TableSaverLoader takes a speed hit for doing a database transaction (which SaveUtils doesn't do). However, TableSaverLoader only serializes at game save and only the table items that have changed since the last save, whereas SaveUtils serializes everything in your table every time you change anything.

What not to do

Don't try to save/load two different global tables with multiple calls to TableSave and TableLoad from the same mod - for example, TableSave(gT1, "MyModTable1") and TableSave(gT2, "MyModTable2"). It won't work. TableSaverLoader assumes that there is only one master table for the mod that is calling it. Such a feature could be added with only a tiny amount of overhead. But if you think you want this, it is most likely that you are misunderstanding something about TableSaverLoader function. Specifically, if you wanted to persist gT1 and gT2, all you would have to do is gT = {gT1, gT2} and then persist only gT.

How to Hook Up

Some conceptual framework first. By calling TableSave(), what you are doing is packing your persisted Lua data into Civ5SavedGameDatabase.db. Whenever a game save occurs, the dll serializes and packs Civ5SavedGameDatabase.db into the Civ5Save file. This process is reversed when loading a game: the dll takes data packed away in the Civ5Save file and uses it to construct Civ5SavedGameDatabase.db. Then you call TableLoad() which unpacks previously saved Lua data from Civ5SavedGameDatabase.db back into Lua tables.

When to call TableLoad()
This is easy. Do it sometime during your mod's Lua init if this is a loaded game. For version 0.16, I've modded TableLoad() so that it returns true if there were DB tables to load (i.e., there was a previous TableSave() for this game) or false otherwise. This way you can use TableLoad() to figure out whether this is a new or loaded game. In my mod it is organized like this:

Code:

function OnModLoaded() --called from end of last mod file to load

	[B]local bNewGame = not TableLoad(gT, "MyMod")[/B]

	[COLOR="Gray"]-- < mod-specific init code here depending on bNewGame >[/COLOR]

	TableSave(gT, "MyMod")	--I talk about this more below
end


OnModLoaded() --add at bottom of whatever file loads last for your mod

Change the "MyMod" part to any short string that is likely to be unique for your mod. Warning: if you keep Civ5SavedGameDatabase.db open in a DB viewer application (SQLite Manager or whatever) while starting a new game, Civ5 won't overwrite it and your code will be fooled into thinking you are loading an existing game! That's never a problem for mod users, but it might cause you (as mod developer) some serious confusion.

When to call TableSave()
This is easy in concept: run it immediately before any game save. Unfortunately, Firaxis did not provided an event hook for game saves. If you can add a GameEvents.GameSave to the dll you are using, do that (see instructions in next section). If not, then use the Lua-only hack below that.

How to catch a game save using a dll mod
In theory this should be super easy (if you are a dll modder) by adding a simple GameEvents.GameSave hook in either CvDllGame::Write() or CvGame::Write() so it fires before Civ5SavedGameDatabase.db serialization into the gamesave file (which happens at the bottom of CvGame::Write()). I did that for a while and it works. However, sometimes the game would go into a permanent hang when player presses Begin Your Journey. The first autosave happens to fire right after that. However, I learned two things through a lot of testing:

1. The hang had nothing to do with TableSave or really even any Lua code running. The initial call gDLL->GetScriptSystem() was sufficient to cause the hang.
2. The hang never happened at any other save for me or any mod players

The solution then (which is very well tested now by many players on different machines) was to not run GameEvents code, in particular gDLL->GetScriptSystem(), during that first save. Maybe a better coder can figure out a simpler way to do this, but here's the code I used (marked in blue) always inside the #ifdef #endif blocks:

Spoiler :

Code:

[B]// In CvGame.h:[/B]

class CvGame
{
public:

[COLOR="Gray"]< snip >[/COLOR]

	bool getFOW();
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE
	void SetGameEventsSaveGame(bool bNewValue)
	{
		m_bSavedOnce = bNewValue;
	};
#endif[/COLOR]
	int getPitbossTurnTime() const;

[COLOR="Gray"]< snip >[/COLOR]

protected:

[COLOR="Gray"]< snip >[/COLOR]

	bool m_bFOW;
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE
	bool m_bSavedOnce;
#endif[/COLOR]
	// slewis - tutorial values
	bool m_bStaticTutorialActive;


[B]// In CvGame.cpp:[/B]

CvGame::CvGame() :
	m_jonRand(false)
	, m_endTurnTimer()
	, m_endTurnTimerSemaphore(0)
	, m_curTurnTimer()
	, m_timeSinceGameTurnStart()
	, m_fCurrentTurnTimerPauseDelta(0.f)
	, m_sentAutoMoves(false)
	, m_bForceEndingTurn(false)
	, m_pDiploResponseQuery(NULL)
	, m_bFOW(true)
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE
	, m_bSavedOnce(false)
#endif[/COLOR]
	, m_bArchaeologyTriggered(false)

[COLOR="Gray"]< snip >[/COLOR]

	m_bFOW = true;
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE
	m_bSavedOnce = false;
#endif[/COLOR]
	m_bFinalInitialized = false;

[COLOR="Gray"]< snip >[/COLOR]

void CvGame::Write(FDataStream& kStream) const
{
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE // Paz - This will fire before Civ5SavedGameDatabase.db serialization into the gamesave file
	if (m_bSavedOnce) // But... running gDLL->GetScriptSystem on initial save causes a game hang, so skip first save
	{
		ICvEngineScriptSystem1* pkScriptSystem = gDLL->GetScriptSystem();
		if (pkScriptSystem)
		{
			CvLuaArgsHandle args;
			bool bResult;
			LuaSupport::CallHook(pkScriptSystem, "GameSave", args.get(), bResult);
		}
	}
#endif[/COLOR]
	// Current version number
	kStream << g_CurrentCvGameVersion;



[B]// In CvDllGame.cpp:[/B]

void CvDllGame::Write(FDataStream& kStream) const
{
	m_pGame->Write(kStream);
[COLOR="Blue"]#ifdef EA_EVENT_GAME_SAVE // Paz - set m_bSavedOnce = true AFTER the first save
	m_pGame->SetGameEventsSaveGame(true);
#endif[/COLOR]
}

Of course, we've jumped through some hoops above to make this GameEvents not work for the first autosave. But that's OK because we know it's going to happen and we can run TableSave before it at the end of mod loading (as I did above). Note that I didn't hook up the first TableSave to LoadScreenClose! I don't really know if this could ever be a problem, but my experience with the initial autosave above made me very leery about doing DB O/I stuff very close to that save. So backing it up to run at the end of mod loading seemed safer to me. For all subsequent saves (no matter how triggered), all you need is the new event:

Code:

local function OnGameSave()
	TableSave(gT, "MyMod")
end
GameEvents.GameSave.Add(OnGameSave)

Done! (You can skip the rest - it doesn't apply to you.)

How to catch a game save using Lua only
It's the same exact concept, but the problem here is being sure that you run TableSave right before any game saves (when the dll packs Civ5SavedGameDatabase.db into the Civ5Save file). That turns out to be a challenge, but luckily for you I've figured out how to do it. The code below intercepts and runs TableSave at exactly 6 situations. I believe this covers every way that a save can happen in Civ5:

1. Player goes to the Save Game screen via the Game Menu
2. Player goes to the Save Game screen by hitting Cntr-s
3. Player presses Quick Save from the Game Menu
4. Player does a quicksave by hitting F11
5. "AutoSave_Initial_0000" autosave that happen right after player presses "Begin your Journey"
6. Subsequent autosaves that happen at end of barb turn after barb unit moves but before the next turn starts (but see Config.ini warning at bottom of post!)

That's a lot if different conditions to try to intercept, but the code below seems to catch them all. And most importantly, catches them immediately before the save. You can put this code anywhere in your mod, or integrate it into existing functions. It's well worth testing for yourself that TableSave runs before any save of any kind (it prints to Fire Tuner so you can know). It should, but post in this thread if it doesn't for you.

Code:

function OnModLoaded() --called from end of last mod file to load

	[COLOR="Gray"]local bNewGame = not TableLoad(gT, "MyMod")

	-- < mod-specific init code here depending on bNewGame >[/COLOR]

	TableSave(gT, "MyMod")  --before the initial autosave that runs for both new and loaded games
end

function OnEnterGame()   --Runs when Begin or Continue Your Journey pressed
	print("Player entering game ...")
	ContextPtr:LookUpControl("/InGame/GameMenu/SaveGameButton"):RegisterCallback(Mouse.eLClick, SaveGameIntercept)
	ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton"):RegisterCallback(Mouse.eLClick, QuickSaveIntercept)
end
Events.LoadScreenClose.Add(OnEnterGame)

function SaveGameIntercept()	--overrides Civ5 code when player presses Save Game from Game Menu or Cntr-s
	TableSave(gT, "MyMod")
	UIManager:QueuePopup(ContextPtr:LookUpControl("/InGame/GameMenu/SaveMenu"), PopupPriority.SaveMenu)
end

function QuickSaveIntercept()	--overrides Civ5 code when player presses Quick Save from Game Menu or F11
	TableSave(gT, "MyMod")
	UI.QuickSave()
end

local autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
function OnGameOptionsChanged()
	autoSaveFreq = OptionsManager.GetTurnsBetweenAutosave_Cached()
end
Events.GameOptionsChanged.Add(OnGameOptionsChanged)

function OnAIProcessingEndedForPlayer(iPlayer)
	if iPlayer == 63 then					--runs on barb turn AFTER barb unit moves (very close to the regular autosave)
		if Game.GetGameTurn() % autoSaveFreq == 0 then	--only need to do on autosave turns
			TableSave(gT, "MyMod")
		end
	end
end
Events.AIProcessingEndedForPlayer.Add(OnAIProcessingEndedForPlayer)

function InputHandler(uiMsg, wParam, lParam)
	if uiMsg == KeyEvents.KeyDown then
		if wParam == Keys.VK_F11 then
			QuickSaveIntercept()		--F11 Quicksave
        		return true
		elseif wParam == Keys.S and UIManager:GetControl() then
			SaveGameIntercept()			--ctrl-s
			return true
		end
	end
end
ContextPtr:SetInputHandler(InputHandler)


OnModLoaded() --add at bottom of whatever file loads last for your mod

Note: It's possible that the above won't work depending on how your mod loads or if you mess around with ContextPtr. For example, the Cntr-S and F11 intercepts weren't working for me in tests until I figured out (after an hour of troubleshooting) that my code had SetHide = true for the mod's main context, preventing InputHandler from ever firing. So the problem was my own fault, but these things can be hard to figure out.

Folks who installed TableSaverLoader previously from my example code should remove all those calls to TableSave (including a LuaEvents.TableSaverLoaderSave in SaveGame.lua) and do it as above. The old system in particular had a big gap between TableSave() (at the begining of the barb turn) and the actual autosave (at the end of the barb turn). This is fixed above.

I'm not really sure but it may be dangerous to hook TableSave up to Events.LoadScreenClose. Theoretically, that is a perfect place for it since it fires right before the autosave that generates the "Initial" save. However, there is something funny about this save in particular (based on my dll modding experience) and I worry about firing up the DB O/I right at that time. That's why I run the first TableSave from OnModLoaded in above code.

Warning about graphic engine Events: (This warning applies only to the Lua method for intercepting game saves, not the dll method.) If you hook up game rule logic to the graphics engine, for example Events.RunCombatSim or Events.EndCombatSim, then there are no guaratees whatsoever about exactly when those will fire. I strongly recommend against using those for anything other than UI. But if you insist on doing it anyway, be advised that those two Events (and possibly other graphic engine Events) could fire between the Lua autosave intercept above and the actual autosave. You could account for this by running TableSave from whatever graphic engine hook you are using after you change some persisted data, but then you are running TableSave with every combat and that is a lot of overhead. The problem here is hooking to the graphic engine. Any GameEvents are perfectly safe (including Gedemon's combat events) because these fire from the dll with actual combat and are all finished when AIProcessingEndedForPlayer fires. But the graphics engine may be acting way behind the dll systems.

Warning about Config.ini file: (This warning applies only to the Lua method for intercepting game saves, not the dll method.) There is a setting in Config.ini that could mess up the Lua autosave intercept:

Code:

; Saves the game after the human player presses 'next turn' but before the game logic advances
PostTurnAutosaves = 0

If a player changed that to 1, then the Lua intercept above would not be firing at the right time to catch autosaves. So you might want to include a warning about that with your mod. Perhaps there is way to test for that setting in-game and then intercept at the correct time. But I haven't spent any time trying to figure that out.

 

This seems to be bug free. I've been using it in a lot of test script for several weeks (sadly, not a lot of other people seem to be using it). So I don't consider it a beta any more. I have one change that I intend to do that will allow the player to save to multiple tables in SaveGameDB (currently, multiple mods can save to different tables, but one mod can save only to one table). That's a very specialized need, though, so I probably won't get around to this until mid-December.

 

I seem to understand the prospect,however, I am lost as how to add to the game. Set some examples. Never ever get discouraged. I have multiple mods which I play with and few times does the game crash to desktop. Is the latest version able to stop this. Also, will it disrupt save replace SaveUtils and ShareUtils. Not that many have posted on thread. Man keep us in mind. Pretty soon dll to modify.

 

@The only,

This is really only useful if you are building your own mod.

All of the very advanced mods have some need for memory, but all of them have implemented their own solutions (or used SaveUtils). I personally feel that TableSaverLoader is a more elegant solution, but that is a biased opinion. It's certainly more generic and up-to-date. (By up-to-date I mean that it uses the new SavedGameDB rather than the once-killed-then-resurrected script functions.)

 

Not much traffic here but I hope some modders are finding this useful.

I'll note that I've been running this for several months now during a lot of experimental modding (sometimes with several thousand table elements with a lot of table deletion/creation). I have not yet ever seen a failed checksum (except when testing the checksum test) so it appears to be entirely stable. Also, I've seen at least one advanced modder claim that SavedGameDB is not entirely stable across game save/reloads. My experience contradicts this.

 

Yes, I do want to use this. But my lua skills are limited it seems. I've tried what I could think of and failed, so I'm trying to get my facts straight (every statement is a question mark for me by now):

I have TableSaverLoader.lua imported into the VFS; ExampleMain.lua into VFS and added to the content as InGameUIAddin. Next I want to use gT in other lua files so I import ExampleMain.lua in said files.
But it doesn't work. I got past the "gT is nil" stage, but now I'm stuck in "gT.gPlots is nil" (or in full: attempt to index field 'gPlots' (a nil value)). I've tried putting in some values in them right after the declaration in ExampleMain.lua, but doesn't help. I also get an error in InGame.lua now:

[43170.593] Runtime Error: [string "Assets/UI/InGame/InGame.lua"]:1076: attempt to index local 'addinFile' (a nil value)

Also, a handful of imports in files' global scope results in many more executions of your files, which I doubt is supposed to happen:

[43169.922] CityStateGreetingPopup: loaded VarSaverLoader.lua
[43169.922] CityStateDiploPopup: loaded VarSaverLoader.lua
[43169.938] TechAwardPopup: loaded VarSaverLoader.lua
[43169.953] WonderPopup: loaded VarSaverLoader.lua
[43170.047] DiploRelationships: loaded VarSaverLoader.lua
[43170.421] ChooseFreeItem: loaded VarSaverLoader.lua
[43170.437] ProductionPopup: loaded VarSaverLoader.lua
[43170.453] LoadMenu: loaded VarSaverLoader.lua
[43170.468] SaveMenu: loaded VarSaverLoader.lua
[43170.531] SaveMenu: loaded VarSaverLoader.lua


In short: I am in need of the basic basics of using the lua files you provided: how to activate, make available, call, ... I have written lua without saving functionality without problems, but somehow I utterly fail at using both this and SaveUtils.

 

Funny, I just made a comment here that may address your problem (see post #4): http://forums.civfanatics.com/showthread.php?t=451968. Basically, you're adding these to your mod twice. Only ExampleMain should be an InGameUIAddin (with VFS=False). TableSaverLoader is added to the mod by VFS=True (that's all!, don't do something else to try to add it again).

ExampleMain is only given here to show you how to link TableSaverLoader to your existing "main" lua file, with all of the needed "save intercepts". But if you're just getting started with Lua, then this file may be a good "template" to use as your main Lua file that includes all other Lua files.


I'm not sure this is the source of all of your problems, but its certainly the source of many problems. Try fixing that and try again.

 

Everything is working smoothly now, thanks for the advice! I also found Onni's UI modding guide extremely enlightening, I should have checked it earlier. There were other problems, including stupid stuff like syntax errors, but my bad organization obfuscated things.

I will continue to use TableSaverLoader; with the example main you provided and some basic lua threading knowledge (which I didn't have up till now), it's basically plug and play. Great job!

 

If anyone is interested in trying this, I think that this will work:

TableSave(MapModData, "MyMod")
TableLoad(MapModData, "MyMod")

Now any tables or values that MapModData points to will be "preserved" on game exit/reload (or more accurately, they will die and be resurrected whole again).

I realized based on some conversation in Spatz's subforum that I didn't do a very good job of explaining the overall thinking behind TableSaverLoader. I explain what it does but not how to use it or why it exists in the first place.

Here's the thing, since I released this in October 2011, I've coded about 3000 lines of Lua for my Éa mod. I have about 500 variables that can change at any time and need to be preserved through game exit/reload. In all these 4 months of coding and debugging, I have spent exactly 0 minutes thinking about game exit/reload issues. I am not exaggerating: 0 minutes coding save/load functions; 0 minutes troubleshooting save/load issues; 0 minutes thinking about it. In fact, I'd have to review my code here to remember how to access SavedGameDB. I have just simply forgotten about SaveGameDB, SaveUtils, or any issue related to saving data. I use Lua tables for all game logic and saving/loading happens automatically without any extra coding or thought. That is the reason for TableSaverLoader.

 

I'm not sure if that works, given the seemingly undefined nature of the base MapModData structure, but I KNOW that adding an additional level, like
TableSave(MapModData.Mythology, "Mythology")
works fine (as does the corresponding TableLoad, of course), where my structure variables are things like MapModData.Mythology.PlayerFavor() and other variables on that same level. It's probably safer to do it this way to begin with, to avoid potential conflicts with other mods (or at least the potentially massive overhead of saving the same structures multiple times), and all it costs you is a single line of declarations before any Load is done (where you set "MapModData.Mythology = {}").

 

The reason I think that will work is that TableSave is just a function call, and all arguments (even MapModData) will be turned into local references in the function itself. You just end up with a local table that points to everything that MapModData used to point to. Of course, you will be saving everything in MapModData, which you may or may not desire.

TableSave won't do this. If you have 5 pointers to one table (due to weird table nesting) it will only save the table once (though it will preserve your weird nesting with the 5 pointers to this table).

 

What I mean is the following. Let's say, hypothetically, that I follow your earlier example in my mod and go

TableSave(MapModData, "Mythology")
where all of my data is actually inside the MapModData.Mythology substructure, as I've mentioned before.

Then, someone else makes a mod, and does
TableSave(MapModData, "Empires")
where all of their data is inside the MapModData.Empires substructure.

Then, a third person does
TableSave(MapModData, "Ascension")
where all of THEIR data is inside MapModData.Ascension.

Wouldn't the script attempt to save three copies of the full MapModData structure (within Mythology_Data/Mythology_Info, Empires_Data/Empires_Info, etc.), each of which is much larger than the size it would be for each individual mod? That is, wouldn't Mythology_Data also include the MapModData entries added by the Empires and Ascension mods, since it'd have no way to identify which variables were specific to that particular mod? Even if it recognized the structure pointers were identical for all three and refused to duplicate the whole superstructure, this'd still seem to have some inefficiencies. And if one of the mods in question didn't use your component, and found some other way to store their MapModData information (or maybe they don't even WANT to store it, and are only using it as temporary storage), then you couldn't even do that sort of redundancy check.

Anyway, that's why I suggested only passing the specific substructure of MapModData used by your particular mod.

 

Yeh, that's bad.

Though if you could coordinate so that only one mod did all of the TableSave/TableLoad calls, it could still work. Everyone would have access to the same data.

Of course, it's not necessary at all. I do essentially the same as you. I call TableSave(gT, "Ea") but then in my init lua and in each UI state I have the lines:

Code:

if not MapModData.gT then
	MapModData.gT = {}
end
local gT = MapModData.gT

so that all of my mod data (in gT) is available anywhere.

 

Sure. And obviously, the three mod names I used are all my own, so it's not hard for me to do that sort of sharing if I wanted to. But I was using the hypothetical case of unconnected mods; since MapModData is just a really, really useful way to pass data around, I expect it to be used by a lot of mods in the future (and as you noted, you use it yourself, so that's at least two of us). So, just for safety's sake, I'll only ever do the explicit reads of MapModData.Mythology and such, to make sure I'm not grabbing anyone else's data. Assuming no one else uses "Mythology", "Empires", or "Ascension", that is; if they did, things could get really awkward.

 

Anyone who runs two "Mythology" mods (thinking something good will come of it) deserves a CTD.

 

Yeah, I've got dibs on that name, so BACK OFF!!!!

More seriously, though, it's just like anything else: whenever possible you should try to steer clear of generic names, to avoid even the possibility of a naming conflict. I really should change that structure name to MapModData.AoM_Mythology to reduce the risk, but honestly, there aren't a whole lot of folks making Civ5 mods right now, and they're all here, so it shouldn't be much of an issue.

 

Well, it's been a year and I finally found a bug.

Version 0.10 crashes if you have a variable that changes type. It's not a good idea to do this anyway, which is why I didn't run into the bug until now. But the updated version (0.11) will now handle this situation properly.

 

I've been using the component for a few days and it works great. Thank you!

I have run into a problem. I have a global function that iterates over one of my saved global tables and returns a boolean; I use this boolean to switch on a new status icon in CityBannerManager.

The global function returns proper values in all contexts except CityBannerManager, however, which apparently cannot access the global table (this is both before and after TableSave).

I have tried a method using saveutils and sharedata, which does expose the global table to CityBannerManager but this somehow intercepts TSL and squashes its functionality.

I realize you don't use sharedata, but I was wondering how you may have solved similar issue in your own mods. I, too, am not worried about cross-mod compatibility.

 

@FramedArchitect

I'm glad someone else is using this.

CityBannerManager (and every other UI) runs in its own state, and so it doesn't have access to any "globals" defined in your mod. If I understand correctly there aren't really any "globals" in Lua, despite the use of that term: everything has some scope that is limited.

Fortunately, there is a table called MapModData this is a "super global". It's seen by all Lua states. So in my mod I have this

Code:

MapModData.gT = MapModData.gT or {}
local gT = MapModData.gT

at the beginning of darn near every file including both UI and files in my mod's main state. It's written so that run order doesn't matter. Whichever file happens to run first creates the gT table and a pointer to it contained in MapModData. The second line creates a local variable in every file that has a pointer to gT. Then I put all my tables I want saved into gT.

 

Btw: I did find one more mild issue (I don't think it deserves to be called "bug"):

If you have string variables that contain international characters like É, then the checksum won't be correct. It'll print an error to Live Tuner and you can see it if you open the MyMod_Data table in the database. However, the international characters seem to load just fine despite the error message. International charaters use variable numbers of bytes, and there is something inconsistent about the way my checksum counts these for save versus load. I'll fix this someday...