TableSaverLoader, for persisting Lua table data through game save/load

[Pazyryk]

I don't know if either method would actually clear the table if the database was not writable for some reason (open in a database viewer, for example.)

As far as I know that is the only reason it might not be writable. The game engine creates this db (and path if needed) or overwrites the existing one whenever you start a new game. And overwrites it again (I think) on game load (or at least the data is cleaned out, in any case).

But if it can't write, or doesn't find anything on load, then you probably want a terminal error anyway, don't you? Maybe it depends on how extensively you depend on that persisted data. For Éa, there is no recovery if the Lua data isn't there for some reason. (And anyway, it's only me the modder that is likely to make the mistake of having that db open in a viewer.)


Yeah, I've been editing text in OP. I only recently implemented the dll mod to add a GameSave event. But I still think there may be easier and better ways to implement the Lua-only game save hack. There are a whole lot of (entirely undocumented) Events fired by the game engine. Some are worth testing:

ExitToMainMenu --I'm not sure if this fires when you exit game to the real main menu or if it fires when you go to the game menu that appears when you click Esc or Menu in the top panel. If the later, then this would be a better way to knock off both "menu saves" (quick and regular) that you get to via this menu. (The keyboard shortcuts cntr-S and F11 are easy enough to intercept as shown in OP.)

AIProcessingEndedForPlayer -- Perhaps this fires at the end of the barb turn before the autosave (assuming player has default PostTurnAutosaves = 0 in their config file)? If so, then running TableSave from that (after testing current player = 63) would work and solve a major problem.

There might be some other useful Event to use, but these have to be tested empirically (since they are all fired from the exe that we don't have source code for). Here's the full set:

Spoiler :

AbortCombatSim, ActivePlayerTurnEnd, ActivePlayerTurnStart, AddPopupTextEvent, AddUnitMoveHexRangeHex, AdvisorDisplayHide, AdvisorDisplayShow, AfterModsActivate, AfterModsDeactivate, AILeaderMessage, AIProcessingEndedForPlayer, AIProcessingStartedForPlayer, AIProcessingStepChanged, AnimationSamplingChanged, AppInitComplete, AudioAdvanceCurrentPlaylistTrack, AudioDebugChangeMusic, AudioPlay2DSound, AudioRewindCurrentPlaylistTrack, AudioVolumeChanged, BeforeModsActivate, BeforeModsDeactivate, BuildingLibrarySwap, CameraProjectionChanged, CameraStartPitchingDown, CameraStartPitchingUp, CameraStartRotatingCCW, CameraStartRotatingCW, CameraStopPitchingDown, CameraStopPitchingUp, CameraStopRotatingCCW, CameraStopRotatingCW, CameraViewChanged, ChangeGameState, CityDestroyedRazedClearedAndSalted, CityHandleCreated, CityRuinsCreated, ClearDiplomacyTradeTable, ClearHexHighlights, ClearHexHighlightStyle, ClearUnitMoveHexRange, ConnectedToNetworkHost, DisplayMovementIndicator, DontRecordCommandStreams, DragCamera, EndCombatSim, EndGameShow, EndTurnBlockingChanged, EndTurnTimerUpdate, EndUnitMoveHexRange, Event_ToggleTradeRouteDisplay, EventOpenOptionsScreen, EventPoliciesDirty, ExitToMainMenu, FloodplainCreated, FrontEndPopup, GameMessageChat, GameOptionsChanged, GameplayAlertMessage, GameplayFX, GameplaySetActivePlayer, GameViewTypeChanged, GenericWorldAnchor, GlobalUnitScale, GoldenAgeEnded, GoldenAgeStarted, GoToPediaHomePage, GraphicsOptionsChanged, GreatWallCreated, HexFOWStateChanged, HexYieldMightHaveChanged, InitCityRangeStrike, InterfaceModeChanged, KeyUpEvent, LandmarkLibrarySwap, LanguageChanging, Leaderboard_ScoresDownloaded, LeavingLeaderViewMode, LoadScreenClose, LocalMachineAppUpdate, LocalMachineUnitPositionChanged, MarshCreated, MarshRemoved, MinimapClickedEvent, MinimapTextureBroadcastEvent, MultiplayerConnectionComplete, MultiplayerConnectionFailed, MultiplayerGameAbandoned, MultiplayerGameHostMigration, MultiplayerGameInvite, MultiplayerGameLastPlayer, MultiplayerGameLaunched, MultiplayerGameListClear, MultiplayerGameListComplete, MultiplayerGameListUpdated, MultiplayerGamePlayerDisconnected, MultiplayerGamePlayerUpdated, MultiplayerHotJoinCompleted, MultiplayerHotJoinStarted, MultiplayerJoinRoomAttempt, MultiplayerJoinRoomComplete, MultiplayerJoinRoomFailed, MultiplayerPingTimesChanged, MultiplayerProfileDisconnected, MultiplayerProfileFailed, NaturalWonderRevealed, NewGameTurn, NotificationActivated, NotificationAdded, NotificationRemoved, NotifyAILeaderInGame, OpenInfoCorner, OpenPlayerDealScreenEvent, ParticleEffectReloadRequested, ParticleEffectStatsRequested, ParticleEffectStatsResponse, PlayerChoseToLoadGame, PlayerChoseToLoadMap, PlayerVersionMismatchEvent, PreGameDirty, RandomSeedSet, RecordCommandStreams, RemotePlayerTurnEnd, RemotePlayerTurnStart, RemoveAllArrowsEvent, RequestYieldDisplay, RestartGame, RiverTileAdded, RunCombatSim, SearchForPediaEntry, SequenceGameInitComplete, SerialEventBuildingSizeChanged, SerialEventCameraBack, SerialEventCameraCenter, SerialEventCameraForward, SerialEventCameraIn, SerialEventCameraLeft, SerialEventCameraOut, SerialEventCameraRight, SerialEventCameraSetCenterAndZoom, SerialEventCameraStartMovingBack, SerialEventCameraStartMovingForward, SerialEventCameraStartMovingLeft, SerialEventCameraStartMovingRight, SerialEventCameraStopMovingBack, SerialEventCameraStopMovingForward, SerialEventCameraStopMovingLeft, SerialEventCameraStopMovingRight, SerialEventCityCaptured, SerialEventCityContinentChanged, SerialEventCityCreated, SerialEventCityCultureChanged, SerialEventCityDestroyed, SerialEventCityHexHighlightDirty, SerialEventCityInfoDirty, SerialEventCityPopulationChanged, SerialEventCityScreenDirty, SerialEventCitySetDamage, SerialEventDawnOfManHide, SerialEventDawnOfManShow, SerialEventEndTurnDirty, SerialEventEnterCityScreen, SerialEventEraChanged, SerialEventEspionageScreenDirty, SerialEventExitCityScreen, SerialEventFeatureCreated, SerialEventFeatureDestroyed, SerialEventForestCreated, SerialEventForestRemoved, SerialEventGameDataDirty, SerialEventGameInitFinished, SerialEventGameMessagePopup, SerialEventGameMessagePopupProcessed, SerialEventGameMessagePopupShown, SerialEventGreatWorksScreenDirty, SerialEventHexCultureChanged, SerialEventHexDeSelected, SerialEventHexGridOff, SerialEventHexGridOn, SerialEventHexHighlight, SerialEventHexSelected, SerialEventImprovementCreated, SerialEventImprovementDestroyed, SerialEventImprovementIconCreated, SerialEventImprovementIconDestroyed, SerialEventInfoPaneDirty, SerialEventJungleCreated, SerialEventJungleRemoved, SerialEventLeaderToggleDebugCam, SerialEventLeagueScreenDirty, SerialEventMouseOverHex, SerialEventQueueFlushed, SerialEventRawResourceCreated, SerialEventRawResourceDestroyed, SerialEventRawResourceIconCreated, SerialEventRawResourceIconDestroyed, SerialEventResearchDirty, SerialEventRoadCreated, SerialEventRoadDestroyed, SerialEventScoreDirty, SerialEventScreenShot, SerialEventScreenShotTaken, SerialEventStartGame, SerialEventTerrainDecalCreated, SerialEventTerrainOverlayMod, SerialEventTest, SerialEventTestAnimations, SerialEventToggleBridgeState, SerialEventTurnTimerDirty, SerialEventUnitCreated, SerialEventUnitDestroyed, SerialEventUnitDestroyedInCombat, SerialEventUnitFacingChanged, SerialEventUnitFlagSelected, SerialEventUnitInfoDirty, SerialEventUnitMove, SerialEventUnitMoveToHexes, SerialEventUnitSetDamage, SerialEventUnitTeleportedToHex, SerialEventUpdateAllHexCulture, ShowAttackTargets, ShowHexYield, ShowMovementRange, ShowPlayerChangeUI, SpawnArrowEvent, SpecialTileAdded, SpecificCityInfoDirty, StartUnitMoveHexRange, StateMachineDumpStates, StateMachineRequestStates, StrategicViewStateChanged, SystemUpdateUI, TaskListUpdate, TeamMet, TechAcquired, ToggleDisplayUnits, ToolTipEvent, UIPathFinderUpdate, UnitActionChanged, UnitDataEdited, UnitDataRequested, UnitDebugFSM, UnitEmbark, UnitFlagUpdated, UnitGarrison, UnitHandleCreated, UnitHexHighlight, UnitLibrarySwap, UnitMarkThreatening, UnitMemberCombatStateChanged, UnitMemberCombatTargetChanged, UnitMemberOverlayAdd, UnitMemberOverlayMessage, UnitMemberOverlayRemove, UnitMemberOverlayShowHide, UnitMemberOverlayTargetColor, UnitMemberPositionChanged, UnitMoveQueueChanged, UnitNameChanged, UnitResetAnimationState, UnitSelectionChanged, UnitSelectionCleared, UnitShouldDimFlag, UnitStateChangeDetected, UnitTypeChanged, UnitVisibilityChanged, UserRequestClose, VisibilityUpdated, WallCreated, WarStateChanged, WonderCreated, WonderEdited, WonderRemoved, WonderStateChanged, WonderTogglePlacement, WonderTypeChanged, WorldMouseOver

 

[DarkScythe]

A large part of my Civ's abilities depends on the data stored within that table, so it is quite important that I have a function run a check to see what its current state is.

Thus far, I haven't had any issue so long as -- as you say -- I don't open up that database file while testing my code changes. But knowing there exists a possibility kind of makes me paranoid and want to code in at least -some- checks for that scenario, so that at least the user or I would know what's going on when reviewing logs.

Right now, my problem scenario is where I started a brand new game with the database open, and then the game proceeds to load all the old data from my previous game, which it really shouldn't be doing.

That's why my init function will scan for data and issue a "new reset" via gT = {}, though my concern here is that if the db isn't writable.. this still wouldn't do anything. I also am reading that the "best" way to empty a table, if I have other references to it (I don't think I do, but perhaps there will come a time when I will) is to use a for loop and manually set each element it finds to nil. This probably also would have no effect if the database is not writable.

I do wish you luck in figuring out a more elegant, non-DLL mod way of catching savegame events.. It definitely would be a lot easier, and maintain wider compatibility.

That said, that is one heck of a list of Events there, though most of them don't seem to have anything to do with saving a game.

For what it's worth, I've tested SerialEventExitCityScreen for other things, and that seems to fire twice for some reason, every time I exit the city info screen. I think I also tested SerialEventCityScreenDirty, and could not figure out when it fired, if at all.

 

[Pazyryk]

Right now, my problem scenario is where I started a brand new game with the database open, and then the game proceeds to load all the old data from my previous game, which it really shouldn't be doing.

I'm not sure why you're paranoid about this particular problem. It is only something you are likely to do (as modder) and not something a mod user will ever do. I have done that myself, but I'm pretty sure the last time was early 2012. If it worries you still, find an independent way to determine load versus new, and then throw an error if this and the db method disagree. Don't try to recover by some fancy code cure. Just quit Civ5, quit the DB viewer, and restart Civ5. (There really is no other reason for this to happen.)

That's why my init function will scan for data and issue a "new reset" via gT = {}, though my concern here is that if the db isn't writable.. this still wouldn't do anything. I also am reading that the "best" way to empty a table, if I have other references to it (I don't think I do, but perhaps there will come a time when I will) is to use a for loop and manually set each element it finds to nil. This probably also would have no effect if the database is not writable.

Your cure might work, but I think it is more dangerous than the disease. You are in some very grey area involving Lua automatic garbage collection and TableSaverLoader code that I wrote 3 yrs ago (and have largely forgotten).

Just to get into the technical side of it, deleting each table in gT manually will make no difference, and never will in Lua (no matter who told you otherwise). Lua will garbage collect unreferenced tables without your help. So gT={} is exactly the same as gT=nil;gT={}, which is exactly the same as for k,v in pairs(gT) do gT[k]=nil end;gT=nil;gT= {}, and so on (you could delete the elements of the elements of gT if you wanted). This is a waste of energy since Lua will automatically garbage collect the old table and any tables that were nested in that old table if there are no other reference to them. Well,... except there is another reference to them stored by TableSaverLoader if you ever ran TableSave (and your loop nilling won't affect that). I think that TableSaverLoader will recognize that these old tables are no longer reference by gT (the "new one" that you sent in the TableSave), delete references to the old gT and all nested tables (allowing Lua to auto-garbage collect them) ... and then everything goes on fine, even if you create new tables that happen to have the same names. But that's an airplane I wouldn't fly myself, even though I built it.

That said, that is one heck of a list of Events there, though most of them don't seem to have anything to do with saving a game.

I'm most hopeful about AIProcessingEndedForPlayer. If that fires after the barb player has done all of its unit moves, then we would be very close to the regular autosave. I'll test that one out to see.

 

[DarkScythe]

Well.. my thought was "maybe another user who's a modder decides to somehow load this while checking the savagame database for some reason." xD

That said, I'll leave my basic checking in place for now since it doesn't really hurt anything.
I wasn't quite aware how your code worked internally, though I thought I was working with a local Lua table once it had loaded data from the database, and figured I would manipulate it as such. Seems I forgot that it has to propagate back over to your code at some point for the TableSave() operation.

Do let me know how that test goes!
If not, I think I can whip up a quick test as well.

 

[Pazyryk]

Well.. my thought was "maybe another user who's a modder decides to somehow load this while checking the savagame database for some reason." xD

Then they will report it as bug. But you will look at their Lua.log and tell them exactly what happened.

, though I thought I was working with a local Lua table once it had loaded data from the database, and figured I would manipulate it as such.

Lua tables are never local or global. Neither are functions or anything else pointed to by a Lua variable. It's only the reference that's local or global. So, for example:

local GameInfoTypes = GameInfoTypes

There's only one table. It happens to be referenced by a global variable in each Lua state called "GameInfoTypes". There are actually many global "GameInfoTypes", one in each Lua UI state ... but only one table. In the code above, I am putting a reference to that table in an entirely new local variable called "GameInfoTypes". You could do the exact same thing with "print". (And then your print statements would work 0.001 seconds faster than they do already. Which would be utterly pointless. But not pointless for some other function you are doing 1000s of times.)

In other words, don't confuse a horse with the name "horse". They are not the same thing.

 

[DarkScythe]

Ahh. I am still quite new to Lua, so I appreciate the time you take to help explain.

On that note, I never quite understood the concept of declaring a function into a variable. I guess I can't wrap my head around how, or why it works. By that, I mean the standard Lua functions such as print().

My mod does output thousands of lines of debug messages, but only when they are turned on, at which point my debug function simply calls print() for every line that I've "labeled" as debug with the debug function. I'm not sure I'll see all that significant of a gain, though.

 

[Pazyryk]

Anything that basically looks like text (i.e., not an operator like +) is either an identifier, a reserved word, or really just a text string (in quotes or used as a table key). There's only a small number of reserved words (ModBuddy shows them in blue) and print is not one of them. Neither are pairs and a lot of other stuff you see. Those are all just identifiers. All functions and tables in Lua are really anonymous. They can be referenced by a local identifier, a global identifier, a table key, or nothing at all (e.g., {} or function() print("hello" end). All the stuff that Firaxis gives you like Players, GameInfo, MapModData and so on are just identifiers. There's nothing privileged about them. You can reuse those names for something else if you like, or assign the thing held by them (a table or function) to a local identifier (of the same or different name) for some speed boost.

Local identifiers are faster because they (the pointer they hold?) are kept in the processor register (registry?). Lua can do this by limiting the number of local variables that you can access in any function to 60. A file can define >60, but a function in that file can only access 60. (If you violate this you get a Syntax Error on file load.) Global identifiers are on the stack somewhere and take longer to find.

You can localize functions held in tables libraries or referencing a static object (such as Map or Game) too. E.g.:
local floor = math.floor
local PlotDistance = Map.PlotDistance
Those kind of localizations can really help if you are running them on 1000s of plots every turn.

 

[DarkScythe]

Ah, I see.

I don't quite know all the specifics here, but I have been in the habit of locally declaring most everything I needed to use in any function. It just felt cleaner to me, rather than stick game API calls in the middle of large functions and if blocks, etc.

While I have the chance here, I've been meaning to ask this as well:
Thus far, in my code, I've handled one type of plot by storing them into the table/database as long as you held control of it, and there's a function to scan the table each turn and remove plots that you no longer control. Otherwise, so long as you control it, any changes are simply a change to one of the innermost fields of the nested table structure I have.

I've since added logic to handle another type of plot, but for some reason I've done it a different way: instead of holding the plot as long as you control it, I delete the entry/row as soon as it no longer qualifies for whatever bonus it's supposed to have.

My question is, is it faster for me to just leave all the plot entries/rows in the table, and simply modify the innermost values repeatedly, or to remove and re-add those rows repeatedly?

 

[Pazyryk]

It'll be faster to leave something in place (with value 0 or false or an empty table) then to delete and add frequently.


Good news! Events.AIProcessingEndedForPlayer works! It fires after barb combat but before the standard autosave occurs (assuming player leaves PostTurnAutosaves = 0 in config file). Code:

local function OnAIProcessingEndedForPlayer(iPlayer)
	print("OnAIProcessingEndedForPlayer", iPlayer)
	if iPlayer == 63 then
		TableSave(gT, "Ea")
	end
end
Events.AIProcessingEndedForPlayer.Add(OnAIProcessingEndedForPlayer)

I think that will finally solve the regular autosave problem (the "AutoSave_Initial_0000 BC-4000.Civ5Save" has a different solution in the OP).

The other puzzle I want to solve is a way to trigger TableSave when player goes to the in-game menu (which they have to do to quicksave or to get to Game Save menu without cntr-S), and to do this without modding front-end UI files (the menu file itself) because there are some potential problems with that. If I can solve that, then we'll have a pretty safe and easy Lua-only solution.

 

[DarkScythe]

Ahh, I was afraid of that, though I kind of thought similarly which was why I asked, so that I could confirm my suspicions.

It's kind of annoying though, since I'll have to rewrite some of my functions and redo the logic to handle the change, but it shouldn't be too bad, since I already have everything in place to handle the actual plot changes.

That said, that is very good news. Does that event only fire after barb combat, or after combat for any player? (Or for the human, I guess after pressing "End turn" and right before passing it off to the next player?)

I'm also not sure if this is meant to replace or complement the current functions to save at the end of a barb turn, and your newly suggested TableSave() on LoadScreenClose.

I am rooting for you though, being able to intercept saves without modding SaveMenu.lua will be great!

 

[Pazyryk]

No, I didn't mean to say it's linked to barb combat. The problem with the OP code is that it fires at the beginning of the barb turn (before the AI moves barb units). But the autosave occurs at the end of the barb turn, so after those moves. So if, for example, you had some code hooked to combat, and it changed something during barb moves that was supposed to be persisted... too bad. But AIProcessingEndedForPlayer solves that. It fires every AI turn (including barb) after they've moved their units.

I've also figure out how to fire a function when the SaveMenu pops up, without modding SaveMenu.lua (this can be added anywhere):

ContextPtr:LookUpControl("/InGame/GameMenu/SaveMenu"):SetShowHideHandler(function(isHide) print(isHide) end)

That's just a test print function there but it prints "false" when this menu opens (whether you got there via Game Menu or cntr-S). So it can be used to drive TableSave. I already know how to intercept other saves, so I just need to put it all together.

 

[Pazyryk]

OK, here's the whole Lua save intercept system re-coded. The good news is that everything can be done in one place in your mod code (no need to replace SaveGame.lua). It looks a little like spaghetti with functions registering each other all over the place. The timing on some of these is important (the ContextPtr's may need to run after entering game depending on how your mod is set up), which is why it looks a little convoluted.

Posted here because not tested yet. Maybe someone will give it a test run? TableSave has some print statements so it's pretty clear when it runs. The test here is whether this code fires it just before any kind of save you can possibly devise.

function OnModLoaded()	--This should run after all your mod code is loaded; in example here it's
		--called at the end of this file, but you should call it at the end of
		--whatever mod file loads last.
	local bNewGame = true
	local DBQuery = Modding.OpenSaveData().Query
	for row in DBQuery("SELECT name FROM sqlite_master WHERE name='Ea_Info'") do
		if row.name then bNewGame = false end	-- presence of Ea_Info tells us that game already in session
	end
	if bNewGame then
		print("Initializiing for new game...")
	else
		print("Initializing for loaded game...")	
		TableLoad(gT, "Ea")
	end

	--My mod has a bunch of init code that fires here. Specifically, it creates a lot of
	--the data that goes into the gT table in a new game.

	TableSave(gT, "Ea")	--runs before the "AutoSave_Initial_0000 BC-4000" autosave and (I think) an autosave that happens
				--immediately on loading a game; there's no harm in TableSave running right after TableLoad					
end

function OnSaveMenuShowHideHandler(isHide)	--runs when player opens Save Menu from Game Menu or by cntr-S; does not replace Save Menu function
	if not isHide then
		TableSave(gT, "Ea")	
	end
end

function QuickSaveIntercept()	--runs when player tries to Quick Save from Game Menu or by F11; replaces base code with own UI.QuickSave() call
	TableSave(gT, "Ea")
	UI.QuickSave()
end

function OnAIProcessingEndedForPlayer(iPlayer)
	if iPlayer == 63 then
		TableSave(gT, "Ea")	--runs on barb turn AFTER barb unit moves (very close to the regular autosave)
	end
end
Events.AIProcessingEndedForPlayer.Add(OnAIProcessingEndedForPlayer)

function InputHandler( uiMsg, wParam, lParam )
	if uiMsg == KeyEvents.KeyDown then
		if wParam == Keys.VK_F11 then
			QuickSaveIntercept()		--F11 Quicksave
        		return true
		end
	end
end

function OnEnterGame()   --Runs when Begin or Countinue Your Journey pressed
	print("Player entering game ...")

	--Setup various intercepts we need
	ContextPtr:SetInputHandler(InputHandler)
	ContextPtr:LookUpControl("/InGame/GameMenu/SaveMenu"):SetShowHideHandler(OnSaveMenuShowHideHandler)
	ContextPtr:LookUpControl("/InGame/GameMenu/QuickSaveButton"):RegisterCallback( Mouse.eLClick, QuickSaveIntercept )
end
Events.LoadScreenClose.Add(OnEnterGame)


OnModLoaded()

 

[DarkScythe]

I can probably toss this in and test in a little bit, though it looks like you have OnModLoaded() running as soon as the Lua file has completely loaded.
Is there any difference between doing it this way, as opposed to hooking it into Events.SequenceGameInitComplete?

Otherwise, besides OnAIProcessingEndedForPlayer(), it looks almost identical to the existing code, only with the addition of a LookUpControl to catch the save menu.

Edit:
I'm comparing these mainly because I try to condense all of my functions into one "primary" function for each game event hook. In essence, I currently have your original "New game check" code (which added itself to LoadScreenClose) combined with my existing set of debug functions which run at the same time, and I've set your code to the top so it runs before any of my debug.

I'm trying to figure out how to restructure everything with your changes.

To illustrate what I mean:

function GameInit()
	TableSaverLoaderInit()
	debugModeCheck()
	debugModList()
	debugPlayerList()
	GameStateInit()
end

Events.LoadScreenClose.Add(GameInit)

 

[Pazyryk]

Yeah, I see what your trying to do.

My code evolved over time to have two init steps. The first init step runs when the last Lua file loads (marked by a comment in code above), and this is the bulk of my init code that might take up to 1 sec. The reason it is there is because it is not noticeable if loading or starting a new game takes 1 sec extra - the player is getting a beer from the fridge anyway. But it is very noticeable if there is a 1 second hang when the player presses "Begin your Journey".

The second init step has only stuff that needs to be delayed for some reason: mostly only the ContextPtr stuff you see above (even this is not necessary depending on how your mod loads).


Btw, version 0.13 of the code itself should be out this weekend. It fixes the issue mentioned here and generally optimizes the code. It has some new restrictions (that are pretty mild). The updated rules will be:

• Your tables must be "tree-like"
• Keys must be integers or strings
• Avoid "$" and "#" anywhere in string keys
• Avoid international or other non-ascii characters in string keys (multi-byte chars may screw up key parsing)
• Don't use "" as a key
• Table values can include any value of type boolean, number, string or table (NOT function, userdata or thread)

"Tree-like" means that branches never reconnect in your table nesting structure. In other words, no table can have >1 reference to it. This rule sort of violates my "arbitrary structure" claim, but it allows me to streamline the code and make the whole thing faster and more robust.

0.13 will more reasonably handle situations like chopping off whole branches - i.e., nilling or overwriting a table that had deeply nested tables. (0.12 could do that too but it was a very slow process.) And it won't be fooled by moving a branch from here to there (the 0.12 bug linked above). 0.13 will treat that situation as "old branch removed" + "new branch created".

 

[DarkScythe]

Well, my Civ's Lua loads as an include() from a main "loader" file, so it seems TableSaverLoader is executing its initial init/load/save routine before moving on to the next print statement that tells me my Lua file loaded. Haha.

That said, my results as I get them.. I've disabled all instances of me manually calling TableSave() in my own functions.
- TSL save triggered before "Begin your journey"
- TSL triggered message "Player entering game.." after pressing "Begin your journey" / no save.
- TSL save not triggered on entering menu (Hitting Esc, or clicking Menu)
- TSL save not triggered on clicking "Load Game" (Unknown if it actually hit the initial autosave, since my tables are initialized as empty in the beginning.)
- TSL save triggered on clicking "Quick Save Game" in Menu
- TSL save triggered on clicking "Save Game" in Menu
- TSL save triggered on pressing F11 for a Quicksave
- TSL save triggered on pressing Ctrl+S to bring up the save window directly
- TSL save triggered at some point after hitting "End Turn" .. assuming that's the Barb one
- TSL save triggered after loading a saved game (Autosave Initial)

..I'm not sure how else one can save, or force this game/code to make a save.

Hope that all helps.

Edit:
Seems I was compiling this list while you were giving your response.

But that still brings me two questions:
1) Is simply having it load when the Lua file loads perceptually different from hooking it into Events.SequenceGameInitComplete? I believe this should fire before LoadScreenClose, so it sounds like it would be running the same time your first init step runs.
2) Could I stick both OnModLoaded() and OnEnterGame() into one function (probably combining them in this case) as I have above?

It seems heavily dependent on "how my mod loads" though I'm not exactly sure how to interpret that..

Edit 2:
I should also note that my game is set to autosave every 1 turn, but I'm pretty sure by default, it was longer than this.

 

[Vicevirtuoso]

• Avoid "$" and "#" anywhere in string keys
• Avoid international or other non-ascii characters in string keys (multi-byte chars may screw up key parsing)

Is ":" (colon) still valid as a character in a key? If not, I may have to stick with v0.12.

And Events.AIProcessingEndedForPlayer was a great find. I'll be implementing that in my mod. Very useful, since my mod does no small amount of stuff during the Barbarians' turn.

 

[Pazyryk]

":" is fine. I reserved $ and # to differentiate string and number keys in the key serialization. All other characters in the ASCII (non-extended) set are good. I'm worried about characters past 127 (the extended set) because they are represented by >1 byte, so might do something funny in the key parsing.


Edit: I've done a lot more testing and things look very good. I'm using a modded dll with a GameSave event that allows me to see the exact timing of all game saves and verify that TableSave runs exactly before it. Just two notes:

All combat functions are complete in the dll for the barb turn before TableSave runs. That means that if you hook something off GameEvents related to combat (either base or a modded dll like Gedemon's combat GameEvents) you are perfectly safe. TableSave runs after that so it will correctly represent persistent data at the time of game save.

However, graphic engine function is a different matter altogether. I think this might be the only thing in Civ5 that is running in a different thread (I don't know that, just guessing). Graphic events can fire off at any time and they sometimes slip between TableSave and the autosave. So that means things like Events.RunCombatSim and Events.EndCombatSim can happen more or less at anytime. So hooking off of those in a way that changes persistent data is very unsafe. There are many warnings about hooking on to those anyway. So this is just one more reason not to. (They are only properly useful for UI, since that is what they are all about anyway.)

 

[Pazyryk]

Good lord 0.13 is fast!:

TableSave time: 0.016000000000076, checksum: 102244, inserts: 5, updates: 97, unchanged: 1687, deletes: 2

That's a typical turn in Éa: about 1800 table items in total (nested at various levels), with almost 100 changing (updates) and just a few new items (inserts) or nilled items (deletes).

IIRC this was something like 0.4 seconds before. Most of that gain was a single logical step where TableSaverLoader was iterating through a table (a Lua representation of the DB) to find an item. But now it just keys it instead. I guess I've learned some coding in the last 3 yrs.

Will have it out soon, with an update on catching saves in Lua. Just doing final tests...

 

[DarkScythe]

Sounds great, thanks!

Hopefully my results earlier were helpful as well.
If things line up well, I can get your updated save catching functions into my mod before I release my next update tonight.

(Hopefully it'll be easy to modify/combine into other functions as well.)

 

[Pazyryk]

Version 0.13 released!

OP also has new and greatly improved code for intercepting game saves. (Not same as above so take a look even if above was working for you. Thanks DarkScythe for help developing this!)

I've also completely re-written the OP to try to make the whole concept of TableSaverLoader more clear.