1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

SaveUtils.lua -- Itemized Data Storage Across Mods

Discussion in 'Civ5 - SDK / LUA' started by Whys, Oct 20, 2010.

  1. Androrc the Orc

    Androrc the Orc Emperor

    Joined:
    Apr 19, 2004
    Messages:
    1,621
    Location:
    Vienna, Austria
    Since the GetScriptData and SetScriptData functions were removed from some instances, SaveUtils may no longer be working.

    But Firaxis has added something much more practical, which it uses in it's scenarios, which is a more direct connection to save data. For those that wish to use the new functions, use this code as an include:

    Code:
    gbLogSaveData = false;
    
    -- Connection to the Modding save data.  Keep one global connection, rather than opening/closing, to speed up access
    g_SaveData = Modding.OpenSaveData();
    g_Properties = nil;
    -------------------------------------------------------------- 
    -- Access the modding database entries through a locally cached table
    function GetPersistentProperty(name)
    	if(g_Properties == nil) then
    		g_Properties = {};
    	end
    	
    	if(g_Properties[name] == nil) then
    		g_Properties[name] = g_SaveData.GetValue(name);
    	end
    	
    	return g_Properties[name];
    end
    --------------------------------------------------------------
    -- Access the modding database entries through a locally cached table
    function SetPersistentProperty(name, value)
    	if(g_Properties == nil) then
    		g_Properties = {};
    	end
    	
    	if (g_Properties[name] ~= value) then
    		g_SaveData.SetValue(name, value);
    		g_Properties[name] = value;
    	end
    end
    
    Here is one example of how to use these GetPersistentProperty and SetPersistentProperty functions:

    Code:
    include( "NewSaveUtils" ); -- use the aforementioned code as an include
    
    ---------------------------------------------------------
    
    function TheHumanGenomeProjectEffects( iPlayer )
    	local player = Players[iPlayer]
    	local team = Teams[player:GetTeam()]
    
    	if (team:GetProjectCount(GameInfo.Projects["PROJECT_THE_HUMAN_GENOME_PROJECT"].ID) > 0 and GetPersistentProperty("TheHumanGenomeProjectApplied") ~= 1) then
    		player:ChangeExtraHappinessPerCity(1)
    		SetPersistentProperty("TheHumanGenomeProjectApplied", 1);
    	end
    end
    
    GameEvents.PlayerDoTurn.Add( TheHumanGenomeProjectEffects )
     
  2. Moaf

    Moaf Warlord

    Joined:
    May 2, 2008
    Messages:
    100
    Location:
    Vienna, Austria
    Hello,
    I've played around a bit with the modding.setvalue, and it seems to be _very_ slow compared to the setscriptdata method (probably because it's a direct db call).
    Is there a way to intercept the save events themselves (e.g. quicksave or normal saving), so that it's possible to call the setvalue only when there is actually a persisting of save data needed?

    BR,
    Moaf
     
  3. ww2commander

    ww2commander Emperor

    Joined:
    Aug 23, 2003
    Messages:
    1,243
    Location:
    Australia
    Apology for the stupid questions up front as I am still trying to learn LUA....

    How does one save a global table using the GetPersistentProperty and SetPersistentProperty functions? Would it be as simple the following to save data:

    Code:
    SetPersistentProperty("GlobalValueTable", gGlobalValueTable);
    I gather this must be the correct approach (from what I understand) as LUA can save a table within tables!
     
  4. Androrc the Orc

    Androrc the Orc Emperor

    Joined:
    Apr 19, 2004
    Messages:
    1,621
    Location:
    Vienna, Austria
    I imagine so, but I haven't tested it.
     
  5. whoward69

    whoward69 DLL Minion

    Joined:
    May 30, 2011
    Messages:
    8,544
    Location:
    Near Portsmouth, UK
    Set/GetPersistentProperty() are wrappers around the underlying Modding database simple name/value methods, so you can only store "simple" values - ie anything that will serialize to and deserialize from a string representation.

    If you want to save information in tables you'll need to explore the Query() method on the Modding databases and create/access your own tables.
     
  6. whoward69

    whoward69 DLL Minion

    Joined:
    May 30, 2011
    Messages:
    8,544
    Location:
    Near Portsmouth, UK
    Code:
    g_SaveData = Modding.OpenSaveData();
    g_SaveData.SetValue("test", {x=255, y=0, z=0, w=255});
    [COLOR="Red"]Runtime Error: [string "g_SaveData.SetValue("test", {x=255, y=0, z=..."]:1: bad argument #2 to 'SetValue' (Must be of type boolean, nil, number, or string)
    stack traceback:
    	[C]: in function 'SetValue'
    	[string "g_SaveData.SetValue("test", {x=255, y=0, z=..."]:1: in main chunk[/COLOR]
    
     
  7. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    9,823
    Location:
    France
    It seemed also very slow to me, that's why I use saveutils for tables and Modding.OpenSaveData() for simple values I don't need often.

    As serialization of large table was taking some time even with saveutils, I also used something to catch most of the save events in R.E.D. WWII (rerouting the key shortcut for quicksave and "hijacking" the normal save menu) if you want an example.
     
  8. ww2commander

    ww2commander Emperor

    Joined:
    Aug 23, 2003
    Messages:
    1,243
    Location:
    Australia
    This questions is probably better directed at Gedemon:

    Being confused about something I never really understood to begin with (!!!)........Did the patch/G&K break the saveutils used in scenarios like R.E.D WWII or am I confusing two separate ways of saving data?
     
  9. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    9,823
    Location:
    France
    SaveUtils mod was able to save on Players, Plots, Units. I use Players and Plots (the latter not in R.E.D.), that's why R.E.D. WWII was not broken even if the units script data has been removed (but if they remove Players and Plots script data, I'm toast)
     
  10. Androrc the Orc

    Androrc the Orc Emperor

    Joined:
    Apr 19, 2004
    Messages:
    1,621
    Location:
    Vienna, Austria
    Oh, excellent that you can confirm that it still works on players!
     
  11. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    9,823
    Location:
    France
    Yes it still works on players and plots. I didn't test on units, I've no mods using that one.
     
  12. ww2commander

    ww2commander Emperor

    Joined:
    Aug 23, 2003
    Messages:
    1,243
    Location:
    Australia
    Just to make sure I understand your response correctly, the following snippet from your RED WWII mod should still work fine.

    I ask as I use this code in my scenario, but have never understood the black magic that occurs behind it!

    Code:
    function LoadAllTable()
    	Dprint("-------------------------------------")
    	Dprint("Loading data tables ...")	
    	g_UnitData = LoadUnitData()
    	g_UnitStatistic = LoadUnitStatistic()
    	g_ReinforcementData = LoadReinforcementData()
    	g_TrackAllCombats = LoadTrackAllCombats()
    end
    function SaveAllTable()
    	Dprint("-------------------------------------")
    	Dprint("Saving data table ...")
    	SaveUnitData( g_UnitData )
    	SaveUnitStatistic( g_UnitStatistic )
    	SaveReinforcementData( g_ReinforcementData )
    	SaveTrackAllCombats( g_TrackAllCombats )
     
  13. Gedemon

    Gedemon Modder Super Moderator

    Joined:
    Oct 4, 2004
    Messages:
    9,823
    Location:
    France
    Yes, will work, but I've changed it since to save one big table instead of multiple table (save times), check corresponding code of actual mod.

    And see LoadUnitData() in utils, it just call the real magic from SaveUtils.lua, but don't ask me what's going on there, that I do not know, except that I had to kill a chicken before writing the include(), it's very important to make the mod work. I pretty sure it's thanks to the chicken that script data was removed by something else than Players if you ask me.
     
  14. ww2commander

    ww2commander Emperor

    Joined:
    Aug 23, 2003
    Messages:
    1,243
    Location:
    Australia
    Really gives new meaning to the phrase 'the blind leading the blind'!

    I guess as long as it keeps working then no further chickens must be sacrificed to the Firaxis gods of change!
     
  15. Pazyryk

    Pazyryk Deity

    Joined:
    Jun 13, 2008
    Messages:
    3,584
    One DB transaction takes a good fraction of a second, whether you happen to be doing 1 or 10000 inserts in that single transaction. That's why my TableSaverLoader component wraps all of your changes into one (or just a couple) transactions at game save.
     
  16. DonQuiche

    DonQuiche Emperor

    Joined:
    May 23, 2011
    Messages:
    1,122
    A nice trick I thought about when writing an article on the wiki on data persistence: an easy way to store tables (for convenience and speed since there is now only one value to store), is to use loadstring on deserialization. Basically we store an lua code string that will create a table when we load it. I proposed a little function that does the job.

    Also, regarding the "quicksave and save menu hijack" that Gedemon talked about... As he described it it causes conflicts with other mods (IGE for example if the user uses its quick-save or save-then-reload features, or any mod that wants to override the save menu). However, did anyone try to just replace the UI.SaveXXX method?

    I know that objects like units and players are context-bound (the player object you get in context 1 is not the same than you get in context 2, although you can communicate them to other threads and they work fine). However, maybe that static objects like UI are not context-bound? And even if they are, maybe their metatable is not context-bound and can be assigned a new function?

    EDIT: Nevermind, I just tried it: UI is contex-bound and anyway it is a read-only table. And its metatable is a... boolean?!
     
  17. daviiadams

    daviiadams Prince

    Joined:
    Jul 7, 2011
    Messages:
    369
    Location:
    London
    How do you go about saving more than one table? I'm sure this is going on in some mods, but unsure on what the syntax should look like (whether the key should be inside quotation marks etc) as there appears to be a fair few variations within the help in SaveUtils(?)
     
  18. DonQuiche

    DonQuiche Emperor

    Joined:
    May 23, 2011
    Messages:
    1,122
    You put them all in one big table. :D
     
  19. daviiadams

    daviiadams Prince

    Joined:
    Jul 7, 2011
    Messages:
    369
    Location:
    London
    Ah ok, that could prove interesting :crazyeye:
     
  20. DonQuiche

    DonQuiche Emperor

    Joined:
    May 23, 2011
    Messages:
    1,122
    Now there are likely other ways to do it but I am not familiar with SaveUtils and I did not want to bother inspecting its code (especially after those glasses of excellent wine). As a result I opted for a safe and funny answer, probably not the most useful one though. That being said, a single table is likely to offer the best performances if this matters for your mod.
     

Share This Page