SaveUtils.lua -- Itemized Data Storage Across Mods

Discussion in 'Civ5 - SDK / LUA' started by Whys, Oct 20, 2010.

  1. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Version 8. :]

    Fixed a potentially significant bug in the Save copy constructor. It would have most likely appeared as a cache issue if you ran into it. So if you had problems with the cache before, give it another try.
     
  2. Deep_Blue

    Deep_Blue Knight

    Joined:
    Aug 2, 2005
    Messages:
    750
    When I save data I get a warning message "cache not shared" is this something to worry about?
     
  3. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Really depends. It means you're accessing the cache before it has been shared between all active lua contexts. If you're performing the save() from one lua context, then peforming the load() from another, and the cache isn't shared yet, then you'll be saving the data to one cache and attempting to retrieve it from another. You'll come up empty handed. But so long as you're using the default cache state 1, serialize on save(), the ScriptData itself will still be synchronized and directly retrievable, but with a small cost to performance for the duplicate deserialization. If you're using cache state 2, serialize on sync(), then you'll also need to call sync() explicitly to keep the ScriptData synchronized with the cache data. And if you're using cache state 0, no cache, then I should probably change a line of code somewhere. No cache means no issues, but then, it also means no performance improvement.

    First, the warning itself can be suppressed in the following manner:
    WARN_NOT_SHARED = false; include( "SaveUtils" ); MY_MOD_NAME = "myMod"

    Second, the cache is automatically shared upon LoadScreenClose, but it can also be shared prior to that with an explicit call to share_SaveUtils(). However, this requires that ShareData.lua already be loaded. The most reliable method of doing that is to put ShareData near the top of InGame.xml, but of course, that hurts interoperability... for now :).
     
  4. Deep_Blue

    Deep_Blue Knight

    Joined:
    Aug 2, 2005
    Messages:
    750
    so if I use Save and load in same lua file there will be no problem? Can you give me example when do I need to use cache state 1 and cache state 2? I still do not get how this is supposed to work in practical uses.

    another question if I am including SaveUtils in several lua files do I need to change MY_MOD_NAME for each lua file?

    Edit: one more question: I only added SaveUtils.lua to the project do I have to add other files? you mentioned ShareData.lua
     
  5. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Cache states are available for the modder's convenience. Typically, you just use the default unless you need to debug something, at which point it can be helpful to temporarily turn off the cache. Cache state 2 is never really necessary. It's just thrown in with an eye to the future. Theoretically, if a mod were to become large enough and complex enough, performance enhancement could be gained by taking manual control of ScriptData synchronization.

    In same file, no problem... 100% with 1% margin of error. Hey, anything is possible; you never know 100%. :)

    Change MY_MOD_NAME to be the same in each lua file. Tho technically, it only exists for the modder's convenience as it simply makes it possible to write this...

    data = load( target, key );

    ...instead of this...

    data = load( target, key, "myMod" );

    Either way, the following is always an option as well, but one better left alone. :)

    save( target, key, mischief, "yourMod" ); :D

    ShareData is only necessary if you wish to share the cache for improved performance. Just keep in mind that it can potentially improve performance, even in a single file mod, and you'd never know what you're missing. That's because in some situations, the benefit only increases for each additional mod you run that also uses SaveUtils. So if you only ever run this one mod, then you're probably not going to care too much either way, but if someone else runs severals mods and yours is the only one that isn't sharing the cache, then there is going to be a highly variable performance loss spread across each active mod.

    Hope this helps. You're definitely asking the right questions.
     
  6. Deep_Blue

    Deep_Blue Knight

    Joined:
    Aug 2, 2005
    Messages:
    750
  7. SamBC

    SamBC Emperor

    Joined:
    Feb 2, 2010
    Messages:
    1,052
    Location:
    Lancaster
    This seems as good a place as any to ask if anyone's noticed something... I've seen a report in Another Place that ScriptData on Units isn't working any more. I'm guessing people using SaveUtils are the most likely to notice this. Failing that, I guess I'll post a new thread to mirror the one in that Other Place...

    (wow, it feels really amusing to use parliamentary euphemisms for this purpose...)
     
  8. Valkrionn

    Valkrionn The Hamster King

    Joined:
    May 23, 2008
    Messages:
    14,450
    Location:
    Crestview FL
    IIRC, some of the ScriptData functions NEVER worked. I can't remember if it was units, or if that's new, though.
     
  9. SamBC

    SamBC Emperor

    Joined:
    Feb 2, 2010
    Messages:
    1,052
    Location:
    Lancaster
    My recollection was that they worked for units, plots and players, but had been specced for cities and didn't work. I could easily be wrong.
     
  10. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    You're correct. I don't know about units not working now, but that would make sense as a first step as things transition to the database.
     
  11. SamBC

    SamBC Emperor

    Joined:
    Feb 2, 2010
    Messages:
    1,052
    Location:
    Lancaster
    Being as how I hope to actually get started on some stuff soon, I thought I'd comment on the idea of SaveUtils moving to the new database of mod data... implementation ideas...

    Firstly, I'd suggest SaveUtils having its own table, to prevent any collision with other mods using simple access. It's also worth considering how the SQL structure can be used to simplify association with game objects like cities, plots, players and and units (and any others), for example using columns for their IDs, or separate tables... otherwise, serialisation of complex data structures can work however it does now.

    I guess it's also worth maintaining the API as it is now, for backwards compatibility, but if the new system suggests any new APIs that are more effective or powerful, those could be used as well, in parallel.

    So, those are my thoughts.

    Demand might seem low, but I'd hope that there'll be more drive in the near future as more GameEvents are added, and hopefully some other requests are fulfilled....
     
  12. Valkrionn

    Valkrionn The Hamster King

    Joined:
    May 23, 2008
    Messages:
    14,450
    Location:
    Crestview FL
    I HAVE had confirmation, btw, that full database access is possible. So it should be possible to make your own tables, as well as simple pairs, fairly easily.
     
  13. MouseyPounds

    MouseyPounds Prince

    Joined:
    Nov 8, 2010
    Messages:
    417
    Location:
    Maryland, USA
    Awesome news Valk. I just tried making and accessing a custom table using the Query method and it worked great! :D

    First I an the following in a running game through the Lua Console: (Please ignore the odd variable names; I was just experimenting after all :p)
    Code:
    AAA = Modding.OpenSaveData()
    _ = AAA:Query('CREATE TABLE MyTest("ID" NOT NULL PRIMARY KEY, "A" INTEGER, "B" TEXT)')
    for row in _ do end
    _ = AAA.Query('INSERT INTO MyTest("ID","A","B") VALUES(1,2,"three")')
    for row in _ do end
    
    Then quit civ completely, cleared the cache, reloaded the game and successfully retrieved everything with another console Query:
    Code:
    whee = Modding.OpenSaveData()
    for row in whee.Query("SELECT * FROM MyTest") do print(row.ID, row.A, row.B) end
    
    So SaveUtils can definitely follow Sam's suggestion and implement either a single SaveUtils table or possibly several tables (e.g. SaveUtils_Plot, SaveUtils_Player, etc) behind the scenes.
     
  14. Valkrionn

    Valkrionn The Hamster King

    Joined:
    May 23, 2008
    Messages:
    14,450
    Location:
    Crestview FL
    More useful, IMO, is to set saveutils up such that it automatically creates a separate table for each mod (ensuring lack of any accidental conflict), and possibly even allow modders to create new tables within that table; Not sure that's possible in SQL, in which case it would require some of the same serialization used for the current SaveUtils.
     
  15. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    You can't create a table inside a table with SQL, but you can reference another table from a table-cell and use some kind of continual suffix naming convention for each nested table. Might be able to eliminate serialization entirely, but what would we call it? Databasilization? :D

    When done this way, the end user would never have to think about creating database tables, just create data-type tables and save and load as normal. If done correctly, the old API would still work as intended and legacy code wouldn't require any rewriting.

    MouseyPounds, thank you much, you've given me something to work with. :)

    I should have time to work on these changes over the next few weeks while I wait for my kreisel aquarium to bio-cycle. Breeding inverts is my other hobby. ;)
     
  16. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
  17. Valkrionn

    Valkrionn The Hamster King

    Joined:
    May 23, 2008
    Messages:
    14,450
    Location:
    Crestview FL
  18. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
  19. PAVLOS_GR300

    PAVLOS_GR300 Greek

    Joined:
    Sep 25, 2011
    Messages:
    235
    Location:
    Thessalonici, Makedonia, Hellas
    i am totaly new to civ modding and i want to ask a question.i recently installed a mod(red ww2)when i start a new game everything works fine but when i load a saved game the date and production dont work and also the buildings restrictions.is there any chance that this code will help me to play my saved games and if yes how can i install/use it? heres is my lua when loading and try to choose production


    [2783.026] Initializing Lua 5.1.4
    [2843.882] InstalledPanel: Refreshing Mods
    [2843.929] InstalledPanel: ModID IN ('d565342c-f7b5-495e-9c1d-c05c2d55c8a3', '544d699d-1c84-4606-b22f-a1b009af9471', '580c14eb-9799-4d31-8b14-c2a78931de89')
    [2843.929] InstalledPanel: Refreshing Mods
    [2844.132] InstalledPanel: Refreshing Mods
    [2846.409] InstalledPanel: Refreshing Mods
    [2847.033] InstalledPanel: Refreshing Mods
    [2907.577] CivilopediaScreen: SetSelectedCategory(12)
    [2907.577] CivilopediaScreen: CivilopediaCategory[CategoryTerrain].DisplayList
    [2908.139] CivilopediaScreen: SetSelectedCategory(1)
    [2908.139] CivilopediaScreen: CivilopediaCategory[CategoryHomePage].DisplayList
    [2912.991] Runtime Error: [string "Assets\Tutorial\lua\Tutorial_MainGame.lua"]:3: attempt to index global 'GlobalTutorialInfo' (a nil value)
    [2912.991] Runtime Error: [string "Assets\Tutorial\lua\TutorialEngine.lua"]:39: bad argument #1 to 'count' (table expected, got nil)
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ConfirmImprovementRebuildPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\CityPlotManagementPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ConfirmCommandPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\MinorCivEnterTerritoryPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\LiberateMinorPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ReturnCivilianPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\AnnexCityPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\DeclareWarMovePopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\BarbarianRansomPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ConfirmGiftPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ConfirmCityTaskPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\PuppetCityPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\DeclareWarRangeStrikePopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\ConfirmPolicyBranchPopup.lua
    [2914.894] GenericPopup: Loaded Popup - Assets\UI\InGame\PopupsGeneric\MinorCivGoldPopup.lua
    [2918.919] Runtime Error: [string "Assets/UI/InGame/TopPanel.lua"]:273: attempt to index global 'g_Calendar' (a nil value)
    [2918.919] Runtime Error: Error loading Assets/UI/InGame/TopPanel.lua.
    [2919.589] RedMain: -------------------------------------
    [2919.589] RedMain: R.E.D. World War II script started
    [2919.589] RedMain: -------------------------------------
    [2920.525] TechTree: AdvisorControl could not be found
    [2920.525] Demographics: Dequeuing demographics
    [2920.525] Demographics: Dequeuing demographics
    [3316.643] Runtime Error: [string "Assets/UI/InGame/TopPanel.lua"]:273: attempt to index global 'g_Calendar' (a nil value)
    [3316.659] Runtime Error: [string "Assets/UI/InGame/TopPanel.lua"]:273: attempt to index global 'g_Calendar' (a nil value)
    [3322.041] Runtime Error: [string "C:\Users\Παύλος\Documents\My Games\Si..."]:260: attempt to index global 'g_Unit_Classes' (a nil value)
     
  20. Moriboe

    Moriboe King

    Joined:
    Nov 30, 2010
    Messages:
    659
    Location:
    Belgium
    I'm using SaveUtils to save total city culture. Basically I do "save (pPlayer, xyString, cultureInt)" and "load (pPlayer, xyString)". I notice the following behaviour:
    • When saving a game, returning to menu, reloading: all works fine.
    • When not going back to the menu but loading from the game world: the loaded values reflect the situation just before loading, not the ones when the game was saved.
    Is this normal behaviour or am I doing something wrong?

    edit: just noticed this in the lua log:
    Runtime Error: [string "C:\Users\Erik\Documents\My Games\Sid Meier'..."]:212: attempt to call global 'type' (a nil value)
    I assume the "type" function is in the Lua library. I'm lost now.

    edit2: apparently I can load a variable only once during a turn or the above error is thrown. I save/load a variable in two different lua files, disabling the operation in one of them "solves" the second problem (the first remains). This complicates things a bit for me (different context and timing). Is there a way around?
     

Share This Page