1. We have added the ability to collapse/expand forum categories and widgets on forum home.
    Dismiss Notice
  2. Photobucket has changed its policy concerning hotlinking images and now requires an account with a $399.00 annual fee to allow hotlink. More information is available at: this link.
    Dismiss Notice
  3. All Civ avatars are brought back and available for selection in the Avatar Gallery! There are 945 avatars total.
    Dismiss Notice
  4. To make the site more secure, we have installed SSL certificates and enabled HTTPS for both the main site and forums.
    Dismiss Notice
  5. Civ6 is released! Order now! (Amazon US | Amazon UK | Amazon CA | Amazon DE | Amazon FR)
    Dismiss Notice
  6. Dismiss Notice
  7. Forum account upgrades are available for ad-free browsing.
    Dismiss Notice

Storing data across saved games from LUA

Discussion in 'Civ5 - Modding Tutorials & Reference' started by robk, Oct 12, 2010.

  1. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    When I originally wrote this, I only knew about one method of saving data but, as you can see in the thread discussion, there are actually two different methods of saving data for your mod. Both have their advantages and disadvantages and I'll try to outline them below.

    Note (2011-05-23): see this reply about recent additions that allow us to save game data in a SQL database that's embedded in the save file.

    Summary

    The first method, which I figured out first and was the original content of this post, involves saving data to the modUserData SQLite database. Even though it's a sql database underneath the covers, you can only access it through a pair of methods that, essentially, turn it into a key-value pair store. One method, SetValue(key, value), saves a value with a key while the other, GetValue(key), retrieves the data from the store. key should be a string while value is typed as variant in the SQLite table, so it can accept both integers and strings.

    For details, check the modUserData section below.

    modUserData Pros
    • Easily accessible from outside of civ V by using standard sql lite tools. Really nice for debugging.
    • Includes built-in controls to make sure data from different mods don't intefere with each other. It's possible to circumvent this but that would have be done on purpose or through carelessness.
    • Grabbing or saving discrete chunks of data is simple and efficient.
    • When SetValue() is called, the data is immediately written to the SQL database so there are no sync'ing issues or missing data in the event of a crash

    modUserData Cons
    • The scope of the key-value store is global. That is, accessing key from one game is the same as accessing that from another instance of civ V on the same computer. If you're saving something like user options for your mod, this is great but data associated with the actual play of a game will have to include some identifier for that particular instance. (The section on modUserData goes into detail about this)
    • Data is saved in a separate file from the standard saved game file. If you need to do some bug squishing and want someone to send you their game files, you'll have to ask for both the save game file and the modUserData related to your mod.
    • I don't have strong data supporting this but, in my experience, grabbing large amounts of data from the SQL store appears to be somewhat slow. Asking for or saving a single key-value pair takes a trivial amount of time while doing the same for 6000+ pairs takes a significant amount of time (up to 5 seconds or so). That makes sense to me because each call to GetValue() should generate a single SQL statement for one row and initiating 6000 successive SQL calls is fairly expensive.

    The second method for saving data involves using the ScriptData functions associated with various elements in the game. If you look at the Player, Plot and Unit tables in the Tuner, you'll notice that each has a function called SetScriptData(string) and GetScriptData(). The first takes a simple string as an argument while the second returns that last string that was set. So, if you call Players[0].SetScriptData("test") and then immediately call Players[0].GetScriptData(), you'll get "test" returned back to you.

    For details, check the ScriptData section below.

    ScriptData Pros
    • Data is actually saved in the game's save file so there is no need to include identifiers for a particular instance of the game.
    • Community built libraries already exist to make this a useful place to store your data.

    ScriptData Cons
    • Without using a library (which, thankfully, is available), you can only save simple strings.
    • As above, without using a library, there is no built in facility to tie data to a particular mod. If your mod uses ScriptData to save data and another mod does the same, you'll both overwrite each other's data, which makes it pretty useless.
    • Data is tied to a particular object so you have to be careful to keep persistent data in a object that always exists, like Players[0] or Map.GetPlotByIndex(0).



    modUserData Method

    While searching around trying to figure out how to create graphs for historical data, I came across these functions in the tutorial mods.

    PHP:
    modUserData Modding.OpenUserData(modidmodver); 
    modUserData.SetValue(rownamedata);
    modUserData.GetValue(rowname);
    When you create the modUserData object, civ V creates a sqlite file called modid-modver.db in Documents\My Games\Sid Meier's Civilization 5\ModUserData. It only has one table in it, SimpleValues, with two columns: name & Value. When you use SetValue(), rowname is inserted as the Name and data is inserted as the value. Essentially, it acts as a key-value store for your mod.

    Here's an example showing how you can store an retrieve a hypothetical option for a mod that describes how many units to show in some context:

    PHP:
    modver 1;
    modid "awesomeMod";
    modUserData Modding.OpenUserData(modidmodver);
    modUserData.SetValue("options-unitsToShow""32");

    -- 
    Later in your code
    myOptions
    .unitsToShow modUserData.GetValue("options-unitsToShow");

    Unfortunately, or fortunately depending on what you're trying to do, the domain or context of the data in that file is global. That is, any mod from any instance of the game can access it. So if you call SetValue("stuff", "things") from one game and then start up another game, GetValue("stuff") will return the string "things".

    For my Info Addict mod, this isn't good because I needed a way to save data within the context of a single game. That is, if I want the military power of player2 at turn 3, I want to make sure it comes from the current game and not some other game that I just finished playing (nor do I want games overwriting each other). Originally, I came up with the idea of counting up resources on the map to identify a game but that makes Info Addict incompatible with any mods that change resources during the course of a game. It also means that a player who likes static maps wouldn't be able to go back to an old save nor bounce back and forth between different games without causing data corruption.

    Currently, I'm using a two pronged approach. My main data is saved in the modUserDatabase while I'm saving a unique identifier in the save file using the Scriptdata method, outlined next.



    ScriptData Method

    Each plot, player and unit element in the game has two functions associated with it that allow you to save data to the saved game file itself: GetScriptData() and SetScriptData(). At its most basic form, using it looks like this:

    PHP:
    pPlayer Players[0];
    pPlayer:SetScriptData("Here is my saved data");
    data pPlayer:GetScriptData();
    Using it in this manner, this method of saving data has three huge issues:
    • You can only save one string per object.
    • Only one mod can use a saved area without causing data corruption for every mod that wants to save data.
    • There are only have a finite number of objects that are guaranteed to exist in the game. You can be pretty sure that Players[0] always exists, but Map.GetPlotByIndex(4096) probably only exists on the largest maps.

    To mitigate all of these problems, a library has been developed by modders (primarily Whys with contributions from killmeplease, Thalassicus, & Afforess) called SaveUtils. It allows you to save your data to one of those in game objects without deleting data that has been previously stored. Further, it allows you to save complex data structures without having to do the string conversions (serialization) yourself.

    Briefly, using the library looks like this:

    PHP:
    include("SaveUtils");  MY_MOD_NAME="awesomeMod";
    pPlayer Players[0];
    dataKey "datakey";
    save(pPlayerdataKey"Here is my saved data");
    data load(pPlayerdataKey);
    The third arg to save() can be most LUA data structures and conversions take place in the library to save it as a string. When load() is called, the reverse process is done to take the string representation of your data and return it to its original form: be it a string, integer, table, etc. As long as all modders use this library to store their data, there should be no data corruption because the library saves data in a namespace based on the MY_MOD_NAME variable. If a mod uses save(), the existing data for other mods will be preserved and everyone can use the guaranteed to exist objects together, such as Players[0] or Map.GetPloyByIndex(0).

    For further reading and much more detail, check out the main thread and a tutorial written by Whys:

    SaveUtils.lua -- Itemized Data Storage Across Mods

    SaveUtils Tutorial



    An example using both methods

    In my Info Addict mod, I'm using the modUserData method to store the bulk of my data and a single game identifier using SaveUtils. This way, I can store large amounts of data in the SQL database but still access it through successive saves without relying on an inconsistent method (i.e. a hack).

    To store the game identifier, I'm generating a unique ID (a timestamp, actually) and saving it in the saved game file like so:

    PHP:
    function GetGameIdent()

      
    local timeKey "timekey";
      
    local plot Map.GetPlotByIndex(1);
      
    local gameStartTime load(plottimeKey);

      if (
    type(gameStartTime) == "table" or gameStartTime == nilthen
        gameStartTime 
    os.time();
        
    logger:info("Saving new gameStartTime: " .. gameStartTime);
        
    save(plottimeKeygameStartTime);
      
    end;

      return 
    gameStartTime;
    end
    To save data, I have a function that basically does this at the end of each turn (the table alldata is set up prior to this running):

    PHP:
    local modver 1;
    local modid "InfoAddict";
    local modUserData Modding.OpenUserData(modidmodver);

    for 
    piddata in ipairs(alldata) do
      
    local rowname GetGameIdent() .. "-turn" .. turn .. "-player" .. pid
      modUserData
    .SetValue(rownamedata)
    end
    .. and retrieving the data looks like this:

    PHP:
    function getSavedData(turnpid)
      
    local GameIdent GetGameIdent();
      
    local modver 1;
      
    local modid "InfoAddict";
      
    local modUserData Modding.OpenUserData(modidmodver);
      
    local rowname GameIdent .. "-turn" .. turn .. "-player" .. pid;
      
    local data modUserData.GetValue(rowname);
      return 
    data;
    end;
    Note: this isn't the exact code but a simplified version for illustration.

    This isn't the only way to save data but just an example that incorporates both methods. When developing your mod, you should examine what you're saving and determine the best place to save your data. If you decide to use the ScriptData method though, please use SaveUtils so your mod and everyone else's don't clobber each other.
     
  2. The_J

    The_J Say No 2 Net Validations Retired Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    29,858
    Location:
    Germany / Netherlands
    I guess there are some parts missing, right?
    Because if that stays like this, i'll move it to the main forum.
     
  3. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    Yeah, sorry about that J. I hit the summit button a little prematurely. :blush:
     
  4. Pazyryk

    Pazyryk Chieftain

    Joined:
    Jun 13, 2008
    Messages:
    3,585
    I sure wish there was a better solution. The problem here is that the data isn't in the gamesave file. If someone sends you their gamesave (say, for bug squashing), then it won't find the proper table entry on your computer. Afforess seems to know a solution over here, but he hasn't volunteered any details yet.
     
  5. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    It would be really nice if we could register tables to be saved. Something like:

    local tableData = {a, b, c, d};
    local tableName = "myPreciousData";
    registerSaveData(tableData, tableName);


    Actually, looking at Afforess' code, he's attempting to do just that, except it looks like you can only save one table and you have to convert it to a string before you can save it.

    Given what we have to work with, it's pretty easy to look into that sqlite db file. We would just have to ask people to send us both their save file and the db file associated with the mod. I'm totally with you, however - it would be nice if we had a easy facility to save data from our lua scripts.
     
  6. Pazyryk

    Pazyryk Chieftain

    Joined:
    Jun 13, 2008
    Messages:
    3,585
    This is from the revolutions / dynamic history mod thread, but the comment is relevant here. If I understand correctly, this script data will be saved with the gamesave file. Even more, each individual unit, plot and player can have a separate stored string. Afforess is expanding the method so that it can store tables. But it is quite powerful even as a string. Just use a "|" or somesuch to separate items and you can store as many things as you want about any particular plot, unit or player.

    The db method in OP still has a usefulness for other things. I'll especially be using this for function troubleshooting in my modding.
     
  7. The_J

    The_J Say No 2 Net Validations Retired Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    29,858
    Location:
    Germany / Netherlands
    No problem ;).
    Nice to see that the people are already doing tutorials :)goodjob:). It will sure help all the new people in modding and will speed it a bit up :).
     
  8. alpaca

    alpaca King of Ungulates

    Joined:
    Aug 3, 2006
    Messages:
    2,322
    Wouldn't it be easier to give each game a numeric ID and store a "highest used ID" counter value instead?

    Not that saving in the savegame wouldn't be better anyways :)
     
  9. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    Before I knew about SetScriptData, there wouldn't be a way to do that because the game wouldn't store that numeric ID in any way. I picked the counting resources thing because that would be a constant value across saved games. Now that I know about the ScriptData thing, yes, it would be way better to generate a numeric ID and just save it in the game file (or even use something like os.time() since that's unique to a single system and should rarely collide even if people are sharing save files).

    Earlier, I thought about doing a hybrid thing, where I would generate that resource string, grab a nextID field from the modUserData and then associate the resource string with the gameID to cut down on the data size for each row. But, I was lazy and didn't get to that.

    In a related note, I saw a Game.SetScriptData() function but it returns NYI, "not yet implemented". That's lame because that would be the perfect place to save game wide data. Right now, it looks like you have to piggy back off of player0 or something.

    I'll have to add the ScriptData stuff to the OP since it seems to be a better place to store some info.
     
  10. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    Just had a thought: wouldn't any mod that uses Get/SetScriptData() be incompatible with any other mod that uses those same functions? If I wrote a simple gameID to Player[0]:SetScriptData(), I'd completely wipe out any other data in there.
     
  11. Onni

    Onni Chieftain

    Joined:
    Oct 9, 2010
    Messages:
    82
    Good find robk! :king:

    For me that also seems to be the downside using Get/SetScriptData(). :( What we could do is to agree on policy where each mods only modifies it's own data within [mod_id] [/mod_id] tag and get/set all other data in there as they were?

    There is also one loop-hole using that non-strategic-resource-id. It only identifies a single unique game, not save games within it. What if a player decides to reload a game from an earlier save instead of the latest one?

    To me the unique id which would fit for every purpose would be one that is tied to, both the game, and also to the last loaded save game. This could propably be accomplish by modifying core files "LoadMenu.lua" and "SaveMenu.lua". Currently the last loaded game info is erased in LoadMenu.lua. What we could do is to store a unique id's for both the game and save game which are tied to the save game creation time(unique). Like this:
    Code:
    [CURRENT] = {datetime = "2010-10-24-14-50", gameId = 1, saveId = 2}
    [1] = {datetime = "2010-10-24-13-45", gameId = 1, saveId = 1}
    [2] = {datetime = "2010-10-24-14-50", gameId = 1, saveId = 2}
    
    So after that any game that needs to get the current games unique id would just use this command:
    Code:
    local modUserData = Modding.OpenUserData("LoadSaveMenuModId", 1);
    local gameId = modUserData.GetValue("CURRENT");
     
  12. Onni

    Onni Chieftain

    Joined:
    Oct 9, 2010
    Messages:
    82
    Just figured that there is a unique id for each map(=game): Network.GetMapRandSeed()

    Using that and perhaps turn counter we could make a compromise solution for even the save game files? Then we wouldn't need any core files changes nor Get/SetScriptData policies. ...Unless we are playing on a fixed map? ;)
     
  13. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    Even better, if someone wrote a common library to do just that we'd be sitting pretty. 'Course, I could see massive amounts of data being stored in a single string causing problems.

    For me, that wasn't a big deal. If someone restarts from an earlier save, the "future" data gets overwritten as the new turns get played. Of course, it causes a problem if someone goes back to an earlier save and then skips back to a save that is in the future of the same game. I'm just going to say that behavior is unsupported right now :mischief:
     
  14. robk

    robk Chieftain

    Joined:
    Sep 8, 2006
    Messages:
    210
    Location:
    Santa Monica, CA
    Neat!

    Aw nuts, I didn't think of that. That kinda removes the possibility of playing more than one game using a fixed map at a time with my mod.
     
  15. Afforess

    Afforess The White Wizard

    Joined:
    Jul 31, 2007
    Messages:
    12,239
    Location:
    Austin, Texas
    I see my named mentioned several times in the thread, and people seem to want my expertise (such as it is...). Please PM me, so I notice quicker than happenstance searching. ;)

    That sounds like an excellent idea, and I am up to the challenge. I'll see what I can do this weekend. Ideally, you would should be able to call a function with an ID of your choosing, unique to your mod (or save data, or whatever), and a table or string, and save it, and then retrieve it with just your ID; Without clashing with everyone else's mod.
     
  16. Duha

    Duha Chieftain

    Joined:
    Jul 13, 2010
    Messages:
    18
    Location:
    Saint-Petersburg, Russia
    IMHO game has such fucntion:
    Code:
    modUserData = Modding.OpenUserData([B]modid, modver[/B]); 
    As I can sugest it creates new table for each modid, modver.
     
  17. Afforess

    Afforess The White Wizard

    Joined:
    Jul 31, 2007
    Messages:
    12,239
    Location:
    Austin, Texas
    The OP said it didn't save the data in-between saves. It just put's it in the SQL table.
     
  18. Onni

    Onni Chieftain

    Joined:
    Oct 9, 2010
    Messages:
    82
    It saves the data into a global SQL table that can be accessed from any save game. You can also save and retrieve your data using row keys as opposite to a single string.
     
  19. Afforess

    Afforess The White Wizard

    Joined:
    Jul 31, 2007
    Messages:
    12,239
    Location:
    Austin, Texas
    But it doesn't store it in the save. It's basically the same as storing it in a text file on the computer. It's fine and dandy if you only have 1 computer, and never share saves...
     
  20. Duha

    Duha Chieftain

    Joined:
    Jul 13, 2010
    Messages:
    18
    Location:
    Saint-Petersburg, Russia
    And how you want to resolve it?
    I dont think that appending such data to save file is good idea.

    You can save additional data to same dir as civ5 saves with same name and different extension. Some games stored their saves in that way.
     

Share This Page