1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

ShareData.lua -- share references across mods

Discussion in 'Civ5 - SDK / LUA' started by Whys, Nov 17, 2010.

  1. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Version 1 (build: 2010.11.20.0000)

    Allows all lua states and concurrent mods to share the same globals, including table, object, and function references.

    Test Example: (mod 1 of 2 running concurrently)
    Spoiler :
    Code:
    include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_1of2";
    include( "WhysUtils" );
    
    pReference = nil;
    value = "1of2";
    
    function shareData()
      pReference = share( "ShareTest.pReference", pReference or {["init"]="First!"} );
    end
    Events.LoadScreenClose.Add( shareData );
    
    function test()
      pReference["test"] = 1;
      pReference["add"] = true;
      pReference["function"] = sharedFunction;
      LuaEvents.test1of2Complete();
    end
    Events.ActivePlayerTurnStart.Add( test );
    
    function sharedFunction()
      return "sharedFunction(): value = "..value;
    end
    


    Test Example: (mod 2 of 2 running concurrently)
    Spoiler :
    Code:
    include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_2of2";
    include( "WhysUtils" );
    
    pReference = nil;
    value = "2of2";
    
    function shareData()
      pReference = share( "ShareTest.pReference", pReference or {["init"]="Second."} );
    end
    Events.LoadScreenClose.Add( shareData );
    
    function test()
      pReference["test"] = 2;
      pReference["add"] = nil;
      print( pReference["function"]() );
    end
    LuaEvents.test1of2Complete.Add( test );
    

    Result:
    Code:
    -- upon Events.LoadScreenClose.
     pReference = {"init"="First!"}
    
    -- upon Events.ActivePlayerTurnStart.
     pReference = {"test"=1,"init"="First!"}
     pReference = {"test"=1,"add"=true",init"="First!"}
     pReference = {"test"=1,"function"=function: 488751D8,"add"=true,"init"="First!"}
    
    -- upon LuaEvents.test1of2Complete.
     pReference = {"test"=2,"function"=function: 488751D8,"add"=true,"init"="First!"}
     pReference = {"test"=2,"function"=function: 488751D8,"init"="First!"}
     sharedFunction(): value = 1of2
    
    This result demonstrates that the shared pReference used by both mods is in fact the very same pReference. Any actions performed on the shared reference by any mod takes immediate effect for all other mods also sharing the reference. The example also demonstrates that the same is true for function references. Mod 1 of 2 places a reference to a local function in the shared table. That function is then called by mod 2 of 2, without the need for a LuaEvent, and unlike a LuaEvent, even gives a return.

    This code also demonstrates ensured execution order of operations across mods without an InGame.xml file. Take note that "init" can equal "First!" or "Second." depending on which order the files load. This is not a bug, but rather intentionally demonstrates that all other code executes as desired regardless of load order.

    ShareData.lua
    Spoiler :
    Code:
    -- vymdt.01.2010.11.20.0000
    -- Created by: Ryan F. Mercer -Open source
    --===========================================================================
    -- ShareData.lua
    --===========================================================================
    --[[
    Manages custom super and context globals accessible to all lua states.
    ]]
    --===========================================================================
    --[[
    Global Variables.
    ]]
    g_super    = {};
    g_context  = {};
    --===========================================================================
    --[[
    Adds boolean case for given key for given context to automatic integer-key
    for given table.  Adds boolean case for given key for global context to
    automatic integer-key for given table when context not given.
    ]]
    function onHasShared( key, tbl, context )
      local t = type( key );
      if not (t == "number" or (t == "string" and key ~= "")) then
        print( "onHasSession(): Invalid first argument of type "..t
            ..", expected number, or unempty-string." );
        return false; --error.
      end
      local t = type( tbl );
      if not t == "table" then
        print( "onHasSession(): Invalid second argument of type "..t
            ..", expected table." );
        return false; --error.
      end
      local t = type( context );
      if not (t == "nil" or (t == "string" and context ~= "")) then
        print( "onHasSession(): Invalid third argument of type "..t
            ..", expected nil, or unempty-string." );
        return false; --error.
      end
      if context == nil then
        if g_super[key] ~= nil then table.insert( tbl, true );
        else table.insert( tbl, false );
        end
      else
        if type( g_context[context] ) == "table" and
            g_context[context][key] ~= nil then table.insert( tbl, true );
        else table.insert( tbl, false );
        end
      end
    end
    LuaEvents.HasShared.Add( onHasShared );
    --===========================================================================
    --[[
    Sets given value to given key for given context.  Sets given value to given
    key for global context when context not given.  Sets given value to automatic
    integer-key when key not given.  Can optionally add previous value for given
    key to given key for given table, providing remove and replace functionality.
    ]]
    function onSetShared( key, value, context, tbl )
      local t = type( key );
      if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
        print( "onSetSession(): Invalid first argument of type "..t
            ..", expected nil, number, or unempty-string." );
        return false; --error.
      end
      local t = type( value );
      if not (t ~= "function" and t ~= "userdata" and t ~= "thread") then
        print( "onSetSession(): Invalid second argument of type "..t
            ..", expected nil, number, string, boolean, or table." );
        return false; --error.
      end
      local t = type( context );
      if not (t == "nil" or (t == "string" and context ~= "")) then
        print( "onSetSession(): Invalid third argument of type "..t
            ..", expected nil, or unempty-string." );
        return false; --error.
      end
      local t = type( tbl );
      if not (t == "nil" or t == "table") then
        print( "onSetSession(): Invalid fourth argument of type "..t
            ..", expected nil, or table." );
        return false; --error.
      end
      if context ~= nil then
        g_context[context] = g_context[context] or {};
        if key ~= nil then
          if tbl ~= nil then tbl[key] = g_context[context][key]; end
          g_context[context][key] = value;
        else table.insert( g_context[context], value ); --automatic integer key.
        end
      else
        if key ~= nil then
          if tbl ~= nil then tbl[key] = g_super[key]; end
          g_super[key] = value; 
        else table.insert( g_super, value ); --automatic integer key.
        end
      end
    end
    LuaEvents.SetShared.Add( onSetShared );
    --===========================================================================
    --[[
    Adds value for given key for given context to given key for given table.
    Adds value for given key for global context to given key for given table when
    context not given.  Adds all values to keys for given context to given table
    when key not given.  Adds all values to keys for global context to given
    table when key and context not given.  Adds entire context table to context
    key for given table for all contexts when given context boolean true.
    ]]
    function onGetShared( key, tbl, context )
      local t = type( key );
      if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
        print( "onGetSession(): Invalid first argument of type "..t
            ..", expected nil, number, or unempty-string." );
        return false; --error.
      end
      local t = type( tbl );
      if not t == "table" then
        print( "onGetSession(): Invalid second argument of type "..t
            ..", expected table." );
        return false; --error.
      end
      local t = type( context );
      if not (t == "nil" or (t == "string" and context ~= "")
          or context == true) then
        print( "onGetSession(): Invalid third argument of type "..t
            ..", expected nil, unempty-string, or boolean true" );
        return false; --error.
      end
      if context ~= nil then
        if context ~= true then
          if key ~= nil then
            if type( g_context[context] ) == "table" and
                g_context[context][key] ~= nil then
              tbl[key] = g_context[context][key];
            end
          else for k,v in pairs( g_context[context] ) do tbl[k] = v; end
          end
        else for k,v in pairs( g_context ) do tbl[k] = v; end
        end
      else
        if key ~= nil then
          if g_super[key] ~= nil then tbl[key] = g_super[key]; end
        else for k,v in pairs( g_super ) do tbl[k] = v; end
        end
      end
    end
    LuaEvents.GetShared.Add( onGetShared );
    --===========================================================================
    --END ShareData.lua
    --===========================================================================
    -- Created by: Ryan F. Mercer -Open source
    
     
  2. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Demonstration mods are available thru the in-game mod screen by the names of:

    Share Test (1 of 2) -- Demo
    Share Test (2 of 2) -- Demo


    They are intended to be run concurrently and demonstrate a shared object reference and a shared function reference.
    *Look in SaveUtils.lua for documentation of share functionality.

    ShareTest_1of2
    Spoiler :
    Code:
    -- vymdt.02.2010.11.20.0000
    -- Created by: Whys Alives -Open source
    --===========================================================================
    -- Test_1of2.lua
    --===========================================================================
    --[[
    Uses lua console to test accessibility of shared globals via ShareData.lua.
    Intended to be run concurrently with Share Test (2 of 2).  Shares table
    reference on Events.LoadScreenClose.  Also demonstrates ensured execution
    order of operations across mods without an InGame.xml.
    
    *Look in SaveUtils.lua for full documentation on share functionality.
    ]]
    include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_1of2";
    include( "WhysUtils" );
    --===========================================================================
    --[[
    Shared Globals.
    ]]
    pReference = nil;
    value = "1of2";
    --===========================================================================
    --[[
    Shares reference.
    ]]
    function shareData()
      print( "shareData()..." );
      pReference = share( "ShareTest.pReference",  pReference or {["init"]="First!"} );
      print( "pReference = "..out( pReference ) );
    end
    Events.LoadScreenClose.Add( shareData );
    --===========================================================================
    --[[
    ...
    ]]
    function test()
    	print( "start..." );
    	print( "pReference = "..out( pReference ) );
    
    	print( "set \"test\" = 1" );
    	pReference["test"] = 1;
    	print( "pReference = "..out( pReference ) );
    
    	print( "add \"added\" = true" );
    	pReference["add"] = true;
    	print( "pReference = "..out( pReference ) );
    
    	print( "add \"function\" = sharedFunction" );
    	pReference["function"] = sharedFunction;
    	print( "pReference = "..out( pReference ) );
    
      LuaEvents.test1of2Complete();
    end
    Events.ActivePlayerTurnStart.Add( test );
    --===========================================================================
    --[[
    ...
    ]]
    function sharedFunction()
    	return "sharedFunction(): value = "..value;
    end
    --===========================================================================
    --END Test_1of2.lua
    --===========================================================================
    -- Created by: Whys Alives -Open source
    

    ShareTest_2of2
    Spoiler :
    Code:
    -- vymdt.02.2010.11.20.0000
    -- Created by: Whys Alives -Open source
    --===========================================================================
    -- Test_2of2.lua
    --===========================================================================
    --[[
    Uses lua console to test accessibility of shared globals via ShareData.lua.
    Intended to be run concurrently with Share Test (1 of 2).  Shares table
    reference on Events.LoadScreenClose.  Also demonstrates ensured execution
    order of operations across mods without an InGame.xml.
    
    *Look in SaveUtils.lua for full documentation on share functionality.
    ]]
    include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_2of2";
    include( "WhysUtils" );
    --===========================================================================
    --[[
    Shared Globals.
    ]]
    pReference = nil;
    value = "2of2";
    --===========================================================================
    --[[
    Shares reference.
    ]]
    function shareData()
      print( "shareData()..." );
      pReference = share( "ShareTest.pReference",  pReference or {["init"]="Second."} );
      print( "pReference = "..out( pReference ) );
    end
    Events.LoadScreenClose.Add( shareData );
    --===========================================================================
    --[[
    ...
    ]]
    function test()
    	print( "start" );
    	print( "pReference = "..out( pReference ) );
    
    	print( "set \"test\" = 2" );
    	pReference["test"] = 2;
    	print( "pReference = "..out( pReference ) );
    
    	print( "remove \"added\"" );
    	pReference["add"] = nil;
    	print( "pReference = "..out( pReference ) );
    
    	print( "execute shared function" );
    	print( pReference["function"]() );
    end
    LuaEvents.test1of2Complete.Add( test );
    --===========================================================================
    --END Test_2of2.lua
    --===========================================================================
    -- Created by: Whys Alives -Open source
    

    These demo mods use SaveUtils.lua
    You can find it here: http://forums.civfanatics.com/showthread.php?t=392958

    These demo mods use WhysUtils.lua
    Spoiler :
    Code:
    -- vymdt.01.2010.11.17.0000
    -- Created by: Whys Alives -Open source
    --===========================================================================
    -- WhysUtils.lua
    --===========================================================================
    --[[
    Global Functions: len(), out().
    ]]
    --===========================================================================
    --[[
    Returns length of both tables and strings.  Returns false when given boolean,
    function, userdata, or thread.
    ]]
    function len( p )
      local r = 0; local t = type( p );
      if t == "boolean" or t == "function" or t == "userdata" or t == "thread"
          then r = false; print( "len(): Invalid type: "..t ); --error.
      elseif t == "table"  then for k,v in pairs( p ) do r = r +1; end
      elseif t == "string" then r = p:len();
      end
      return r;
    end
    --===========================================================================
    --[[
    Returns a string representation of any given data type.
    ]]
    function out( p )
      local r = ""; local t = type( p );
      if p ~= nil then
        if t ~= "table" then
          if t == "boolean" or t == "number" or t == "function"
              or t == "userdata"  or t == "thread" then
            r = tostring( p );
          else r = '"'..p..'"';
          end
        else
          r = "{"; local b = false;
          for k,v in pairs( p ) do
            if b then r = r..","; end
            r = r..out( k ).."="..out( v );
            b = true;
          end
          r = r.."}"
        end
      end
      return r;
    end
    --===========================================================================
    --END WhysUtils.lua
    --===========================================================================
    -- Created by: Whys Alives -Open source
    
     
  3. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
  4. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
  5. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
  6. smellymummy

    smellymummy King

    Joined:
    Jul 31, 2002
    Messages:
    705
    edit: so i had to post a noobie question first to get my brain wrapped around this ;)

    very nice work whys:goodjob: it's this kind of stuff that shows how far we can push lua
     
  7. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    My one gripe? I have my tabs set to 2 spaces and it's really messing with some of the formatting. Time to go back to using only spaces. :p

    So I only verified function references earlier today, and I am very excited about it. Truly global functions accessible to all mods! :D
     
  8. lemmy101

    lemmy101 Emperor

    Joined:
    Apr 10, 2006
    Messages:
    1,064
    Awesome work Whys! Very awesome! :D

    Annoyingly I came up with my own solution to this problem (after much headache :D) which while likely not as elegant, is too engrained in the code now to actually implement this. But I imagine it will be hugely helpful for a lot of modders, so thanks for your tireless and awesome Lua trickery and pokery :D.
     
  9. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    I do understand lemmy, but keep in mind, I'm pushing this out with SaveUtils. Thus this will likely become somewhat standard for sharing globals across mods. If you don't wish for your mods to be compliant and accessible, that is your choice, but you aren't going to find an easier to use implementation than what I've created here. I don't know what your code looks like, but mine boils down to a single statement wrapped around an object or function.

    local data = share( "this", data ); --too hard? ;)
     
  10. lemmy101

    lemmy101 Emperor

    Joined:
    Apr 10, 2006
    Messages:
    1,064
    In mine I have an ISData class that is automatically pulled out / pushed into the map plot 0,0 whenever the CRC changes, and is accessible anywhere that includes the ISDataCore lua file.

    If it's wrapped in with Save Utils then I will look into adding it then, for compatibilities sake. If it does the serialising stuff so I can call the share function once on ISData.data and it share automatically all the stuff in it, otherwise it would be a *huge* task to add it as I have a mass of data in there for religions, cities, players etc and it's used in a whole bunch of UI lua files.

    Basically at the moment I do this:

    File A
    Code:
    include ("ISPlayer")
    include ("ISReligion")
    include ("ISCity")
    
    local player = ISPlayer:new(Game.GetActivePlayer());
    
    local religion = player:GetStateReligion();
    
    local city = player:GetCapital();
    
    city:ChangeReligionPopulation(religion:GetName(), 1);
    
    

    File B
    Code:
    include ("ISPlayer")
    include ("ISReligion")
    include ("ISCity")
    
    local player = ISPlayer:new(Game.GetActivePlayer());
    
    local religion = player:GetStateReligion();
    
    local city = player:GetCapital();
    
    print(city:GetReligionPopulation(religion:GetName())); -- has new population as soon as A has run.
    
    -- equivalent of....
    
    print(city.Data.Religions[religion.Data.name].Population);
    
    -- equivalent of...
    
    print(ISData.Data.ISCity["0:8192"].Data.Religions[ISData.Data.ISPlayer[0].Data.stateReligion].Population);
    
    
    basically the :new binds the new classes to an entry in the ISData.Data table, e.g. ISData.Data.ISPlayer[0] <- for the above example, either creating it or binding to it based on whether it exists yet or not, and it is this data that gets saved/loaded from the SaveUtils. So I can create these helper classes anywhere and they automatically contain the up to date data for that player / religion / city or whatever, and if I change them in any state it will update them on all other states transparently..

    And to add a new variable I just set them inside the object's Data table and that is now part of the serialized data for that particular city, or player, I don't need to do anything else.

    As long as I can get the same behaviour using your system at the base, I'll look into putting it in at some point soon.
     
  11. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    By the way, I don't know if I'd call this trickery or pokery. :) I'm simply using the LuaEvents as they are intended to be used. But one advantage to what I've created here is that once you share a function, you no longer have to use LuaEvents. Just use the shared function by reference.
     
  12. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    My hope is to make serialization a non-thought. By sharing the cache data across mods, you don't have to serialize and deserialize all the time, but can simply work from cache and then serialize upon user save game. That makes for awesome performance. Just one deserialize on game load and one serialize on game save and you never have to think about it. Just use the shared cache.

    Sigh... I wonder what the save game event is.
     
  13. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    ISData.Data = share( "ISData.Data", ISData.Data );

    I believe this will do it. :)

    Any file you put this in will be using the same global reference to ISData.Data. You can still save as you normally would I believe.
     
  14. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Let me look at your ISDataCore lua file. I'm betting I can have this working for you easy.
     
  15. lemmy101

    lemmy101 Emperor

    Joined:
    Apr 10, 2006
    Messages:
    1,064
    Awesome. :)

    So I could put this directly in the ISDataCore and just include it as I currently do? i.e. the line is identical in all files that share it?
     
  16. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Yeah. But I don't know what ISDataCore looks like. It might make more sense to share the entire ISData or perhaps the ISData.Data from within ISData. My own preference would be the first one.
     
  17. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    You sure you don't just want to use SaveUtils to save your data though? It really might not be that much to implement, just attach ISData.Data to pPlot for 0,0 on call to save(), and then all the data is stored in SaveUtils shared cache. It's shared now, so you don't have to turn off caching.

    You can probably just place the following in your ISData's save function.
    save( pPlot, "ISData.Data", Data );
     
  18. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    Seriously guys, I don't do this for nothing. By using SaveUtils with ShareData, there are only two functions you ever have to think about.

    save() and load().

    These two functions will allow you to keep what ever (serializable) persistant data you want and it is automatically shared between all mods. ALL MODS! Plllleeeeaase, take a close look at what it is I'm doing here. It's a big deal, for all of us.
     
  19. lemmy101

    lemmy101 Emperor

    Joined:
    Apr 10, 2006
    Messages:
    1,064
    I am. :p

    I've just implemented my own way of sharing the data between lua states, but that method still uses SaveUtils to serialize the data so should anything else be in there from another mod using SaveUtils it should work. Will definitely look into implementing your sharing system though, you're right, mod interoperability is important. :D

    Thanks again! I understand how important these kind of tools are. :)
     
  20. Whys

    Whys Between the Lines

    Joined:
    Oct 20, 2007
    Messages:
    456
    I see. But that means you can only share serializable data. share() will allow you to share classes, functions, whatever. Plus you won't take the performance hit from deserializing every time you want a lua state to have the data.
     

Share This Page