ShareData.lua -- share references across mods

Version 1 (build: 2010.11.20.0000)

Allows all lua states and concurrent mods to share the same globals, including table, object, and function references.

Test Example: (mod 1 of 2 running concurrently)

Spoiler :

Code:

include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_1of2";
include( "WhysUtils" );

pReference = nil;
value = "1of2";

function shareData()
  pReference = share( "ShareTest.pReference", pReference or {["init"]="First!"} );
end
Events.LoadScreenClose.Add( shareData );

function test()
  pReference["test"] = 1;
  pReference["add"] = true;
  pReference["function"] = sharedFunction;
  LuaEvents.test1of2Complete();
end
Events.ActivePlayerTurnStart.Add( test );

function sharedFunction()
  return "sharedFunction(): value = "..value;
end

Test Example: (mod 2 of 2 running concurrently)

Spoiler :

Code:

include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_2of2";
include( "WhysUtils" );

pReference = nil;
value = "2of2";

function shareData()
  pReference = share( "ShareTest.pReference", pReference or {["init"]="Second."} );
end
Events.LoadScreenClose.Add( shareData );

function test()
  pReference["test"] = 2;
  pReference["add"] = nil;
  print( pReference["function"]() );
end
LuaEvents.test1of2Complete.Add( test );

Result:

Code:

-- upon Events.LoadScreenClose.
 pReference = {"init"="First!"}

-- upon Events.ActivePlayerTurnStart.
 pReference = {"test"=1,"init"="First!"}
 pReference = {"test"=1,"add"=true",init"="First!"}
 pReference = {"test"=1,"function"=function: 488751D8,"add"=true,"init"="First!"}

-- upon LuaEvents.test1of2Complete.
 pReference = {"test"=2,"function"=function: 488751D8,"add"=true,"init"="First!"}
 pReference = {"test"=2,"function"=function: 488751D8,"init"="First!"}
 sharedFunction(): value = 1of2

This result demonstrates that the shared pReference used by both mods is in fact the very same pReference. Any actions performed on the shared reference by any mod takes immediate effect for all other mods also sharing the reference. The example also demonstrates that the same is true for function references. Mod 1 of 2 places a reference to a local function in the shared table. That function is then called by mod 2 of 2, without the need for a LuaEvent, and unlike a LuaEvent, even gives a return.

This code also demonstrates ensured execution order of operations across mods without an InGame.xml file. Take note that "init" can equal "First!" or "Second." depending on which order the files load. This is not a bug, but rather intentionally demonstrates that all other code executes as desired regardless of load order.

ShareData.lua

Spoiler :

Code:

-- vymdt.01.2010.11.20.0000
-- Created by: Ryan F. Mercer -Open source
--===========================================================================
-- ShareData.lua
--===========================================================================
--[[
Manages custom super and context globals accessible to all lua states.
]]
--===========================================================================
--[[
Global Variables.
]]
g_super    = {};
g_context  = {};
--===========================================================================
--[[
Adds boolean case for given key for given context to automatic integer-key
for given table.  Adds boolean case for given key for global context to
automatic integer-key for given table when context not given.
]]
function onHasShared( key, tbl, context )
  local t = type( key );
  if not (t == "number" or (t == "string" and key ~= "")) then
    print( "onHasSession(): Invalid first argument of type "..t
        ..", expected number, or unempty-string." );
    return false; --error.
  end
  local t = type( tbl );
  if not t == "table" then
    print( "onHasSession(): Invalid second argument of type "..t
        ..", expected table." );
    return false; --error.
  end
  local t = type( context );
  if not (t == "nil" or (t == "string" and context ~= "")) then
    print( "onHasSession(): Invalid third argument of type "..t
        ..", expected nil, or unempty-string." );
    return false; --error.
  end
  if context == nil then
    if g_super[key] ~= nil then table.insert( tbl, true );
    else table.insert( tbl, false );
    end
  else
    if type( g_context[context] ) == "table" and
        g_context[context][key] ~= nil then table.insert( tbl, true );
    else table.insert( tbl, false );
    end
  end
end
LuaEvents.HasShared.Add( onHasShared );
--===========================================================================
--[[
Sets given value to given key for given context.  Sets given value to given
key for global context when context not given.  Sets given value to automatic
integer-key when key not given.  Can optionally add previous value for given
key to given key for given table, providing remove and replace functionality.
]]
function onSetShared( key, value, context, tbl )
  local t = type( key );
  if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
    print( "onSetSession(): Invalid first argument of type "..t
        ..", expected nil, number, or unempty-string." );
    return false; --error.
  end
  local t = type( value );
  if not (t ~= "function" and t ~= "userdata" and t ~= "thread") then
    print( "onSetSession(): Invalid second argument of type "..t
        ..", expected nil, number, string, boolean, or table." );
    return false; --error.
  end
  local t = type( context );
  if not (t == "nil" or (t == "string" and context ~= "")) then
    print( "onSetSession(): Invalid third argument of type "..t
        ..", expected nil, or unempty-string." );
    return false; --error.
  end
  local t = type( tbl );
  if not (t == "nil" or t == "table") then
    print( "onSetSession(): Invalid fourth argument of type "..t
        ..", expected nil, or table." );
    return false; --error.
  end
  if context ~= nil then
    g_context[context] = g_context[context] or {};
    if key ~= nil then
      if tbl ~= nil then tbl[key] = g_context[context][key]; end
      g_context[context][key] = value;
    else table.insert( g_context[context], value ); --automatic integer key.
    end
  else
    if key ~= nil then
      if tbl ~= nil then tbl[key] = g_super[key]; end
      g_super[key] = value; 
    else table.insert( g_super, value ); --automatic integer key.
    end
  end
end
LuaEvents.SetShared.Add( onSetShared );
--===========================================================================
--[[
Adds value for given key for given context to given key for given table.
Adds value for given key for global context to given key for given table when
context not given.  Adds all values to keys for given context to given table
when key not given.  Adds all values to keys for global context to given
table when key and context not given.  Adds entire context table to context
key for given table for all contexts when given context boolean true.
]]
function onGetShared( key, tbl, context )
  local t = type( key );
  if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
    print( "onGetSession(): Invalid first argument of type "..t
        ..", expected nil, number, or unempty-string." );
    return false; --error.
  end
  local t = type( tbl );
  if not t == "table" then
    print( "onGetSession(): Invalid second argument of type "..t
        ..", expected table." );
    return false; --error.
  end
  local t = type( context );
  if not (t == "nil" or (t == "string" and context ~= "")
      or context == true) then
    print( "onGetSession(): Invalid third argument of type "..t
        ..", expected nil, unempty-string, or boolean true" );
    return false; --error.
  end
  if context ~= nil then
    if context ~= true then
      if key ~= nil then
        if type( g_context[context] ) == "table" and
            g_context[context][key] ~= nil then
          tbl[key] = g_context[context][key];
        end
      else for k,v in pairs( g_context[context] ) do tbl[k] = v; end
      end
    else for k,v in pairs( g_context ) do tbl[k] = v; end
    end
  else
    if key ~= nil then
      if g_super[key] ~= nil then tbl[key] = g_super[key]; end
    else for k,v in pairs( g_super ) do tbl[k] = v; end
    end
  end
end
LuaEvents.GetShared.Add( onGetShared );
--===========================================================================
--END ShareData.lua
--===========================================================================
-- Created by: Ryan F. Mercer -Open source

 

Demonstration mods are available thru the in-game mod screen by the names of:

Share Test (1 of 2) -- Demo
Share Test (2 of 2) -- Demo

They are intended to be run concurrently and demonstrate a shared object reference and a shared function reference.
*Look in SaveUtils.lua for documentation of share functionality.

ShareTest_1of2

Spoiler :

Code:

-- vymdt.02.2010.11.20.0000
-- Created by: Whys Alives -Open source
--===========================================================================
-- Test_1of2.lua
--===========================================================================
--[[
Uses lua console to test accessibility of shared globals via ShareData.lua.
Intended to be run concurrently with Share Test (2 of 2).  Shares table
reference on Events.LoadScreenClose.  Also demonstrates ensured execution
order of operations across mods without an InGame.xml.

*Look in SaveUtils.lua for full documentation on share functionality.
]]
include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_1of2";
include( "WhysUtils" );
--===========================================================================
--[[
Shared Globals.
]]
pReference = nil;
value = "1of2";
--===========================================================================
--[[
Shares reference.
]]
function shareData()
  print( "shareData()..." );
  pReference = share( "ShareTest.pReference",  pReference or {["init"]="First!"} );
  print( "pReference = "..out( pReference ) );
end
Events.LoadScreenClose.Add( shareData );
--===========================================================================
--[[
...
]]
function test()
	print( "start..." );
	print( "pReference = "..out( pReference ) );

	print( "set \"test\" = 1" );
	pReference["test"] = 1;
	print( "pReference = "..out( pReference ) );

	print( "add \"added\" = true" );
	pReference["add"] = true;
	print( "pReference = "..out( pReference ) );

	print( "add \"function\" = sharedFunction" );
	pReference["function"] = sharedFunction;
	print( "pReference = "..out( pReference ) );

  LuaEvents.test1of2Complete();
end
Events.ActivePlayerTurnStart.Add( test );
--===========================================================================
--[[
...
]]
function sharedFunction()
	return "sharedFunction(): value = "..value;
end
--===========================================================================
--END Test_1of2.lua
--===========================================================================
-- Created by: Whys Alives -Open source

ShareTest_2of2

Spoiler :

Code:

-- vymdt.02.2010.11.20.0000
-- Created by: Whys Alives -Open source
--===========================================================================
-- Test_2of2.lua
--===========================================================================
--[[
Uses lua console to test accessibility of shared globals via ShareData.lua.
Intended to be run concurrently with Share Test (1 of 2).  Shares table
reference on Events.LoadScreenClose.  Also demonstrates ensured execution
order of operations across mods without an InGame.xml.

*Look in SaveUtils.lua for full documentation on share functionality.
]]
include( "SaveUtils" ); MY_MOD_NAME = "ShareTest_2of2";
include( "WhysUtils" );
--===========================================================================
--[[
Shared Globals.
]]
pReference = nil;
value = "2of2";
--===========================================================================
--[[
Shares reference.
]]
function shareData()
  print( "shareData()..." );
  pReference = share( "ShareTest.pReference",  pReference or {["init"]="Second."} );
  print( "pReference = "..out( pReference ) );
end
Events.LoadScreenClose.Add( shareData );
--===========================================================================
--[[
...
]]
function test()
	print( "start" );
	print( "pReference = "..out( pReference ) );

	print( "set \"test\" = 2" );
	pReference["test"] = 2;
	print( "pReference = "..out( pReference ) );

	print( "remove \"added\"" );
	pReference["add"] = nil;
	print( "pReference = "..out( pReference ) );

	print( "execute shared function" );
	print( pReference["function"]() );
end
LuaEvents.test1of2Complete.Add( test );
--===========================================================================
--END Test_2of2.lua
--===========================================================================
-- Created by: Whys Alives -Open source

These demo mods use SaveUtils.lua
You can find it here: http://forums.civfanatics.com/showthread.php?t=392958

These demo mods use WhysUtils.lua

Spoiler :

Code:

-- vymdt.01.2010.11.17.0000
-- Created by: Whys Alives -Open source
--===========================================================================
-- WhysUtils.lua
--===========================================================================
--[[
Global Functions: len(), out().
]]
--===========================================================================
--[[
Returns length of both tables and strings.  Returns false when given boolean,
function, userdata, or thread.
]]
function len( p )
  local r = 0; local t = type( p );
  if t == "boolean" or t == "function" or t == "userdata" or t == "thread"
      then r = false; print( "len(): Invalid type: "..t ); --error.
  elseif t == "table"  then for k,v in pairs( p ) do r = r +1; end
  elseif t == "string" then r = p:len();
  end
  return r;
end
--===========================================================================
--[[
Returns a string representation of any given data type.
]]
function out( p )
  local r = ""; local t = type( p );
  if p ~= nil then
    if t ~= "table" then
      if t == "boolean" or t == "number" or t == "function"
          or t == "userdata"  or t == "thread" then
        r = tostring( p );
      else r = '"'..p..'"';
      end
    else
      r = "{"; local b = false;
      for k,v in pairs( p ) do
        if b then r = r..","; end
        r = r..out( k ).."="..out( v );
        b = true;
      end
      r = r.."}"
    end
  end
  return r;
end
--===========================================================================
--END WhysUtils.lua
--===========================================================================
-- Created by: Whys Alives -Open source

 

...reserved...

 

...reserved...

 

...reserved...

 

edit: so i had to post a noobie question first to get my brain wrapped around this

very nice work whys it's this kind of stuff that shows how far we can push lua

 

My one gripe? I have my tabs set to 2 spaces and it's really messing with some of the formatting. Time to go back to using only spaces.

So I only verified function references earlier today, and I am very excited about it. Truly global functions accessible to all mods!

 

Awesome work Whys! Very awesome!

Annoyingly I came up with my own solution to this problem (after much headache ) which while likely not as elegant, is too engrained in the code now to actually implement this. But I imagine it will be hugely helpful for a lot of modders, so thanks for your tireless and awesome Lua trickery and pokery .

 

I do understand lemmy, but keep in mind, I'm pushing this out with SaveUtils. Thus this will likely become somewhat standard for sharing globals across mods. If you don't wish for your mods to be compliant and accessible, that is your choice, but you aren't going to find an easier to use implementation than what I've created here. I don't know what your code looks like, but mine boils down to a single statement wrapped around an object or function.

local data = share( "this", data ); --too hard?

 

In mine I have an ISData class that is automatically pulled out / pushed into the map plot 0,0 whenever the CRC changes, and is accessible anywhere that includes the ISDataCore lua file.

If it's wrapped in with Save Utils then I will look into adding it then, for compatibilities sake. If it does the serialising stuff so I can call the share function once on ISData.data and it share automatically all the stuff in it, otherwise it would be a *huge* task to add it as I have a mass of data in there for religions, cities, players etc and it's used in a whole bunch of UI lua files.

Basically at the moment I do this:

File A

Code:

include ("ISPlayer")
include ("ISReligion")
include ("ISCity")

local player = ISPlayer:new(Game.GetActivePlayer());

local religion = player:GetStateReligion();

local city = player:GetCapital();

city:ChangeReligionPopulation(religion:GetName(), 1);

File B

Code:

include ("ISPlayer")
include ("ISReligion")
include ("ISCity")

local player = ISPlayer:new(Game.GetActivePlayer());

local religion = player:GetStateReligion();

local city = player:GetCapital();

print(city:GetReligionPopulation(religion:GetName())); -- has new population as soon as A has run.

-- equivalent of....

print(city.Data.Religions[religion.Data.name].Population);

-- equivalent of...

print(ISData.Data.ISCity["0:8192"].Data.Religions[ISData.Data.ISPlayer[0].Data.stateReligion].Population);

basically the :new binds the new classes to an entry in the ISData.Data table, e.g. ISData.Data.ISPlayer[0] <- for the above example, either creating it or binding to it based on whether it exists yet or not, and it is this data that gets saved/loaded from the SaveUtils. So I can create these helper classes anywhere and they automatically contain the up to date data for that player / religion / city or whatever, and if I change them in any state it will update them on all other states transparently..

And to add a new variable I just set them inside the object's Data table and that is now part of the serialized data for that particular city, or player, I don't need to do anything else.

As long as I can get the same behaviour using your system at the base, I'll look into putting it in at some point soon.

 

By the way, I don't know if I'd call this trickery or pokery. I'm simply using the LuaEvents as they are intended to be used. But one advantage to what I've created here is that once you share a function, you no longer have to use LuaEvents. Just use the shared function by reference.

 

My hope is to make serialization a non-thought. By sharing the cache data across mods, you don't have to serialize and deserialize all the time, but can simply work from cache and then serialize upon user save game. That makes for awesome performance. Just one deserialize on game load and one serialize on game save and you never have to think about it. Just use the shared cache.

Sigh... I wonder what the save game event is.

 

ISData.Data = share( "ISData.Data", ISData.Data );

I believe this will do it.

Any file you put this in will be using the same global reference to ISData.Data. You can still save as you normally would I believe.

 

Let me look at your ISDataCore lua file. I'm betting I can have this working for you easy.

 

Awesome.

So I could put this directly in the ISDataCore and just include it as I currently do? i.e. the line is identical in all files that share it?

 

Yeah. But I don't know what ISDataCore looks like. It might make more sense to share the entire ISData or perhaps the ISData.Data from within ISData. My own preference would be the first one.

 

You sure you don't just want to use SaveUtils to save your data though? It really might not be that much to implement, just attach ISData.Data to pPlot for 0,0 on call to save(), and then all the data is stored in SaveUtils shared cache. It's shared now, so you don't have to turn off caching.

You can probably just place the following in your ISData's save function.
save( pPlot, "ISData.Data", Data );

 

Seriously guys, I don't do this for nothing. By using SaveUtils with ShareData, there are only two functions you ever have to think about.

save() and load().

These two functions will allow you to keep what ever (serializable) persistant data you want and it is automatically shared between all mods. ALL MODS! Plllleeeeaase, take a close look at what it is I'm doing here. It's a big deal, for all of us.

 

I am.

I've just implemented my own way of sharing the data between lua states, but that method still uses SaveUtils to serialize the data so should anything else be in there from another mod using SaveUtils it should work. Will definitely look into implementing your sharing system though, you're right, mod interoperability is important.

Thanks again! I understand how important these kind of tools are.

 

I see. But that means you can only share serializable data. share() will allow you to share classes, functions, whatever. Plus you won't take the performance hit from deserializing every time you want a lua state to have the data.