ShareData.lua -- share references across mods

Yeah that sounds great. One question though. At the moment since any changes are being serialised at the end of the turn, saving is taken care of automatically. If I'm just sharing the data how do I always make sure that the data is in the plot scriptdata before a save occurs.

 

Can you post your core file?

 

I'm not in a hurry to to be honest. would rather wait until the mod is a bit more complete / I've tidied up a bit, if only for personal pride.

 

I don't know what this is or how it works, but i'de imagine it's complex and has taken alot of thought. You deserve credit. =D

...End random post...

 

I understand, it's just hard for me to visualize in detail what you mean. I guess what I'm saying is, let me partner with you on this and I'll do the work. I can't think of a better mod currentlly in development for establishing the ShareData standard. Your religion mod will spread like the gospel.

 

Not doing it for the credit and tho it is complex, the point is it makes things simple for anyone using it, while at the same time opens new doors to mod creation and mod inter-operation. If you build a mod, you'll want to use these files and these three functions: save(), load(), and sometimes share(). These three functions do exactly what you would expect them to do.

 

I updated the OP with a fairly straight forward example.

 

Version 2.

InGame.xml no longer required!

Just share your globals in a shareData() function and attach it to Events.LoadScreenClose.

 

Hi, here's a re-formatted and re-commented version. I'll look at writing a tutorial later. How do you usually add ShareData? I load it as an InGameUIAddin, is this how it's supposed to be done?

Code:

-- vymdt.01.2010.11.20.0000
-- Created by: Ryan F. Mercer -Open source
--===========================================================================
-- ShareData.lua
--===========================================================================
--[[
	Manages shared variables for different mods. ShareData uses the LuaEvent system to set and get variable values that are supposed to be shared between lua contexts.
	
	You can store variables with just a key, which will be written to the global context, or you can store them with your own context key. 
	For example, if your mod is called MyMod and you want to share variables in a namespace called "MyMod", you should supply "MyMod" as context.

	It is highly recommended to use the more user-friendly share() function from SaveUtils to share data. Just add ShareData as an InGameUIAddin and it will be accessible in your whole mod.
]]
--===========================================================================
--[[
Global Variables.
]]
g_super    = {};
g_context  = {};
--===========================================================================
--[[
	Finds out if a variable has been shared by a mod.
	Arguments:
		key: String or number. The key for the shared variable
		tbl: Table. The table to write the return value to.
		context: String. The key of the context table the variable is stored in. Defaults to global context if not given
	Returns:
		Doesn't really return anything but appends a "true" value to tbl if the key is already shared and a false value otherwise
]]
function onHasShared( key, tbl, context )
	-- do type checking in case people supply bad arguments
	local t = type( key );
	if not (t == "number" or (t == "string" and key ~= "")) then
		print( "onHasShared(): Invalid first argument (key) of type "..t..", expected number, or unempty-string." );
		return false; --error.
	end
	
	local t = type( tbl );
	if not t == "table" then
		print( "onHasShared(): Invalid second argument (tbl) of type "..t..", expected table." );
		return false; --error.
	end
	
	local t = type( context );
	if not (t == "nil" or (t == "string" and context ~= "")) then
		print( "onHasShared(): Invalid third argument (context) of type "..t..", expected nil, or unempty-string." );
		return false; --error.
	end
	-- end type checking
	
	-- if no context is supplied, default to the global context
	if context == nil then
		if g_super[key] ~= nil then 
			table.insert( tbl, true );
		else 
			table.insert( tbl, false );
		end
	else
		if type( g_context[context] ) == "table" and g_context[context][key] ~= nil then 
			table.insert( tbl, true );
		else 
			table.insert( tbl, false );
		end
	end
end

LuaEvents.HasShared.Add( onHasShared );
--===========================================================================
--[[
	Sets the value of a shared variable. If desired, this function also writes the current value of a shared variable into the return table.
	Arguments:
		key: String or number. The key for the shared variable. If nil, a numeric key will be automatically chosen
		value: number, string, boolean or table. The value of the shared variable. 
		context: Table. Optional. The context to store this variable in. Defaults to global context if not given
		tbl: Table. Optional. The table to write the current value of the shared variable to, if any.
	Returns:
		Doesn't really return anything but inserts the shared value into tbl[key] if tbl is given and the key is already shared
]]
function onSetShared( key, value, context, tbl )
	-- do type checking in case people supply bad arguments
	local t = type( key );
	if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
		print( "onSetSession(): Invalid first argument (key) of type "..t..", expected nil, number, or unempty-string." );
		return false; --error.
	end
	
	local t = type( value );
	if not (t ~= "function" and t ~= "userdata" and t ~= "thread") then
		print( "onSetSession(): Invalid second argument (value) of type "..t..", expected nil, number, string, boolean, or table." );
		return false; --error.
	end
	local t = type( context );
		if not (t == "nil" or (t == "string" and context ~= "")) then
		print( "onSetSession(): Invalid third argument (context) of type "..t..", expected nil, or unempty-string." );
	return false; --error.
	end
	
	local t = type( tbl );
	if not (t == "nil" or t == "table") then
		print( "onSetSession(): Invalid fourth argument (tbl) of type "..t..", expected nil, or table." );
		return false; --error.
	end
	-- end type checking
	
	-- if no context is supplied, default to the global context
	if context ~= nil then
		g_context[context] = g_context[context] or {};
		if key ~= nil then
			if tbl ~= nil then 
				tbl[key] = g_context[context][key]; 
			end
			g_context[context][key] = value;
		else 
			table.insert( g_context[context], value ); --automatic integer key.
		end
	else
		if key ~= nil then
			if tbl ~= nil then 
				tbl[key] = g_super[key]; 
			end
			g_super[key] = value; 
		else 
			table.insert( g_super, value ); --automatic integer key.
		end
	end
end
LuaEvents.SetShared.Add( onSetShared );
--===========================================================================
--[[
	Returns the value of a shared variable.
	Arguments:
		key: String, number or nil. The key of the shared variable. If nil, this will return all variables in the given context.
		tbl: Table. The table to write the return value to
		context: String or bool true. If string, the key of the context table the variable is stored in. If true, the function returns the a copy of all context tables instead. Defaults to the global context if nil.
	Returns:
		Doesn't really return anything but inserts the shared value into tbl[key] if tbl is given and the key is shared. A special case exists if context == true, which will insert the table containing all the shared contexts into tbl.
]]
function onGetShared( key, tbl, context )
-- do type checking in case people supply bad arguments
	local t = type( key );
	if not (t == "nil" or t == "number" or (t == "string" and key ~= "")) then
		print( "onGetSession(): Invalid first argument (key) of type "..t..", expected nil, number, or unempty-string." );
		return false; --error.
	end
	
	local t = type( tbl );
	if not t == "table" then
		print( "onGetSession(): Invalid second argument (tbl) of type "..t..", expected table." );
		return false; --error.
	end
	
	local t = type( context );
	if not (t == "nil" or (t == "string" and context ~= "") or context == true) then
		print( "onGetSession(): Invalid third argument (context) of type "..t..", expected nil, unempty-string, or boolean true" );
		return false; --error.
	end
	-- end type checking
	
	-- default to the global context if none is given
	if context ~= nil then
		-- check special case: If context is true, return a copy of all shared contexts.
		if context ~= true then
			if key ~= nil then
				if type( g_context[context] ) == "table" and g_context[context][key] ~= nil then
					tbl[key] = g_context[context][key];
				end
			else 
				for k,v in pairs( g_context[context] ) do 
					tbl[k] = v; 
				end
			end
		else 
			for k,v in pairs( g_context ) do 
				tbl[k] = v; 
			end
		end
	else
		if key ~= nil then
			if g_super[key] ~= nil then 
				tbl[key] = g_super[key]; 
			end
		else 
			for k,v in pairs( g_super ) do 
				tbl[k] = v; 
			end
		end
	end
end
LuaEvents.GetShared.Add( onGetShared );
--===========================================================================
--END ShareData.lua
--===========================================================================
-- Created by: Ryan F. Mercer -Open source

 

That's fine, but I only collapse code like that when it never needs to be looked at or rewritten. ShareData is really so basic in its design, there isn't much point in picking apart the code or function specific documentation, and none of the functions are intended to be called directly. <shrug> Use SaveUtils. It provides the functions you should be using. Everything you need to know about ShareData functionality is documented there. ShareData simply needs to be a separate file for a unique lua state accessible by lua events. This thread exists primarily as reference for SaveUtils.

I apologize if it was confusing. Thank you for helping others to understand it.

Adding it as an InGameUIAddin is ideal, but if you wish to share data from global scope then you will need to add ShareData as a LuaContext entry near the top of InGame.xml.

 

Well, the thing is that if you want me to use something as a blackbox I need a pretty good reason for it. So what happened to me was: I looked at SaveUtils because I needed saving, not because I needed to pass things to another context. I used it for saving things, and when I wanted to use something from another context it simply was easier to me to add the variable to Player because it was a player-specific thing.

Where's the incentive in using your share function for this? Only for other people. But there's no good tutorial telling me what I have to do and how it works, and I honestly didn't understand what share did without looking at what ShareData does. Maybe it's just me but I have trouble understanding your documentation for the share part of save utils:

Code:

Shared data can be of any type, including an object or function reference.
Data shared as super-global requires key be unique for all lua states. Data
shared as context-global only requires key be unique for that context.  Share
data as super-global when you want all lua states to act on that same data,
and share data as context-global when you want to make it accessible to any
lua state that specifies your context (mostly for read only).  The context
should be the name of the LuaContext entry or UIAddin file name without file
extension.

It is recommended that data be shared upon Events.LoadScreenClose() to ensure
synchronized order of operations across concurrent mods.  For example,
SaveUtils automatically shares cache and cache-state in the following manner.

When reading this for the first time: What does it mean to share things as super-globals? What does context-global mean? What the heck is a context in this context? Looking into the function didn't help, either, because I didn't know how ShareData works so I couldn't understand it.

I still don't understand why you should share on LoadScreenClose instead of, say, when the context is loaded. Could you explain this again?

It's fair enough to say that ShareData shouldn't even have to be understood by people but I didn't have the impression that that was the intent. Since you opened another thread for it, I assumed it was meant to be stand-alone and I simply had no idea what you're supposed to do with it. So I ignored it, as simple as that

 

I see what you are saying, and I thank you for taking the time to point it out. Keep in mind that ShareData is relatively new and I still need to update a couple of my other threads, so yeah, I was intentionally treating it as a black box. I like to make these things available for people to use in a stand alone manner, but at the same time I wish to encourage others to use SaveUtils, rather than ShareData as a stand alone. It was my hope that people would simply trust this logically minor extension to the SaveUtils standard. I assumed anyone who didn't trust it, would have little trouble understanding the code itself. Perhaps that is overly idealistic.

As to sharing on LoadScreenClose, this deals with execution order and the InGame.xml vs. InGameUIAddin. It is a fairly complex issue when explained in detail, so no, my documentation doesn't go into depth on it. Nor is it helped by the fact that several of the built-in lua files for civ aren't optimally designed for concurrent interoperability, requiring the individual modder to either reinvent the wheel (rewrite the built-in file), or use an InGame.xml edit rather than an InGameUIAddin. The advantage to InGameUIAddins is that they don't require an InGame.xml edit, thus can be run concurrently without merging the InGame.xml file.

Ideally, data should be shared on LoadScreenClose as a synchronization point for concurrent mods. The issue is if you try to call share() from a lua file's global scope, then that code will be executed immediately when the files are being loaded one by one into the system. That means the code might execute before another file has loaded, resulting in possible failure. This can be resolved by hard coding the execution order with an InGame.xml edit, but there is no reliable means for controlling the execution order of InGameUIAddins being run concurrently as separate mods. By placing the share() statement within LoadScreenClose (the defacto game init for modders), then all files will already be loaded, including InGameUIAddins, when data is shared, making the load order irrelevant.

I do appreciate your efforts to help and clearly I've overlooked the importance of explaining ShareData in detail. Once I've updated my other threads, I'll version ShareData with your recommendations.

 

I actually think it might be advantageous to treat it clearly as a part of SaveUtils and offer a combined download of both, along with installation instructions and well-commented usage examples.

It is certainly possible to trust the share function as a black box, and I'm probably unusual in that I usually try to understand how things work (maybe it's a physicists' problem?) but to do that you need to feel like you know what you're doing at least on the layer you're working with. And that means you have to know how to install it, and how to use the share function and the save/load mechanism in simple terminology.

Your tendency seems to be to assume that everyone who tries to create Lua code is a software developer or exposed to coding on a professional level. I have no idea about most programming lingo so obviously I have trouble understanding it. However, if you want to establish a standard for modders it's important to understand that a lot of modders are just smart kids (although sometimes quite old smart kids) who want to fool around a bit with their game, rather than professionals who know what they're doing. A standard should be for everyone

I'm unsure about the OnLoadScreenClose, to be honest. There are a number of problems with it: One is that it's just weird if you click on "Begin your journey" and the game takes a minute to load. Much more natural to extend the loading screen time by writing things directly into Lua context code. Do you not still have issues because the order of OnLoadScreenClose is not guaranteed? So mod1 could share something that mod2 tries to load but the OLSC event of mod2 is executed first? It still depends on activation order and which context is executed first, doesn't it?

Also, what happens if two mods have SharedData. In which context is the shared value stored?

If you don't want to do it yourself I can try writing some kind of short tutorial about saving and data sharing using your scripts and post it in the tutorials section. It's probably not really necessary for saving because there's simply no good way to do it except for SaveUtils but you need to "sell" ShareData a bit better I think because it's not really clear what the advantage is.

 

Oh boy... one at a time. :]

First, I fully agree that this shouldn't be only for professionals, but that's Civ's failing far more than mine. What I'm not sure you fully appreciate is that all of us in this community walked into this as it is and are still very much in the middle of figuring out the base code and optimizing mod interoperability. Civ has done next to nothing for us in this matter. They want the modding community to improve on their rough draft and that is what we are doing. But even my hard to understand documentation in ShareData is more informative than what they originally gave us to work with.

When I first started, it was my hope that modding would be a manageable endeavor for "smart kids", as you put it. But the gaps in operational logic left behind by the game developers soon had the most "professional" of us delving into our more complex programming backgrounds to make optimal solutions available to everyone. Unfortunately, there is still much work to be done, we aren't getting paid to do it, and now it's the holidays. User friendliness is simply lower on the agenda than hard functionality and not everything is "there" yet.

For my own part, I've chosen to focus on the back end tools for the purpose of developing modding standards for interoperability, performance, and ease of use. Killmeplease, Thalassicus, and Afforess, first outlined the need for a tool like SaveUtils and had designed a basic serialize and deserialize process. Later I advanced their ideas and rewrote the serialize and deserialize to make it more robust. But the point of the matter is that just the deserialize method alone represents about 30 hours of work to create and refine. I wish I could say a "smart kid" could have done it, but probably not. It was a significant operational gap left by Civ for us semi-professionals to fill in ourselves. Why it was left to the modding community to hash out a serialization standard for themselves, only the game developers could tell you.

Unfortunately, the list goes on. ShareData is but one small part of this and tho I understand your critique, it hasn't made things more difficult to understand and use, but rather easier. Yes, this is easier than what we were given, which was next to nothing. With luck, it will eventually be easy enough for any "smart kid" to use.

Events.LoadScreenClose()

While it is true that operations placed on the same event will execute according to load order, the point is to not place order dependent operations within the same event. It makes sense to share data upon LoadScreenClose because that is the first sensible opportunity to do anything after all files have loaded. If something requires that all files not only be loaded, but that all files already have shared data, then you would place that logic on an event that occurs after LoadScreenClose, such as ActivePlayerTurnStart. So if we know all files are loaded before data is shared, and we know all data is shared before we use shared data, any logic placed on ActivePlayerTurnStart won't care about file load order.

I agree ShareData needs selling, and I've thought about creating just such a tutorial. I simply haven't gotten around to it and if you wish to, then I welcome that. I would appreciate the opportunity to give it a final check before publishing it however. Just to ensure accuracy and completeness.

 

I'm uncertain what you are asking.

 

I am not trying to demean your work and I fully appreciate the time spent on it, but I think for projects like this, which are meant to be used by other people, simple-to-understand usage instructions are at least as important as the code itself. I'm not saying SaveUtils is diffcult to use, it isn't, but it's difficult to find that out

Well, if you have two mods that define ShareUtils as a InGameUIAddin, I think that this will create two contexts, not one, due to the way they are indexed by InGame.lua (by numeric key). This probably wouldn't create problems but it would store every value twice.

 

Hmm I don't think I'll get around to this very soon. Just to let you know so you don't rely on me posting one in the next days. I think the tutorial should tell people how they can use SaveUtils to save their mod data, and create an example of how to use share(), for example by setting something in an InGameUIAddin that should be displayed in the UI, say the TopPanel

 

What a coincidence. Now do understand?

Incorrect. Only one file for each file name is ever loaded. This is why InGame.xml files have to be merged when used. Otherwise they fight and only one wins. Works perfectly to my knowledge, but if you actually want to do some testing, be my guest.

 

I understood very well before, but that doesn't mean I can't prod.

The community might be better served by a beginner Lua tutorial that, among other things, talks about how to use SaveUtils and ShareData. This, however, takes a lot of time to do so I can't finish it in just a few days.