Globals and file imports

[Thalassicus]

Okay, so I have a few things imported to the VFS:

• File included in several core game files, with functions that override base API functions (to fix bugs or add new material)
• Need some global variables for those functions

I discovered the hard way that it's not possible to just put the globals in the file they're used, since they get deleted by garbage collection at some point (due to multithreading?)...

So I moved the globals out to a separate file:

• The main file
• Global variables in an initialization file

How do I control the order these files get imported? These files need to be loaded before most other game files (which use them). These globals are only used by the functions in a single file, so I'd prefer to avoid the added overhead from inter-mod variable saving procedures.

I'm somewhat aware this problem is solved in Civ 5 with contexts... if anyone knows a good reference source to learn how contexts work it'd be greatly appreciated if you can give a link. I don't know much about the context subject and tried looking at some other mods to get an idea of how to do this, without finding what I needed (since other mods like InfoAddict that use custom contexts don't override base functions).

 

[alpaca]

Files further up in the list are loaded first if they all have the same content type, I think. Not sure if that's what you're looking for, I don't exactly understand the question.

 

[Thalassicus]

The problem is the init file defining the globals gets loaded after the functions where the globals are used.

The initialization file is first in the file list, but still gets loaded last.

 

[SamBC]

Not sure of a solution, but maybe an explanation... if the other files using the globals are replacements for builtins, those always get loaded fairly immediately, and other files are added as contexts at various points within them... not sure about the implicit UI extension LUA/XML combinations, haven't worked with them.

Also, I have a vague idea that globals aren't readily readable between contexts... Whys might know better, perhaps, as this is the sort of thing ShareUtils helps with...

 

[Sneaks]

I've encountered the same problem recently, trying to make CCMAT II work. With this many files, it becomes an absolute burden to try to get files to load in the proper order.

 

[alpaca]

If I may suggest something to alleviate your burden, check out the Common Lua Events thread. I started implementing some of them and, among other things, it has a content type "ModScript" that allows you to define load order priority in the description field. A higher number means the file will be loaded later. If you like, copy the code and add your files with that method, then make sure if file A depends on file B, the description of file B is an integer lower than that of file A

 

[Sneaks]

For vanilla art overrides, should the gr2 stuff load before the artsefines XML?

 

[Thalassicus]

For mod scripts there's workarounds, and what you're doing is important work on that alpaca. The real problem is when I need these files to reliably load before built-in files that aren't loaded through additions to the content tab.

 

[Thalassicus]

Well, at this point the only feasible solution I know of is including InGame.xml in the project, and putting the init file for the globals above everything else. It's not an ideal solution, but at least it works.

 

[SamBC]

Maybe time for another community standard, for a standard alteration to InGame.xml that adds a new AddIn type for early loading... as I imagine it'll be something that comes up.

 

[Sneaks]

Hmm..

Call a Lua that then calls everybody else to the early load party? Not a bad idea.

Hell, replacing all of the vanilla lines in the InGame with that file might not hurt, adding the capacity for entry points between each.

Assign each vanilla Lua a value (i.e. WorldView = 20, CityView = 30). If I needed to load in before CityView, I give the description a value of 25 for my LoadOrderAddin.

Of course this is all dependent upon the idea that the new LoadOrder.lua would manage to sort then fire off all these files before the game finished going through InGame.

 

[Ryan F. Mercer]

ShareData has been waiting for this party all year!

 

[Thalassicus]

I somehow overlooked a really obvious solution that popped into my head today.

#1
CityWeights = {}

#2
if CityWeights == nil then
    CityWeights = {}
end

#3
MapModData.TBM = {}
MapModData.TBM.CityWeights = {}

#4
if MapModData.TBM == nil then
    MapModData.TBM = {}
    MapModData.TBM.CityWeights = {}
end

I'd attempted #1-3, but somehow overlooked trying #4, which does work. So the solution is to:

• Use a predefined superglobal table like MapModData
• Check if the sub-table is nil before initializing it

This creates 'lazy' initialization of superglobals to be include()'d by multiple files across multiple threads, regardless of load order, and without the need for an init file loaded first.

Here's how I adapted it to my LuaLogger class at the top of a utilities file:

if MapModData.TBM == nil then
    MapModData.TBM = {}
    MapModData.TBM.LoggerType = {}

    print("Initialized MapModData.TBM")
end

LOG_TRACE    = "TRACE"
LOG_DEBUG    = "DEBUG"
LOG_INFO     = "INFO"
LOG_WARN     = "WARN"
LOG_ERROR    = "ERROR"
LOG_FATAL    = "FATAL"

local LEVEL = {
    [LOG_TRACE] = 1,
    [LOG_DEBUG] = 2,
    [LOG_INFO]  = 3,
    [LOG_WARN]  = 4,
    [LOG_ERROR] = 5,
    [LOG_FATAL] = 6,
}

if MapModData.TBM.LoggerType.New == nil then
    function MapModData.TBM.LoggerType:New()
        local logger = {};
        setmetatable(logger, self);
        self.__index = self;

        logger.level = LEVEL.INFO;

        logger.setLevel = function (self, level)
            self.level = level;
        end

        logger.log = function (self, level, message)
            if LEVEL[level] < LEVEL[self.level] then
                return false;
            end
            print(level .. string.rep(" ", 7-level:len()) .. message);
            return true;
        end

        logger.trace = function (logger, message) return logger:log(LOG_TRACE, message) end
        logger.debug = function (logger, message) return logger:log(LOG_DEBUG, message) end
        logger.info  = function (logger, message) return logger:log(LOG_INFO,  message) end
        logger.warn  = function (logger, message) return logger:log(LOG_WARN,  message) end
        logger.error = function (logger, message) return logger:log(LOG_ERROR, message) end
        logger.fatal = function (logger, message) return logger:log(LOG_FATAL, message) end
        return logger;
    end
end

 

[Sneaks]

In your example, if another file that used your logger executed before your LuaLogger file, wouldn't the MapModData.TBM.LoggerType:New() simply not work?

 

[Thalassicus]

If file1 needs to use functions in file2, doesn't file1 always include("file2")? Since the included statements in file2 execute before the remainder of file1, it happens in the proper order.

In an example without any include statements, I have a CityWeights global I needed in a file, so I set it up like this:

if MapModData.TBM.CityWeights == nil then
    MapModData.TBM.CityWeights = {}
end

-- alternatively:
-- MapModData.TBM.CityWeights = MapModData.TBM.CityWeights or {}

function GetPlayerTotalWeight(playerID)
    if playerID == nil then
        logger:error("GetPlayerTotalWeight: Invalid playerID")
        return 1
    end
    MapModData.TBM.CityWeights[playerID] = MapModData.TBM.CityWeights[playerID] or {}
    ...
    ...

 

[Sneaks]

I have been having issues in my CCMAT where included files were not, in fact being loaded first.

 

[Thalassicus]

I'm not sure what circumstances could change this, but normally the lua virtual machine executes included files at the moment of inclusion. I tested this with:

-- UP_General.lua
print("1 Including ThalsUtilities")
include( "ThalsUtilities" )
print("  3 Done ThalsUtilities")

-- ThalsUtilities.lua
print(" 2 Loading ThalsUtilities.lua")

...output...

TechPopup: 1 Including ThalsUtilities
TechPopup:  2 Loading ThalsUtilities.lua
TechPopup:   3 Done ThalsUtilities
TechPopup: 1 Including ThalsUtilities
TechPopup:  2 Loading ThalsUtilities.lua
TechPopup:   3 Done ThalsUtilities
TechPanel: 1 Including ThalsUtilities
TechPanel:  2 Loading ThalsUtilities.lua
TechPanel:   3 Done ThalsUtilities
TechPanel: 1 Including ThalsUtilities
TechPanel:  2 Loading ThalsUtilities.lua
TechPanel:   3 Done ThalsUtilities
DiploList: 1 Including ThalsUtilities
DiploList:  2 Loading ThalsUtilities.lua
DiploList:   3 Done ThalsUtilities
CityView: 1 Including ThalsUtilities
CityView:  2 Loading ThalsUtilities.lua
CityView:   3 Done ThalsUtilities
CityView: 1 Including ThalsUtilities
CityView:  2 Loading ThalsUtilities.lua
CityView:   3 Done ThalsUtilities
TechTree: 1 Including ThalsUtilities
TechTree:  2 Loading ThalsUtilities.lua
TechTree:   3 Done ThalsUtilities
TechTree: 1 Including ThalsUtilities
TechTree:  2 Loading ThalsUtilities.lua
TechTree:   3 Done ThalsUtilities
...

 

[Sneaks]

Perhaps running those files as contexts might be the issue

 

[alpaca]

Contexts might be meant to work in different threads. No-one knows, I guess, except for the devs.

 

[SamBC]

Where one file is included by several others, and those are in different contexts, I believe all first-class instances (tables, functions, etc) are created separately for each context. I'm still getting my head around this context business, but they appear to be partly namespaces and partly (potentially) separate threads.

However, where there's a single instance which is propagated by some cross-context means (such as LuaEvents), that instance seems to be shared by the contexts. Not sure how any potential multi-threading of such is handled, but Lua does a lot of stuff automagically, so it might sync such things in some clever way.