Solution for Core Lua files overriding and conflicts discussion

[Vaderkos]

For a long time I was wondering how to fix problems.
I'd be glad hear other opinions.

1. Core Lua files being overriden if at least 2 mods want to modify same file.
2. Mod creators have to explicitly provide lua files compatibility by creating manually merged files. (Community-Patch-DLL contains multiple such files, for example for EUI)

Proposed solution.
1. Create a table in sql (as table modification are loaded at startup)
* Determines entrypoint file for each MOD modified Core Lua File instead of overriding it.
* Describes mod dependencies for other mods
* Describes mod incompatibility
2. Prepare common loader solution that is injected into each core lua file
* It checks table for mods that want to modify specified file
* It loads mods entrypoint in an ordered manner into specified file
* It checks mod conflicts


Diagram describes my idea better than text I think...
And I think I could sponsor it myself

View attachment Lua_WIP1.png

 

[hokath]

Well if you could do this, it would be epic.

For example, every new ideology must modify the social popup and ideology choose lua files. So if I want to provide 3 ideologies, and support all combinations... there will be many lua files to provide the user.

 

[Vaderkos]

@hokath From mod creator point of view the structure of project can be any, the only 2 requirements:
1. globally unique filenames
2. And entrypoint to the desired file.
Within entrypoint you can have any imports you want.

Or I don't get the idea..

For sure base filez should provide some hooks, or bus to inject into processes and verify existing modifications

 

[azum4roll]

Is it possible for a Lua file to modify existing Lua files instead of overriding the whole thing? Assuming correct load order and different file names.

Can we have a proof of concept?

 

[Vaderkos]

Is it possible for a Lua file to modify existing Lua files instead of overriding the whole thing? Assuming correct load order and different file names.

Can we have a proof of concept?

Work In Progress

 

[Vaderkos]

@azum4roll, @hokath I've done POC, it works .

CIV5-Lua-Loader-POC/POC at main · vaderkos/CIV5-Lua-Loader-POC

Contribute to vaderkos/CIV5-Lua-Loader-POC development by creating an account on GitHub.

github.com

To test:
1. Put all 3 mods into your mods directory
2. Load game and select all 3 mods
3. Connect with FireTuner and switch context to ActionInfoPanel.
4. PROFIT both files were loaded into ActionInfoPanel

P.S. ModLoader mod overrides ActionInfoPanel.lua to inject loader code at the end, its actual code is from Community Patch.

 

[hokath]

So in this, and correct me if I'm wrong, the base file to be changed has this "injection statement" (the for loop in the screenshot) at the end, and it calls all the other files of the same name after it has run through the usual 'vanilla' code.

Can you explain how I would use this to do the following basic example:
1) I want ChooseIdeologyPopup.lua (UI code) to display an additional item for Mod A and Mod B
2) To do this the

function RefreshList()

needs to have its local variable appended to so there is one more guy in the table

local ideologies = { [stuff here],
        {
            PolicyBranchType = "POLICY_BRANCH_HERITAGE",
            VictoryTypes = {"DiplomaticVictory", "DominationVictory", "ReligiousVictory"},
            Image = "SocialPoliciesDevotion.dds",
            ImageOffsetX = 0,
            ImageOffsetY = 0,
        }
}

for each mod.

So if I run mod A, the above item is added. If I run mod B, a different item is added.
If both mod A and B run then both items are added (doesn't matter about order).
How do you do this kind of inherited (?) behaviour in lua, using your method?

 

[Vaderkos]

So in this, and correct me if I'm wrong, the base file to be changed has this "injection statement" (the for loop in the screenshot) at the end, and it calls all the other files of the same name after it has run through the usual 'vanilla' code.

Can you explain how I would use this to do the following basic example:
1) I want ChooseIdeologyPopup.lua (UI code) to display an additional item for Mod A and Mod B
2) To do this the

function RefreshList()

needs to have its local variable appended to so there is one more guy in the table

local ideologies = { [stuff here],
        {
            PolicyBranchType = "POLICY_BRANCH_HERITAGE",
            VictoryTypes = {"DiplomaticVictory", "DominationVictory", "ReligiousVictory"},
            Image = "SocialPoliciesDevotion.dds",
            ImageOffsetX = 0,
            ImageOffsetY = 0,
        }
}

for each mod.

So if I run mod A, the above item is added. If I run mod B, a different item is added.
If both mod A and B run then both items are added (doesn't matter about order).
How do you do this kind of inherited (?) behaviour in lua, using your method?

Short answer yes using method.
Long answer below

That's why in theory we'd have to write injection code for lua. Like a bus for actions that can be done within specified file.
In your case you want to add just an item to a list.
We'd do either global list of items so any loaded file can interact with it, or IMHO better solution is that default files provide special functions to interact.
Let's imagine.

-- base file
local ideologies = { --[[ stuff here]] }

Loader = {}
function Loader.GetIdeologies()
    return ideologies
end

function Loader.AddIdeology(tbl)
    table.insert(ideologies, tbl)
end

-- your Mod A file

Loader.AddIdeology({
    --[[ stuff here ]]
})

-- your Mod B file

Loader.AddIdeology({
    --[[ stuff here ]]
})

We can also add mode to override default file completely, or just run code within default one, like if there is a mod that want to completely rewrite the logic of the file.

 

[azum4roll]

Hmm so you can use include this way... nice to know.

So we'll need to do the following for this to work in the real VP context:
1. Create the table.
2. Add the injection code to the end of every .lua file.
3. Ask the modmodders to do the rest.

3.1. Change their .lua file names so they're unique in VFS, no longer overriding existing .lua files.​

3.2. Insert the file names into the new table.​

I notice that your injection code currently doesn't factor in dependencies though. How would that work?

Also, will there be any memory problem if the VFS loads in even more files?

@adan_eslavo you might be interested.

 

[Vaderkos]

@azum4roll , @hokath I think this should not be at the end of file but rather at start to allow whole file overriding, and to simplify injection because lua doesn't simply hoist variables. So the structure for default files should be like

1. API code
2. Then code that checks table and does includes
3. Then default logic or nothing if whole file is overriden.

Each file should be done case by case, because bindings and hoisting.

If we are talking about VFS, I do not know what limitations are here, but as I tryied it doesn't actually load files when you start game, but when you include them. It uses IncludeFileList from Main State.

 

[Vaderkos]

I'm also working on project I called CPK aka Community Patch Kit. I've added such solution to make includes more strict.

--- Includes specified file from VFS. Ensures only 1 file is included.
    --- @param filename string
    --- @return nil
    local function Import(filename)
        local type = _lua_type(filename)

        if type ~= 'string' then
            local message = 'Failed to include file because filename is not string but ' .. type
            _lua_error(message)
        end

        local included = _civ_include(filename)

        if #included == 1 then
            print('Succesfully included file "' .. filename .. '"')
            return nil
        end

        local failed = 'Failed to include file "' .. filename .. '".'

        if #included < 1 then
            local message = failed .. ' File not found.'
                .. '\n' .. 'Please verify if specified file is registered in VFS.'
                .. '\n' .. 'Please verify if specified file exists.'

            _lua_error(message)
        end

        local tokens = {}

        for _, path in _lua_next, included, nil do
            _lua_table_insert(tokens, '"' .. path .. '"')
        end

        local message = failed .. ' File name is not unique enough. Multiple candidates found: {'
            .. '\n\t' .. _lua_table_concat(tokens, ',\n\t')
            .. '\n}'

        _lua_error(message)
    end

 

[hokath]

Aha ok. So this is like having some kind of #ifdef inside the VP files that will give you entrypoints into the functions. I think I understand.
At least it is nice and modular.