Mods / Plugins / User-Extensibility

[Puppeteer]

I'm breaking this out as a separate topic:

Then you get into areas like interfaces where mods could hook in, and not having worked on a project where modules could be loaded dynamically (especially modules in another languages, such as Lua), I'm much less certain how to approach it.

My current project uses plugins in Python, and I've done Minecraft Forge modding which is based on event callbacks, so I have some experience with the pattern but I don't know yet how closely those ideas carry over to Godot and especially Lua. It would be great if we can find a working example of runtime Godot plugins.

Embedded Script Interpreters

Lua and Python are interpreted script languages. Lua (including its C# implementation MoonSharp) and IronPython in a particular Python are embedded (optionally so for IronPython), meaning the interpreter is compiled into your project. No dynamic loading beyond reading a text file or string is needed because the project understands Lua or Python script.

You can just expose the script environment to a class or interface definition and let the script have at it. For non-privileged scripts, just design the class methods and setters so you can't break the rules. e.g. At the "control this unit" class level, prevent a settler from walking on water or taking too many moves if the script tries to.

I prototyped this in SimpleGame. I defined a Turn API/class in C# code and document it in the readme. In SimpleGame I have the Lua scripts defined as constants or getters or something, but that's only because I couldn't be bothered to write the code to read in a text file or provide a script editor for a proof of concept.

Spoiler Lua handler setup code :

See explanation following code.

    class LuaAI : Script, IAI {
        public LuaAI(string script) : base(CoreModules.Preset_HardSandbox) {
            UserData.RegisterType<Turn>();
            DoString(script);
        }
        public void PlayTurn(Turn turn) {
            DynValue _ = Call(Globals["player_turn"], turn);
        }
    }

This LuaAI class implements the IAI interface (Artificial Intelligence, a turn-taker) and is a child of MoonSharp.Script which is the Lua interpreter class.

The constructor registers the Turn class/interface definition in the Lua environment, meaning the Lua script can access the properties and methods of the C# Turn object. (It also calls the parent constructor in a way to sandbox itself; to prevent the script from accessing the filesystem, for example.)

The MoonSharp.Script.DoString() call runs the passed script in the Lua environment instantiated by being a child of MoonSharp.Script. See example Lua code below. In this case I'm defining a Lua handler function player_turn. (Note there are some automatic capitalization changes between C# names and Lua names.)

The PlayTurn() method of the code above (part of the IAI interface) handles the Turn object by passing it directly to the player_turn function inside the configured-in-constructor Lua environment.

        function player_turn(turn)
            if turn.isEnemyInRange == true then
                turn.attack()
            else
                turn.move(1,0)
            end
        end

Other Mod Methods

Don't think about Godot here. Think about C#. Godot is just our display and UI. (Possible exceptions for handling various media, but I think we're talking code extensibility here.)

In C# all CLR, DLL assemblies can be made. In the SimpleGame examples shown above, a class API is defined that can be exposed to an interpreted script. We can pass that same interface/class to any CLR assembly it seems to me. In SimpleGame I pass a Turn object to a turn handler function/method, so why can't that turn handler be loaded in a CLR DLL? I haven't proof-of-concepted this yet, but I can't see any blockers from here.

Ozymandias was hyped about "a dll for each civ," and I think we can have exactly that if we want. I'm hand-waving away some of the implementation, but it seems to me a path to a DLL could be specified in a BIQ-analog, and the module class/interface API in question can use the handler function defined in that DLL for that particular thing.

Note that this opens modding to any CLR language. But it would need to reference the interface definition and be precompiled into DLLs, of course, unlike the embedded options which could read any script string/text on the fly. I think .NET DLLs are platform-independent, but I'm not entirely sure of that.

Editing to add: In my idealistic vision, this interface defining how to do a thing (e.g. take a turn) would be the same accessed by the UI, so the human player and any script are controlling the game through the same interface. That may be much easier said than done, though...I haven't really thought that through completely. That would mean lots of probing the interface class for legal options, and I'm not sure that will be fun, productive, or practical to code as such, especially in a remote server multiplayer environment.

 

[Quintillus]

(This post evolved into writing out my understanding so errors in it can be pointed out, but still has a fair amount of questions)

Hmm... I have read through it a couple times and tried to cross-reference it to the SimpleGame repo, but I still have some possibly-stupid questions. It still seems a bit magical.

   class LuaAI : Script, IAI {
        public LuaAI(string script) : base(CoreModules.Preset_HardSandbox) {
            UserData.RegisterType<Turn>();
            DoString(script);
        }
        public void PlayTurn(Turn turn) {
            DynValue _ = Call(Globals["player_turn"], turn);
        }
    }

All right, so this is the C# code. The "base(CoreModules.Preset_HardSandbox)" must be what's sandboxing it (and that prevents file system access and... some other things? would there be cases we prefer a "soft" sandbox, if that's a thing?).

I think I follow that it's registering the turn class (is it always UserData.RegisterType to make something available to Lua? I.e. is UserData a name you chose, or one that is always used for this Lua interoperability?) DoString executing the script seems straightforward, although in practice I'd have questions about what path we should pass for the script.

I think I'm only starting to get the PlayTurn function now that I look at the full AI.cs class on GitHub again. It is defined on the IAI interface (as the only method), and the SimpleAI class has a C# definition. So, the LuaAI class (which is basically a C# class, which is a child of MoonSharp.Script) also implements PlayTurn, just like SimpleAI. But its implementation uses some MoonSharp functionality that calls the "player_turn" method in the script. So when we create a new LuaAI, and it hits the DoString line, we've already defined that PlayTurn should call the Lua player_turn function. It appears to also be responsible for passing the "turn" object, which answers my question about, "how does the Lua script know about instances of C# objects?"

I'm still not sure what the "DynValue _" part of that line does, though. Is is just required syntax to call Lua?

It also helps my understanding to realize that with MoonSharp, we have a CLR implementation of Lua. That makes the fact that it can access objects less magical and more believable.

There may be some stupid questions in the above, but hopefully it shows where I either understand or misunderstand what's going on. And just writing it has helped me think through what's going on and (hopefully) understand a bit more of it.

-------------

The Lua code actually makes sense now that I think I understand how the Lua is invoked, and how it gets the turn variable. One question is, would that code snippet be the entirety of the Lua file? And kind of related to the "what path should we pass for the script" question in C#, where would it live? I guess in theory it could be anywhere if we could pass the right path, but a full example of that would help cement things.

---------

Overall, having the C# SimpleAI and LuaAI to both look at definitely helped my understanding, and this explanation (and reading it a few times and rubber-duck-debugging my understanding of it) has resulted in a much better understanding than when I had just seen LuaAI without a Lua script or a more detailed explanation.

 

[Puppeteer]

The "base(CoreModules.Preset_HardSandbox)" must be what's sandboxing it (and that prevents file system access and... some other things? would there be cases we prefer a "soft" sandbox, if that's a thing?).

Lua in general is a simple syntax, has one dynamic variable type, and additional functionality is provided through "core modules". The MoonSharp.Script constructor allows a bit flag parameter to disable core modules, and there are a few handy presets such as the one I used which among other things disables the `io` module (which is the global `io` name in the Lua namespace). https://www.moonsharp.org/sandbox.html

I think I follow that it's registering the turn class (is it always UserData.RegisterType to make something available to Lua? I.e. is UserData a name you chose, or one that is always used for this Lua interoperability?)

UserData is part of the MoonSharp API; I'm not sure offhand if that concept is defined by Lua definitions or if each Lua implementation does its own thing. There is also UserData.RegisterAssembly() along with marking classes in code with `[MoonSharpUserData]` to specify which class definitions are available to the Lua script environment. https://www.moonsharp.org/objects.html

DoString executing the script seems straightforward, although in practice I'd have questions about what path we should pass for the script.

I'm not sure I understand what you mean here. The MoonSharp.Script instance—or in this case its child LuaAI—is a standalone Lua environment/state with global variable/modules set. DoString() is running the provided string/text/script in the Lua environment/state.

In this particular case, after the DoString, the function player_turn exists in the Lua environment which persists with the C# instance, because the passed string defined that function. https://www.moonsharp.org/tutorial2.html#step-2-change-the-code-to-create-a-script-object

(Oops, DoString returns a DynValue I didn't code for here.) More on DynValue in a bit.

So, the LuaAI class (which is basically a C# class, which is a child of MoonSharp.Script) also implements PlayTurn, just like SimpleAI. But its implementation uses some MoonSharp functionality that calls the "player_turn" method in the script. So when we create a new LuaAI, and it hits the DoString line, we've already defined that PlayTurn should call the Lua player_turn function. It appears to also be responsible for passing the "turn" object, which answers my question about, "how does the Lua script know about instances of C# objects?"

DoString() runs the string in the Lua environment. In this particular case, all it does is define the function `player_turn`. And yes, PlayTurn just hands off the Turn object to what it expects to be a function in the Lua environment. Note that this is not checked anywhere. If I had passed the script `print "hur durr"` with no function definition...actually I'm not sure what would happen. I think it would just silently fail? Or throw an exception that `player_turn` doesn't exist or isn't a function.

The interface isn't necessary for general MoonSharp; I'm just using an interface so we can have very different implementations for the functionality.

I'm still not sure what the "DynValue _" part of that line does, though. Is is just required syntax to call Lua?

(TL;DR it just discards the return value of type MoonSharp.DynValue)

Ok, so, I've dealt with Lua implementations in two different languages now. Lua itself sort-of has one variable type, a dynamic value. It my represent a function, a table, a string, a number, and maybe another thing or two. https://www.lua.org/manual/5.2/manual.html#2.1

Strongly-typed languages of course care more about the type. MoonSharp does some pretty darn good auto conversion in most cases (overloaded functions with parameters differing only by int/float are an edge case as Lua just has the Number dynamic type for all numericals...not sure what MoonSharp does in this case). Go is a very explicit language and requires type assertion syntax for everything, so MoonSharp in C# is a lot easier to use than gopher-lua in Go.

So, in the native language (C# in our case), all values passed between the native language and the Lua environment are type DynValue. Since `Call` returns a DynValue that I don't care to use, I'm using the underscore instead of a variable name. I should have also done it for the DoString() call earlier, too, but didn't realize it. C# seems to discard unassigned values, anyway.

I find myself suddenly confused about passing multiple values. I wanted to say they get put in a table (basically an associative array that can also behave as a 1-based indexed array), but I recall in the more explicit Go coding of a Lua function you have to return a native integer which is the count of values on the stack. (Lua pushes and pops values on the Lua stack for passing data.) I'm not sure if that's specific to the gopher-lua implementation or if Lua works that way internally. In any case, inside Lua it's pretty chill about whether or not you pass it the right number of parameters to a function, so if it's important in Lua you need to check if the value is not null (e.g. `param = param or 0` to provide a default value and inherent type).

One question is, would that code snippet be the entirety of the Lua file? And kind of related to the "what path should we pass for the script" question in C#, where would it live? I guess in theory it could be anywhere if we could pass the right path, but a full example of that would help cement things.

We get to decide. It's like how we were stuffing nearly everything into Game.cs and then started breaking it out.

We can have as many Lua states/environments as we want. What I mean by state/environment is the global state. Say we run a script `x = 42`. The global variable `x` is then a dynamic value / number type equal to 42. We can run additional scripts in that same state/environment/MoonSharp.Script-instance and reference, modify, or delete `x`. Or maybe `x` is a function to handle turns or a diplomacy check or a game rule.

So we'll have to figure out if it's one Lua state per mod, per game, per invocation...really don't know how to organize that yet.

Then there's what you asked about, the script text. Maybe each mod has a monster list of handler functions in one file. Maybe we'll have some sort of file tree for scripts, one per function or one per module. (A Lua rule to determine if a civ is allowed to research or trade for a particular tech may be a completely different thing than a Lua script to decide if a tile is passable by a particular unit right now, like maybe a land unit crossing a drawbridge.) Or maybe we'll just stick short Lua snippets in JSON files where convenient. Maybe something like this as part of a BIQ-analog JSON file:

{
  "ruleOverride": {
    "type": "lua",
    "script": "function handle_rule (task, game_status) do_stuff(task, game_status) end"
  }
}

I think it will come down to how many things we allow Lua to handle, and then the organization of the scripts/files might emerge from that.