NetSyncTools - MP Compatible Custom UIs

Vicevirtuoso · Jan 25, 2016

NetSyncTools
version 1

Overview
NetSyncTools is a utility which is designed to allow custom user interface elements to function properly in multiplayer without desychronizing the game.

Why It's Needed
All Lua elements run locally on each user's machine in a multiplayer game. Otherwise, every player would be seeing identical information! Many actions the player can take, such as selecting one of their cities or units, or opening the Culture Overview, have no actual effect on the game state. Thus, no information about these acts are ever sent to the other players in a multiplayer game. However, many other actions do have an effect on the game state. A player selecting a city to view has no relevance to the other players, but that player choosing the city's production does.

When actions which affect the game state are handled in Lua, specific functions are called from the Game API or the Network API. These functions will actually send game state data to the DLL, which will in turn be sent to all other players.

In mods which include a custom Lua script which is intended to change the state of the game (i.e new Civilizations' UAs), the network synchronization is most often handled by the functions in the script being tied to GameEvents. If the mod is properly coded, functions which run from GameEvents hooks will produce identical results on all machines on the network. (Therefore, this utility is not needed if your script functions are called entirely from GameEvents.)

However, new user interface elements intended to allow the player access to new actions which perform changes to the game state run into an issue. Because there is no call to a Game or Network API, the changes will only apply to the local user's machine. Lua will, essentially, think that adding new units or policies is as trivial as opening up your Military Overview screen! None of this data will be sent to the other users, causing the game state to be different for each player, which causes a desync. The problem is that there is no "generic" Game or Network API function to send custom data to the other players. Every one of these functions has a specific purpose - sometimes very specific. (SendMinorNoUnitSpawning, anyone?) Without being able to define any way to send custom data over the network, what is a modder with dreams of multiplayer compatibility to do?

Enter NetSyncTools!

How It Works
Several months ago (yes, I was very slow in deciding to actually write and test this), I discovered that there was some interesting behavior in Network.SendSellBuilding, and the associated GameEvents.CitySoldBuilding. As it turns out, as long as you have a city ID, you can send whatever arbitrary number you want through SendSellBuilding as its building ID, and CitySoldBuilding will be called with that number as its building ID -- even if that building ID doesn't even exist in the Buildings table!

Testing with some arbitrarily defined large numbers as fake "building IDs", I was successfully able to make multiple UI elements which send their game state changes through the network. Examples can be found in all of my Hyperdimension Neptunia civilizations, with double examples in Planeptune and Ultra Planeptune; the Broken Lords also have an example.

However, randomly assigning hardcoded building ID numbers and just hoping they never conflict with those of other modders is a poor solution at best.

What It Does
NetSyncTools removes the guesswork of assigning these fake building ID numbers by keeping a table of functions and their associated building ID numbers. The modder will register their function to NetSyncTools using a string key, and will call the function again using the same string key.

By default, registering a new function uses up one fake building ID number. However, you can optionally define a number of IDs to reserve for your function. If you do so, you can have your function be passed additional integer data. For example, you could reserve 22 IDs. When you call your NetSynced function, you can pass a variable between 1 and 22 to it. This could, for example, identify a specific other player for the function to use. This part may be confusing, so an example of this will be shown below.

How to Implement
NetSyncTools should be loaded in your mod as an InGameUIAddin. The script will detect multiple copies such that only one will run at a time for all mods.

How to Define and Register Your Functions
LuaEvents.AddNetSyncFunction(fFunction, sKey, iNumbersToReserve)
fFunction is the entire function to be registered to the table, sKey is the string key you wish to use, and iNumbersToReserve is the number of IDs you wish to reserve for your function to use to be able to pass integer data. If you don't need this (i.e. the button is basically just an on/off switch), you do not have to define it.

How to Call Your Functions
LuaEvents.NetSyncCall(sKey, iCity, iData)
sKey is the string key for your function registered above.
iCity is the City ID which will be passed to GameEvents.CitySoldBuilding. If not defined, will default to the capital. Only necessary if your function affects cities in some way.
iData is optional integer data passed to a function. Must have defined iNumbersToReserve in AddNetSyncFunction, and the number must be less than or equal to the value iNumbersToReserve was.

Examples

Spoiler :

Example using additional integer data to kill a random unit belonging to a specific enemy player:

Spoiler :

And to invoke this function against, for example, player 4 (assuming you have pPlayer defined as the pointer object for the active player):
LuaEvents.NetSyncCall("VV_KILL_RANDOM_UNIT", pPlayer:GetCapitalCity:GetID(), 4)

Potentially Major Drawback
SendSellBuilding doesn't work if the player sending it has no cities. Any Civ with a "stays alive without any cities" element of their UA, or the use of the Complete Kills option, will render players unable to use any NetSynced functions when they are in such a situation.

Vicevirtuoso · Feb 11, 2016

Soo...anyone attempted to use this in their mods yet?

TPangolin · Feb 12, 2016

Not yet, but we're toying around with when we're planning to update our civs with this!

Vicevirtuoso · Feb 14, 2016

Good to hear. Let me know if you run into any issues, or if using it gets confusing! (I kinda feel like I didn't write the instructions very clearly.)

Serp · Feb 27, 2016

I want to try your solution, but I did not get it completely.

Vicevirtuoso said:

Example using additional integer data to kill a random unit belonging to a specific enemy player:

Spoiler :

So let's analyse this example.

Is the "Events.LoadScreenClose.Add(function ()" necessary? what does it? and there is a bracket missing?

Why there is no "LuaEvents.NetSyncCall(sKey, iCity, iData)" in this example? I thought this is the way it works?

Beside these things. How does the network will now know which unit from which player died? Maybe the iPlayer and iOtherPlayer is sent to the network, but where is the unit sent?
Or is everything done within "KillRandomUnit" sent to the network?

How about the parameters from KillRandomUnit. So the seconds param always has to be "iCity" and when I call the function, it has to be a valid city ID ... random one, but it has to be a city from the player at this machine (GetActivePlayer()) ?
The first and third parameter can be whatever I want? Are also 4th and 5th ... parameters possible?

Typhlomence · Feb 27, 2016

Great to see this!

I'm embarrassed to say, though, it took me a long time to figure out how the "iPlayer" parameter in each of those example functions got a value - I assume your Lua code somehow sends the player who pressed the UI button through the network as well, and therefore the first parameter of functions attached to NetSyncCall will always be the player who sent it (like the second parameter will always be the city)?

Vicevirtuoso said:
By default, registering a new function uses up one fake building ID number. However, you can optionally define a number of IDs to reserve for your function. If you do so, you can have your function be passed additional integer data. For example, you could reserve 22 IDs. When you call your NetSynced function, you can pass a variable between 1 and 22 to it. This could, for example, identify a specific other player for the function to use. This part may be confusing, so an example of this will be shown below.

If the minimum value is 1, does that mean if I want a range of 0 to 22, I would have to set the number of IDs to 23 and subtract 1 from the integer that is sent?

Vicevirtuoso said:
How to Implement
NetSyncTools should be loaded in your mod as an InGameUIAddin. The script will detect multiple copies such that only one will run at a time for all mods.

As it's an InGameUIAddin, that means I don't need an include for NetSyncTools in my main Lua script, right? I just add the "Events.LoadScreenClose.Add(function () LuaEvents.AddNetSyncFunction() end)" to my script, and call LuaEvents.NetSyncCall() when necessary?

Also, I'm a bit uncertain about another thing - what happens on the active player's machine when LuaEvents.NetSyncCall() is called? In the function that fires when the UI button is pressed, will the function I call through NetSyncCall() run for the player that pressed the button, as well, or do I have to handle that separately? I hope that question makes sense...

I'm going to try answering your questions as well, Serp - VV, correct me if I'm wrong on anything here, please.

Serp said:
Is the "Events.LoadScreenClose.Add(function ()" necessary? what does it? and there is a bracket missing?

VV just missed out the "end)" for that example - it's there at the end in the first example.

Serp said:
Why there is no "LuaEvents.NetSyncCall(sKey, iCity, iData)" in this example? I thought this is the way it works?

I assume you put that call in the function that fires when the button in your UI is pressed. Although an example of doing this might be nice, VV - it might answer my last question up there, too.

Serp said:
Beside these things. How does the network will now know which unit from which player died? Maybe the iPlayer and iOtherPlayer is sent to the network, but where is the unit sent?
Or is everything done within "KillRandomUnit" sent to the network?

If you look at the example again, Game.Rand() is used. That function generates a random number, but the number will be the same for all players. Each player in a specific multiplayer game has the same seed, and so they will get the same sequence of numbers from Game.Rand(). Therefore, the random unit picked out will be the same, and no communication between computers needs to occur to keep the game in sync.

That's why for anything that needs a random number, you should use Game.Rand().

Serp said:
How about the parameters from KillRandomUnit. So the seconds param always has to be "iCity" and when I call the function, it has to be a valid city ID ... random one, but it has to be a city from the player at this machine (GetActivePlayer()) ?
The first and third parameter can be whatever I want? Are also 4th and 5th ... parameters possible?

The first parameter in that function, I assume (see the very first paragraph of this reply) is the player who pressed the button in the custom UI. I also assume the third parameter in the function is the integer data you sent in the "iData" part of "LuaEvents.NetSyncCall(sKey, iCity, iData)". So in that case, no, you can't have more parameters here, as you can only send one integer in the first place.

The second iCity parameter is the one you sent in the NetSyncCall, or the capital if you did not specify one.

Serp · Feb 28, 2016

Thanks

I missunderstood the NetSyncTools

I thought after killing the random unit, the tool would notify everyone, that a specific unit was killed. But instead the tool forces everyone to run the function "KillRandomUnit"

If it is really the case, that you can only send exactly these 3 parameters, it is difficult to use.
I thought about the "Multiple Upgrades" Mod from whoward. When upgrading a unit, there you can choose in a popup, if you wish to upgrade e.g. to a spearman or a swordsman.
So we would have no way of telling the function which is called everywhere, which unit exactly is upgraded, right? So we would nee another workaround, like getting the selected unit or we could use the SaveUtilis Tool, to save all releveant information in a game object.

Am I right? Or don't we need another workaround to pass all information?

I really would like to see a complete example mod

At best a simple one like the KillRandomUnit Example, but with the need to pass additional information.

whoward69 · Feb 28, 2016

Serp said:
If it is really the case, that you can only send exactly these 3 parameters, it is difficult to use.

If you need to send multiple parameters, assuming I'm reading the code correctly, you will need to encode the multiple parameters into a single parameter.

For example, if you have three params (A, B and C) and param A can have 22 values (major civs) and param B up to 50 values (an upper limit on the number of possible units per civ) and param C up to 3 values (no upgrade, upgrade choice 1, upgrade choice 2) you will need to allocate (at least - probably more) 22 x 50 x 3 = 3300 "building slots" and then use some funky maths to X = encode(A, B, C) and pass X as the single parameter and then at the other end do A, B, C = decode(X). Easiest way to do this will be as binary masks, but then you need to round the max param values up to a multiple of two, so we now need 32 x 64 x 4 = 4096 building slots.

In some cases you will get the current player for free (so assuming the upgrade is for the current player, we only need to do X = encode(B, C) for 256 building slots), but this won't always be the case (eg when dealing with other civs).

The system appears to be designed for "what choice did this civ in this city make", eg, it would work well for my Morindim civ, where you can pick the type of totem (earth, water, air) to associate with the newly built totem pole in the city, but for general purpose UI mods, without multiple parameters, it's going to be very hard to use.

Unless I've completely misinterpreted how it works ...

Vicevirtuoso · Feb 28, 2016

Serp said:
So let's analyse this example.

Is the "Events.LoadScreenClose.Add(function ()" necessary? what does it? and there is a bracket missing?

Yes, this registers the function to NetSyncTools' Lua context. Without doing this, your function won't properly be called by it. The missing "end)" is because I'm dumb and don't proofread my posts.

Why there is no "LuaEvents.NetSyncCall(sKey, iCity, iData)" in this example? I thought this is the way it works?

That would be how you'd call that function after registering it, yeah. I'll put a note below the example to say this.

Typhlomence said:
Great to see this!

I'm embarrassed to say, though, it took me a long time to figure out how the "iPlayer" parameter in each of those example functions got a value - I assume your Lua code somehow sends the player who pressed the UI button through the network as well, and therefore the first parameter of functions attached to NetSyncCall will always be the player who sent it (like the second parameter will always be the city)?

When Network.SendSellBuilding is called, the resulting invocation of the GameEvents.CitySoldBuilding hook will be passed the player ID of whoever actually sent that Network request through the network.

Example, if Player ID 1 were to click on a button which did:
Network.SendSellBuilding(4196, 1)
Then the resulting GameEvents.SendSellBuilding would be passed:
(1, 4196, 1)

If the minimum value is 1, does that mean if I want a range of 0 to 22, I would have to set the number of IDs to 23 and subtract 1 from the integer that is sent?

You'll reserve 23 numbers, but you won't need to add or subtract anything from the player ID when you send it from a NetSyncCall.

(Do remember that MAX_MAJOR_CIVS usually only goes up to 21, not 22, though...

)

As it's an InGameUIAddin, that means I don't need an include for NetSyncTools in my main Lua script, right? I just add the "Events.LoadScreenClose.Add(function () LuaEvents.AddNetSyncFunction() end)" to my script, and call LuaEvents.NetSyncCall() when necessary?

Right. The reason it's an InGameUIAddin and not included via VFS is so that only one copy will ever run at a time. It has duplicate script prevention like sukritact's MCIS. If every script had their own NetSyncTools, then you'd have every function duplicated a number of times equal to how many other mods are using a copy of NetSyncTools, which...yeah. :crazyeye:

Also, I'm a bit uncertain about another thing - what happens on the active player's machine when LuaEvents.NetSyncCall() is called? In the function that fires when the UI button is pressed, will the function I call through NetSyncCall() run for the player that pressed the button, as well, or do I have to handle that separately? I hope that question makes sense...

It works on both the active player's machine and all other machines on the network. It works in single player games, too.

VV just missed out the "end)" for that example - it's there at the end in the first example.

As previously stated, I'm dumb. :cringe:

I assume you put that call in the function that fires when the button in your UI is pressed. Although an example of doing this might be nice, VV - it might answer my last question up there, too.

For example, to invoke the "KillRandomUnit" function in the example against player 4, and assuming you have pPlayer defined as the pointer object for the active player, you would use:
LuaEvents.NetSyncCall("VV_KILL_RANDOM_UNIT", pPlayer:GetCapitalCity:GetID(), 4)
I will add this to the OP because, looking back on it, it's kinda silly I didn't include this.

If you look at the example again, Game.Rand() is used. That function generates a random number, but the number will be the same for all players. Each player in a specific multiplayer game has the same seed, and so they will get the same sequence of numbers from Game.Rand(). Therefore, the random unit picked out will be the same, and no communication between computers needs to occur to keep the game in sync.

That's why for anything that needs a random number, you should use Game.Rand().

Right. As long as you aren't getting your randomization through the math.random() function, it should remain consistent.

Serp said:
Thanks
I missunderstood the NetSyncTools I thought after killing the random unit, the tool would notify everyone, that a specific unit was killed. But instead the tool forces everyone to run the function "KillRandomUnit"

If it is really the case, that you can only send exactly these 3 parameters, it is difficult to use.

It indeed has limitations. It's a bit of a hacked-up workaround to the issue; its primary benefit is that it does not require a modified DLL. Therefore, there are fewer potential conflicts, and Mac/Linux users aren't left out to dry.

If you are comfortable with having a dependency on DLL VMC or Community Patch for your mod, then I would recommend using custom unit missions as a better solution.

I thought about the "Multiple Upgrades" Mod from whoward. When upgrading a unit, there you can choose in a popup, if you wish to upgrade e.g. to a spearman or a swordsman.
So we would have no way of telling the function which is called everywhere, which unit exactly is upgraded, right? So we would nee another workaround, like getting the selected unit or we could use the SaveUtilis Tool, to save all releveant information in a game object.

Well, in the case of that mod, it requires his DLL VMC to work. As such, he already has network compatibility built into those new upgrade buttons.

However, it could be done with NetSyncTools. I will admit to it being rather unwieldy though. Essentially, to make a function tied to a specific unit...you will need to register a NetSync function and reserve 99,999,999 numbers for it. Possibly even more than that; I'm really not sure what the upper bound of numbers would be when the game assigns Unit IDs. When the button is clicked, you would need to code it such that it gets the selected unit's Unit ID, then passes that as data to the NetSync function.

NetSyncTools really does work best for more limited "on/off switch" functions; if you plan on adding new unit actions, need to send something like a string, or need the function to work for city-less players, I would recommend making your mod dependent on DLL VMC or Community Patch instead.

I really would like to see a complete example mod At best a simple one like the KillRandomUnit Example, but with the need to pass additional information.

Planeptune is the one Civ I have which has been updated to use NetSyncTools -- others have had to wait until I get my fill of FFXIV. So this would be the best example of a mod in action which I've tested and confirmed works...at least, in a single player environment. I lack multiplayer testers :sad:

Her code is a bit involved, so to spare you the search, all of the NetSyncTools-relevant code is in NeptunePurpleHeartTraitScript.lua between lines 1093 and 1143.

More generally, these two Civs, while not having been updated to use NetSyncTools, still show examples of how to use the SendSellBuilding workaround using arbitrarily defined constants as building IDs:
http://steamcommunity.com/sharedfiles/filedetails/?id=595485589
http://steamcommunity.com/sharedfiles/filedetails/?id=460969393 (the simpler of the two)

whoward69 said:
If you need to send multiple parameters, assuming I'm reading the code correctly, you will need to encode the multiple parameters into a single parameter.

For example, if you have three params (A, B and C) and param A can have 22 values (major civs) and param B up to 50 values (an upper limit on the number of possible units per civ) and param C up to 3 values (no upgrade, upgrade choice 1, upgrade choice 2) you will need to allocate (at least - probably more) 22 x 50 x 3 = 3300 "building slots" and then use some funky maths to X = encode(A, B, C) and pass X as the single parameter and then at the other end do A, B, C = decode(X). Easiest way to do this will be as binary masks, but then you need to round the max param values up to a multiple of two, so we now need 32 x 64 x 4 = 4096 building slots.

It gets even worse than that, because you would have to take that 3300 and then add a fourth parameter, D, for every possible Unit ID which can be assigned to a unit. Otherwise, the other players on the network will have no way of knowing which particular unit you're trying to upgrade, save for some sort of limiting factor like "upgrade the unit stationed in city X." Since I'm allowing for a theoretical maximum of 8 digits for a Unit ID based off of your answers in this thread, that would mean 3300 x 99,999,999 = 329,999,996,700 -- far beyond the maximum size of 2,147,483,647 which can be passed!

However, there is a fair bit of leeway in what could be snuck through the function if you aren't having to account for unit IDs, depending on how creative you are with it.

whoward69 · Feb 28, 2016

Vicevirtuoso said:
It gets even worse than that, because you would have to take that 3300 and then add a fourth parameter, D, for every possible Unit ID which can be assigned to a unit. Otherwise, the other players on the network will have no way of knowing which particular unit you're trying to upgrade,

I was assuming all machines will iterate the units in the same order (so the 5th unit, not the unit with ID 234873) - but I've never tested that ...

Edit: Thinking about it, you can force the iteration. Simply order the units by ID ascending, and then pick the Nth - you'll be doing it so infrequently that the overhead will be irrelevant.

Vicevirtuoso · Feb 28, 2016

whoward69 said:
I was assuming all machines will iterate the units in the same order (so the 5th unit, not the unit with ID 234873) - but I've never tested that ...

This could theoretically be a solution to the problem, but it definitely needs testing in a multiplayer environment to see if iteration is indeed the same between machines.

whoward69 · Feb 28, 2016

See the edit to my post, you can force the iteration. And on the sender side, you don't even need to sort the units ... just count all of those with a lower ID value and add 1

Serp · Mar 13, 2016

Vicevirtuoso said:
Well, in the case of that mod, it requires his DLL VMC to work. As such, he already has network compatibility built into those new upgrade buttons.

You say all of whowards mods should already work in MP (if DLL required)? They don't, so why are you assuming this? Is there a easy trick that whoward did not see or simular?

About the problem of giving more parameters to the function:
Are you already close to a solution?
If not, how about the SaveUtilis mod? http://forums.civfanatics.com/showthread.php?t=392958
Just save the needed info in a game object with this and all players can access this info. After it is done, the info can be deleted.

So if you think this would work, could you include SaveUtilis in your tool?

whoward69 · Mar 14, 2016

Serp said:
You say all of whowards mods should already work in MP

VV didn't say that.

DLL-VMC has a set of events for custom missions, "any DLL-VMC dependant mod using those events will work in MP". That's significantly different from "any DLL-VMC dependant mod will work in MP"

Vicevirtuoso · Mar 17, 2016

Serp said:
About the problem of giving more parameters to the function:
Are you already close to a solution?

There really isn't one for this particular method. Anything more than passing along a city ID and a single integer value is going to require a custom DLL.

If not, how about the SaveUtilis mod? http://forums.civfanatics.com/showthread.php?t=392958
Just save the needed info in a game object with this and all players can access this info. After it is done, the info can be deleted.

So if you think this would work, could you include SaveUtilis in your tool?

This wouldn't help since it (and TableSaverLoader) don't send any data through the network; it's all saved to the local machine like most other Lua data.

whoward69 · Mar 18, 2016

What happens if you send a chat message to a player/team not in the game?

In Fire/LiveTuner console (needs to be on one line)

Code:

Events.GameMessageChat.Add(function (ifromPlayer, iToPlayer, sText, iTargetType) print(string.format("Chat - from: %i, to: %i, type: %i, txt: %s", ifromPlayer, iToPlayer, iTargetType, sText)) end

then

Code:

Network.SendChat("{x=100, y=200, z=300}", -1, -1)

or possibly

Code:

Network.SendChat("{x=100, y=200, z=300}", 21, 21)

assuming player/team 21 isn't in the game

Vicevirtuoso · Mar 18, 2016

whoward69 said:
What happens if you send a chat message to a player/team not in the game?

This could potentially work. Will need someone to test it. Firetuner does not work in MP games (for obvious reasons), so a simple test script will need to be written. Probably just containing:

Code:

Events.GameMessageChat.Add(function (ifromPlayer, iToPlayer, sText, iTargetType) print(string.format("Chat - from: %i, to: %i, type: %i, txt: %s", ifromPlayer, iToPlayer, iTargetType, sText)) end
Events.SequenceGameInitComplete.Add(function() Network.SendChat("{x=100, y=200, z=300}", 21, 21) end)

I'm also curious if the Chat functions work at all in single player games.

whoward69 · Mar 18, 2016

Vicevirtuoso said:
I'm also curious if the Chat functions work at all in single player games.

Your library doesn't need them to, as you can detect MP/SP and use plain LuaEvents for SP

Vicevirtuoso · Mar 19, 2016

whoward69 said:
Your library doesn't need them to, as you can detect MP/SP and use plain LuaEvents for SP

Right, just curious if I'll have to make separate functions for MP and SP with this method. Network.SendSellBuilding works identically in SP and MP, so there are currently no checks for which type of game it is.

whoward69 · Mar 19, 2016

Just need one check as your library loads

Code:

-- This variable ends up holding a pointer to a function,
-- that can be called as SendNetworkMessage(...)
local SendNetworkMessage

-- The SP aware function
function SpSendNetworkMessage(...)
  -- do something here with LuaEvents
end

-- The MP aware function
function MpSendNetworkMessage(...)
  -- do something here with chat messages
end

-- Which function should we be using?
if (isMPGame()) then
  SendNetworkMessage = MpSendNetworkMessage
else
  SendNetworkMessage = SpSendNetworkMessage
end

User now just calls SendNetworkMessage(...) and doesn't care if they are in SP or MP

NetSyncTools - MP Compatible Custom UIs

The Modetta Man

Attachments

The Modetta Man

Just the worst person

The Modetta Man

King

Magical Tomomo

King

DLL Minion

The Modetta Man

DLL Minion

The Modetta Man

DLL Minion

King

DLL Minion

The Modetta Man

DLL Minion

The Modetta Man

DLL Minion

The Modetta Man

DLL Minion

Similar threads