Mod Conflicts/Mod Manager Proposal

PlotinusRedux

Warlord
Joined
Jul 11, 2013
Messages
196
[EDIT: I've got an experimental version of this ready at https://forums.civfanatics.com/resources/civ-6-mod-manager-experimental.25880/ ]

I'm just getting started on Civ modding--I never did any with the early Civ's, though I've been an avid player since Civ 1.

Apparently most UI mods require completely overriding existing .lua and .xml files under Base\Assets through <Import>. I'd have thought they'd make a system where you could override specific functions in .lua and tags in .xml, but apparently not.

I've already run into an incompatibility with another mod because we both modified DiplomaticActionView.lua and .xml--even though we modified different parts of both files and our mods would actually complement each other.

The differences are easily automatically resolved with any 3-way merge application, but most users won't even know what a "3-way merge" is much less how to do it. One user has already merged our mods and posted a link to it, but he'll have to update that any time either of us make a change and then if someone else posts a 3rd mod affecting the same files....

So I have a proposal I'd like feedback on.

I've used a an open-source 3-way merge library that handles merges fine as long as neither branch modified the same line. I've tested it on the mod conflict above and it produced a new mod combining both of them that works fine. Of course that won't always be be the case--if both branches modified the same line (by heuristics, not line #), it will fail, as it should, because that represents a true incompatibility. Also, since it doesn't understand the logic of the files it's merging, the merged file might not work--if one person deleted a global the other person refers to, etc. But 95% of the time, unless the mods really do conflict, it will work.

So my proposal is to write a ModManager application (sorry, mac users, in C#, I'm just not familiar enough with Java to do it in Java--Java is one of the few major languages I've never spent significant time with. I could do it in C++, but I'd end up making Windows calls for the UI anyway). It would read through the mods directory and list all mods installed, along with any that conflicted with each other (at least initially just on the basis of overriding the same files in the Import section of the .modinfo file). It could also list additional information such as version--maybe eventually linking to Civfanatics.com to indicate if a new version was available, though that would probably require Civfanatics.com to add a query page--or at least if the .modinfo included a custom <url> tag under Mod\Properties, provide a hot link to that, which could be the Civfanatics page for the mod.

Anyway, the main thing would be from there the user could check conflicting mods and it would attempt to build a new mod that included them all using 3-way merge to resolve differences, with a warning that the result might not work and a failure message if the merge found conflicts. That mod would automatically be added to the mods directory with a description listing all the mods (and their authors) included in it and a custom Mods/Properties value that would allow the ModManager to recognize it as a merged mod and offer to rebuild it if any of the included mods changed.

That's something I could write in 2-3 days--I wrote the base already to test merging my mod with the one it conflicts with. Of course I'd post the source to on GitHub.

Would something like this be useful and a good idea? Would a more limited tool for modders showing differences from the base files and letting them manually resolve any 3 way merge conflicts be better (the for-general-consumption one would just fail if there were any conflicts that couldn't be auto-resolved), though they'd then have to post the resulting combined mod for others to use and redo it when any of the underlying mods changed?
 
Last edited:
BTW, if you don't think it's a good idea, don't worry about hurting my feelings--I'm asking because I'm new and recognize there's a lot I don't know.

I know once they get Steam Workshop up that would make everything but the merging redundant, and once they release the dll source (or at least the debug .pdb to make it easier to modify in IDA) that would be a better place for the merging of incompatible mods as an on-the-fly attempt among selected mods.

In fact, really, with the dll source, we could make a single ModExtension mod that bypassed the current mod system and allowed modders to create files that modified base .xml files using xpath instead of replacing them, overrode specific functions in base .lua files rather than the entire files, etc. Was anything like that ever done for Civ 5, or did it just have a better system to begin with?
 
Personally, I would love this as both a user and creator. There's a lot of things I want to hit upon, but watching the conflicts already arising from a lot of really great mods is keeping me going very, very slowly ahead.

Something like this would help tremendously on the user side but even more on the creator side... Compatibility patches would be far easier to make and (in theory) if they had to be done manually with a listing of what the issues were.

I would like to ask you to consider looking at NexusMods as well... there aren't a lot of mods there atm, but I only have mine there as of right now (I'm looking to put them here as well at some point).
 
I'll start with a side note : in civ5 you were able to replace any element of the UI without overriding any file, I've not tested it in civ6, but it should be possible too.

That said it's much easier to replace the original file, and even if every modders was doing it (if it's still possible), conflict could still happen on UI elements moddified by two mods (and then being impossible to resolve from a mod manager)

And then there are some files that can't be modded without an override (every files related to map creation for example, you can replace utility functions in a new map script, but afaik you can't replace them for any map script)

So yes a manager would always be useful (showing possible sources of conflict even without being able to merge would still be a great help as pointed by Mynex), and I suppose that a merging tool is not something we're going to get from the official SDK.
 
I'll start with a side note : in civ5 you were able to replace any element of the UI without overriding any file, I've not tested it in civ6, but it should be possible too.

Could you elaborate on that, @Gedemon, or point me to a link? I've done some googling of Civ 5 UI modding, but haven't seen a way to replace an element of the UI without overriding the .xml that defined it.
 
Could you elaborate on that, @Gedemon, or point me to a link? I've done some googling of Civ 5 UI modding, but haven't seen a way to replace an element of the UI without overriding the .xml that defined it.
see LookUpControl for ContextPtr and all related UI functions, like SetHide and ChangeParent

As said I've not tested any of this in civ6 yet, but it should still work (the events may differ), here is an example from InfoAddicts

InfoAddictHooks.lua
Code:
-- InfoAddictHooks
-- Author: Rob
-- DateCreated: 7/9/2012 6:33:58 PM
--------------------------------------------------------------

include("InfoAddictLib")
logger:setLevel(INFO);
logger:trace("Loading InfoAddictHooks");

-- UI Hooks are set up here upon initialize of the mod. The scroll menu entry is added and
-- then buttons are added to the leader screens.



-- Add an item to the DiploCorner drop down (scroll menu) to access InfoAddict

function OnDiploCornerPopup()
  UIManager:PushModal(MapModData.InfoAddict.InfoAddictScreenContext)
end

function OnAdditionalInformationDropdownGatherEntries(additionalEntries)
  table.insert(additionalEntries, {
    text=Locale.ConvertTextKey("TXT_KEY_INFOADDICT_MAIN_TITLE"),
    call=OnDiploCornerPopup
  })
end
LuaEvents.AdditionalInformationDropdownGatherEntries.Add(OnAdditionalInformationDropdownGatherEntries)
LuaEvents.RequestRefreshAdditionalInformationDropdownEntries()



-- Set up buttons that will open InfoAddict from other screens. Once the game is done loading,
-- these buttons are moved to the appropriate locations.

function OnInfoAddict()
  local InfoAddictControl = MapModData.InfoAddict.InfoAddictScreenContext;
  UIManager:PushModal(InfoAddictControl);
end;
Controls.IAB_DiploTrade:RegisterCallback( Mouse.eLClick, OnInfoAddict );
Controls.IAB_DiscussLeader:RegisterCallback( Mouse.eLClick, OnInfoAddict );
Controls.IAB_DiscussionDialog:RegisterCallback( Mouse.eLClick, OnInfoAddict );


-- Had to tie this to LoadScreenClose because the LeaderHead contexts are not loaded before
-- mods (or, at least, are not available though LookUpControl)

function MoveIAButtons()
 
  logger:info("Changing InfoAddict button visibility to LeaderHead contexts");

  Controls.IAB_DiploTrade:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiploTrade"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiploTrade"):ReprocessAnchoring();
 
  Controls.IAB_DiscussionDialog:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog"):ReprocessAnchoring();
 
  Controls.IAB_DiscussLeader:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog/DiscussLeader"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog/DiscussLeader"):ReprocessAnchoring();

end;
Events.LoadScreenClose.Add(MoveIAButtons)

InfoAddictHooks.xml
HTML:
<?xml version="1.0" encoding="utf-8"?>
<!-- Created by robk on 7/9/2012 6:12:54 PM -->
<Context>
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiploTrade" />
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiscussionDialog" />
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiscussLeader" />
</Context>

In that example it adds elements to civ5 leader screen, but you can use similar methods to hide or replace UI element from another InGame context using your own UI (use ContextPtr:LookUpControl to get the UI element and its parent, hide the UI element, attach your replacement to the parent), loaded using the <UserInterface> tag in the .modinfo (you put the YourUI.xml under that tag, and both YourUI.xml and YourUI.Lua under the <Files> section, but not in the <ImportFiles> section in that case)
Code:
        <UserInterface>       
            <Properties>
                <Context>InGame</Context>
            </Properties>
            <Items>
                <File>YourUI.xml</File>
            </Items>
        </UserInterface>

All that would benefit from a complete tutorial, but working on the UI have always been one of the most difficult part of civ5 modding for me, and I don't like to write tutorials on things I'm not confortable with myself.
 
Thanks, @Gedemon, I'll take a look at that some more and see if it can do the kinds of things I'm needing.

For one of my mods (ShowDiplomaticDeals) I was modifying the base XML to add a place in the Gossip area to show the deals:

Code:
 <Instance    Name="IntelGossipHistoryPanel">
    <Container ID="Background" Size="300,parent"    Offset="6,38" >
      <Image Texture="Controls_Gradient" Size="parent+2, 150"  Anchor="L,B" Color="24,47,70,230"/>
      <ScrollPanel ID="GossipScrollPanel"  Size="300, parent" Vertical="1" AutoScollbar="1">
        <ScrollBar Anchor="R,C" Size="5,14" AnchorSide="O,I"  Texture="Controls_ScrollBarBacking2" StateOffsetIncrement="0,0" Vertical="1" SliceCorner="2,2" SliceTextureSize="5,5">
          <Thumb Size="5,12" Texture="Controls_ScrollHandle" StateOffsetIncrement="0,0" SliceCorner="2,6" SliceTextureSize="5,12" Color="46,56,62,255"/>
        </ScrollBar>
    <Stack StackGrowth="Bottom" Size="parent,0" InnerPadding="8,8" Offset="0,0">
<!-- ShowDiplomaticDeals:Start -->
          <Container Size="parent,0" AutoSize="V">
            <Grid Style="DividerGrid" Size="parent-8,8" Anchor="C,C" Color="34,48,59,255"/>
            <Grid Style="DropShadow4" AutoSize="1" Anchor="C,C">
              <Label String="MY DEALS" Anchor="C,C"  Style="DiplomacyGossipHeader"/>
            </Grid>
          </Container>
          <Stack ID="MyDeals" Size="parent, 0"/>
          <Container Size="parent,0" AutoSize="V">
            <Grid Style="DividerGrid" Size="parent-8,8" Anchor="C,C" Color="34,48,59,255"/>
            <Grid Style="DropShadow4" AutoSize="1" Anchor="C,C">
              <Label String="OTHERS' DEALS" Anchor="C,C"  Style="DiplomacyGossipHeader"/>
            </Grid>
          </Container>
          <Stack ID="OtherDeals" Size="parent, 0"/> 
          <Container Size="parent,0" AutoSize="V">
            <Grid Style="DividerGrid" Size="parent-8,8" Anchor="C,C" Color="34,48,59,255"/>
            <Grid Style="DropShadow4" AutoSize="1" Anchor="C,C">
              <Label String="LOC_DIPLOMACY_INTEL_LAST_TEN_TURNS" Anchor="C,C"  Style="DiplomacyGossipHeader"/>
            </Grid>
        </Container>
<!-- ShowDiplomaticDeals:End -->
          <Stack ID="LastTenTurnsStack" Size="parent, 0"/>
          <Container ID="OlderHeader" Size="parent,0" AutoSize="V">
            <Grid Style="DividerGrid" Size="parent-8,8" Anchor="C,C" Color="34,48,59,255"/>
            <Grid Style="DropShadow4" AutoSize="1" Anchor="C,C">
              <Label String="LOC_DIPLOMACY_INTEL_OLDER" Anchor="C,C"  Style="DiplomacyGossipHeader"/>
            </Grid>
          </Container>
          <Stack ID="OlderStack" Size="parent, 0"/>
        </Stack>
        <Grid Anchor="C,T" Size="parent+13, 38" Offset="1,0" Style="DiplomacyTitleBarGrid" AnchorSide="I,O">
          <Label Anchor="C,C" String="LOC_DIPLOMACY_INTEL_GOSSIP_NAME" Style="DiplomacyIntelHeader" Offset="0,2"/>
        </Grid>
      </ScrollPanel>
      <Image Size="parent+2, 22" Texture="Controls_GradientSmall" Anchor="L,B" Rotate="180" Color="0,0,0,255"/>
    </Container>
  </Instance>

So what I'd need to do is either replaced that entire instance, or somehow from another file insert my Containers and Stacks into the anonymous Stack inside the "GossipScrollPanel" ScrollPanel of the "IntelGossipHistoryPanel" Instance.

I was thinking XPath commands would be the ideal way to do such mods as they would let multiple people modify the same XML file, and even both add new objects in the same Stack of the same Control, but if I can just override the "IntelGossipHistoryPanel" with my version using the ContextPtr stuff, that would solve the issue as well (as long as two of use weren't both trying to add things to that same control).
 
Writing the base classes to parse and merge .modinfo files led me to look at a lot of mods already out there and realize I needed to start with a firm understanding of what elements can actually appear in .modinfo and what they do.

After some research and testing, I came up with an XSD--https://forums.civfanatics.com/threads/civ6-modinfo-schema.606784/

Now that I've got that firmed up, I can adjust my base classes to the actual structure of .modinfo in short order.

I'm not sure what this project is going to turn out to be at this point, though--a users tool or a modders tool, or maybe I'll do both.

It's clear from looking at the number of mistakes in .modinfo files out there that a ModBuddy type program would be useful, I wish I knew how far out the SDK was, though, I don't want to write that then have Firaxis release the real thing a week later.

The basic mod conflict detection and resolution functionality would still be needed in any case, and that's easy enough for <ImportFiles> conflicts, but I'm seeing at lot of database conflicts as well, which is making me want to also start parsing the actual .xml and .sql files for conflicts--although all I could really do there is flag potential conflicts based on updates (rather than inserts) of the same tables since very different WHERE clauses could modify the same rows, a modder would then have to look to see if there was an actual conflict and determine how to resolve it
 
see LookUpControl for ContextPtr and all related UI functions, like SetHide and ChangeParent

As said I've not tested any of this in civ6 yet, but it should still work (the events may differ), here is an example from InfoAddicts

InfoAddictHooks.lua
Code:
-- InfoAddictHooks
-- Author: Rob
-- DateCreated: 7/9/2012 6:33:58 PM
--------------------------------------------------------------

include("InfoAddictLib")
logger:setLevel(INFO);
logger:trace("Loading InfoAddictHooks");

-- UI Hooks are set up here upon initialize of the mod. The scroll menu entry is added and
-- then buttons are added to the leader screens.



-- Add an item to the DiploCorner drop down (scroll menu) to access InfoAddict

function OnDiploCornerPopup()
  UIManager:PushModal(MapModData.InfoAddict.InfoAddictScreenContext)
end

function OnAdditionalInformationDropdownGatherEntries(additionalEntries)
  table.insert(additionalEntries, {
    text=Locale.ConvertTextKey("TXT_KEY_INFOADDICT_MAIN_TITLE"),
    call=OnDiploCornerPopup
  })
end
LuaEvents.AdditionalInformationDropdownGatherEntries.Add(OnAdditionalInformationDropdownGatherEntries)
LuaEvents.RequestRefreshAdditionalInformationDropdownEntries()



-- Set up buttons that will open InfoAddict from other screens. Once the game is done loading,
-- these buttons are moved to the appropriate locations.

function OnInfoAddict()
  local InfoAddictControl = MapModData.InfoAddict.InfoAddictScreenContext;
  UIManager:PushModal(InfoAddictControl);
end;
Controls.IAB_DiploTrade:RegisterCallback( Mouse.eLClick, OnInfoAddict );
Controls.IAB_DiscussLeader:RegisterCallback( Mouse.eLClick, OnInfoAddict );
Controls.IAB_DiscussionDialog:RegisterCallback( Mouse.eLClick, OnInfoAddict );


-- Had to tie this to LoadScreenClose because the LeaderHead contexts are not loaded before
-- mods (or, at least, are not available though LookUpControl)

function MoveIAButtons()
 
  logger:info("Changing InfoAddict button visibility to LeaderHead contexts");

  Controls.IAB_DiploTrade:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiploTrade"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiploTrade"):ReprocessAnchoring();
 
  Controls.IAB_DiscussionDialog:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog"):ReprocessAnchoring();
 
  Controls.IAB_DiscussLeader:ChangeParent(ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog/DiscussLeader"));
  ContextPtr:LookUpControl("/LeaderHeadRoot/DiscussionDialog/DiscussLeader"):ReprocessAnchoring();

end;
Events.LoadScreenClose.Add(MoveIAButtons)

InfoAddictHooks.xml
HTML:
<?xml version="1.0" encoding="utf-8"?>
<!-- Created by robk on 7/9/2012 6:12:54 PM -->
<Context>
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiploTrade" />
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiscussionDialog" />
  <GridButton Style="FrameButton200"  Anchor="R,B" Offset="40,40" String="TXT_KEY_INFOADDICT_CHECK_INFOADDICT" Font="TwCenMT14" ID="IAB_DiscussLeader" />
</Context>

In that example it adds elements to civ5 leader screen, but you can use similar methods to hide or replace UI element from another InGame context using your own UI (use ContextPtr:LookUpControl to get the UI element and its parent, hide the UI element, attach your replacement to the parent), loaded using the <UserInterface> tag in the .modinfo (you put the YourUI.xml under that tag, and both YourUI.xml and YourUI.Lua under the <Files> section, but not in the <ImportFiles> section in that case)
Code:
        <UserInterface>     
            <Properties>
                <Context>InGame</Context>
            </Properties>
            <Items>
                <File>YourUI.xml</File>
            </Items>
        </UserInterface>

All that would benefit from a complete tutorial, but working on the UI have always been one of the most difficult part of civ5 modding for me, and I don't like to write tutorials on things I'm not confortable with myself.

So here's my problem:

This works: local pControl:table = ContextPtr:LookUpControl( "/InGame/CivicsTree/NodePointer"); where NodePointer is defined as "<Instance Name="NodeInstance">.
This works: local pControl:table = ContextPtr:LookUpControl( "/InGame/DiplomacyActionView/");.
This returns nil: local pControl:table = ContextPtr:LookUpControl( "/InGame/DiplomacyActionView/IntelGossipHistoryPanel"); where IntelGossipHistoryPanel is defined as <Instance Name="IntelGossipHistoryPanel">.

I've spend 5 hours trying different variations. I thought for a bit this was working: "/InGame/DiplomacyActionView/PlayerContainer/IntelGossipHistoryPanel", as it didn't return nil, but pControl:GetID() showed it was really just returning PlayerContainer.

It must have something to do with how the instances are instantiated. I tried hooking to events that I know occur after IntelGossipHistoryPanel is instantiated through the InstanceManager but I still couldn't get a pointer to it. Any ideas?
 
So here's my problem:

This works: local pControl:table = ContextPtr:LookUpControl( "/InGame/CivicsTree/NodePointer"); where NodePointer is defined as "<Instance Name="NodeInstance">.
This works: local pControl:table = ContextPtr:LookUpControl( "/InGame/DiplomacyActionView/");.
This returns nil: local pControl:table = ContextPtr:LookUpControl( "/InGame/DiplomacyActionView/IntelGossipHistoryPanel"); where IntelGossipHistoryPanel is defined as <Instance Name="IntelGossipHistoryPanel">.

I've spend 5 hours trying different variations. I thought for a bit this was working: "/InGame/DiplomacyActionView/PlayerContainer/IntelGossipHistoryPanel", as it didn't return nil, but pControl:GetID() showed it was really just returning PlayerContainer.

It must have something to do with how the instances are instantiated. I tried hooking to events that I know occur after IntelGossipHistoryPanel is instantiated through the InstanceManager but I still couldn't get a pointer to it. Any ideas?
Not sure if you have solved this yourself already, but LookUpControl only works with IDs, not with Names. Your first example is actually defined by an ID, while the third one isn't (and the second one is just a call to whole file).

Code:
1: <Tutorial ID="NodePointer" Style="TutorialContainer" Anchor="L,C" Offset="180, -50" TriggerBy="TutorialChangeCivic">
3: <Instance	Name="IntelGossipHistoryPanel">

I currently have a similar problem though, and the elements I try to select actually do have an ID.
This works: local EspionageChooser1 = ContextPtr:LookUpControl("/InGame/EspionageChooser/MissionStack");
This doesn't: local EspionageChooser2 = ContextPtr:LookUpControl("/InGame/EspionageChooser/MissionDistrictIcon");

I can iterate through the first element with for i, entry in ipairs(EspionageChooser1:GetChildren()) do, but it's ugly as hell and I really would prefer not to do this (especially since I have to do this 6 levels deep to arrive at the MissionDistrictIcon label element I'm trying to access). I also don't seem to be able to call all of the methods for these iterated child elements. For example, :SetHide() works, but :SetText() doesn't.
Is there some way to traverse the XML nodes, or a "proper" way to select a specific child element? Or an instance?
 
Back
Top Bottom