Problem: Mod Conficts... Solution: Mod Compatibility Index

[jdblue]

The new in-game mod system seems to be a boon, both in terms of ease of mod installation (no more download-then-unzip-then-drag-and-drop) and in terms of the ability for one-click mod removal. Also, I believe it will encourage more mod usage for the small minority who are mystified by the inner workings of the folder structures for any given OS or program. All of these are good things.

The best feature of the new mod system is also its greatest weakness: concurrent mod loading. The one-click ability to combine the works of two or more great mod authors (or teams) and further customize the game is a great boon to players who before now would have had to rely on mod alchemists to synthesize the combination they were looking for. The problem becomes apparent on first crash, however: there is nothing to prevent the user from loading two or more incompatible mods.

The ideal solution, in my mind, is to create a centralized index of mods that gives the user a basic idea of what should (and does) work together. It would hopefully also provide mod authors with some good feedback on compatibility. This index would be based on a few data sets, so as to provide a wide and solid base of information for the user to make informed choices. It could also potentially expose users to new mods they can use in combination with their old favorites, as well as provide recommendations on which mods synergize best and which combinations, while technically workable, should be avoided.

The first data set would be author-reported compatibility. That would make it easy to make sure mod representation on the index is fairly high. Many of the most popular mods are already have some author-reported compatibility in their threads and this is a good basis from which to begin.

The second data set would be from user-reported testing. I'm not 100% sure yet how to structure the data collection parameters here but user-reported data on mod compatibility would have to be more than just a PM saying "yo, this Procylon mod here doesn't work with Lainey's Gossip mod." At the very least, users would need to provide basic system information, a description of the incompatibility issues (does the game hang/crash out, do the two mods create unintended exploits, etc...), and version numbers (as well as names) of all mods involved in their report. This gives mod authors more data to work with if they want to resolve unexpected incompatibilities, allows the MCI editor(s) data needed to attempt to reproduce the error, and also allows for better comparison with other incompatibility issues. Also, I imagine this data set would not be treated with quite as much reverence as the author-reported compatibility.

The third data set would probably be based on mod dependencies. (I assume I'm using the right term here. If not, please correct me.) As I understand it, dependencies are created when a mod edits an existing file or creates a new one. Spatzimaus explains it better than I could. This data would be more difficult for me to collect, being a novice at modding and thus slower than others at figuring out where to look to find the information. However, this data is probably quite useful since, as I understand it, two mods that each change natural resources in game have to change the same extant .lua and would thus be incompatible. (Right?)

The fourth data set would be reviews of different mod combinations, perhaps called synergy reports. I think these data points would probably be (for the most part) rarer than the others, but it's certainly worthwhile information to have.

So that's my idea. It certainly has room for refinement, and I'm open to suggestions and stealing... er... borrowing good ideas anyone has for improvements. Please post constructive feedback below.

I'm also open to a few volunteers. One who would like to assist me in drafting up a schema for the organization and visual layout of the index. (I'd like to publish the index within a forum thread, if at all possible.) Another individual or two who can help comb through mod threads and gather data, one who could gather mod dependency data, and finally once the index is up, any mod-happy players like myself who would want to occasionally play-test various combinations and report in their results.

 

[tyhar]

dear civ fanatics, here's a list of mods i used for a single game. it worked until turn 184, then black screen and crash. can any of you help me save my game, or is it lost? would be brilliant cause i'm up at rank 1 and would love to continue. have already tried setting the map to strategic view and i'm playing on the newest DX version (am on XP, so DX9).

am playing on 1.0.1.247, historic time, huge earth, 15 players game.

List of compatible mods (to round 184):
Blitzkrieg (v 1)
BuildingHappinessIncrease (v 1)
City State Diplomacy Mod (CSD) (v 22)
City States Leaders (v 3)
City States UU (v 2)
CityStatesRebalance (v 2)
Custom Notifications (v 3)
Early Exploration (v 1)
Emigration (v 2)
Garrison Training (v 4)
Global - 5 Units Per Tile (v 1)
Historic Game Speed (v 1)
ICBMv2 (v 3)
InfoAddict (v 13)
International Space Station (v 3)
Mercenaries (v 2)
Mod List (addin) (v 4)
Petroleum (v 3)
Policy - Free Warrior (v 2)
Policy - Reveal Capitals (v 1)
R.E.D. modpack (v 10)
RailroadCostReduce (v 1)
Random Events (v 7)
Tech - Extra Classics (v 1)
Tech - Satellites Reveal Cities (v 2)
TraitsRebalance (v 1)
Units - Herdsmen (v 1)
Units - Mounted Units (v 1)
Units - Naval Upgrades (v 3)
Units - Population (v 1)
Units - Prospectors (v 1)
Units - Rangers (v 1)
Units - TBM - No Core Changes (v 2)
Unofficial Patch and Thals Balance (v 72)
Utils - ActionIcons Extension (v 1)
Utils - Tech Tree Delayed Load (v 1)

have also included a mass of wonder mods and were all working up until turn 184.

hope the list of running together mods helps and also help that one of you can help me save my game and continue playing.

tons of thanks,
tyhar

 

[whoward69]

And there is the problem with such a system of "this is compatible with that" that makes it doomed to failure from the outset. Tyhar has listed 36 mods and an unspecified list of "a mass of wonder mods".

13 of those mods are mine. I know they work together partly because I play with them but mainly because I designed them to work together as I know intimately how they work.

So, for me to test if this list is compatible with my mods (even if I assume that the "mass of wonder mods" are irrelevant to the cause of the crash - which I wouldn't) I would have to test 1,124,000,727,777,607,680,000 combinations of mods. Even if I could test 1 a second (which is impossible) it would still take longer than the age of the Earth so far (about 4.5 billion years) - I doubt my PC will be around that long!

If you want to look at compatibility (which is a noble cause), you need to look at how mods are constructed - what do they "tweak". An off-the-top-of-my-head list would be

• XML row/update (or equivalent sql) only
• XML delete (or equivalent sql) for relationships only
• Custom UI addin XML and LUA (no changes to core code)
• XML delete for key tables (eg delete a unit, building)
• UI update XML only (changes to core code)
• UI update XML and LUA (changes to core code)
• Changes to InGame.xml and/or InGame.lua
• Changes to ActionIcons.lua
• LUA event handling

Any mods that only do the first three are probably "safe" together. Others will have varying levels of potential interaction - although how you quantify that interaction is a matter for debate.

W

 

[tyhar]

And there is the problem with such a system of "this is compatible with that" that makes it doomed to failure from the outset. Tyhar has listed 36 mods and an unspecified list of "a mass of wonder mods".

13 of those mods are mine. I know they work together partly because I play with them but mainly because I designed them to work together as I know intimately how they work.

So, for me to test if this list is compatible with my mods (even if I assume that the "mass of wonder mods" are irrelevant to the cause of the crash - which I wouldn't) I would have to test 1,124,000,727,777,607,680,000 combinations of mods. Even if I could test 1 a second (which is impossible) it would still take longer than the age of the Earth so far (about 4.5 billion years) - I doubt my PC will be around that long!

If you want to look at compatibility (which is a noble cause), you need to look at how mods are constructed - what do they "tweak". An off-the-top-of-my-head list would be

• XML row/update (or equivalent sql) only
• XML delete (or equivalent sql) for relationships only
• Custom UI addin XML and LUA (no changes to core code)
• XML delete for key tables (eg delete a unit, building)
• UI update XML only (changes to core code)
• UI update XML and LUA (changes to core code)
• Changes to InGame.xml and/or InGame.lua
• Changes to ActionIcons.lua
• LUA event handling

Any mods that only do the first three are probably "safe" together. Others will have varying levels of potential interaction - although how you quantify that interaction is a matter for debate.

W

good point! but as i'm not the greatest modder (to be honest, not a modder at all!), can you give me a hint of which mods you use for playing? I'll use them and then hopefully play on happily ever after.

big cheers,
ty

 

[Spatzimaus]

If you want to look at compatibility (which is a noble cause), you need to look at how mods are constructed - what do they "tweak".

Correct. Basically, you can put mods into three categories, in terms of compatibility, based on what they do internally:

GREEN: Completely safe to mix and match. Generally, this applies to mods that add new XML rows or update existing rows, or use the SQL equivalents. No deletions allowed.
Also, most mods that adds new Lua event-based functions will fall into this category; while they might be very complex, there's little or no interaction with any other components. You can have a half-dozen functions all keying off the start of the active player's turn without any problems.
Smaller "content" XML mods (custom civs, for instance) are often in this category, although if the creator handled certain parts badly (like the text keys) there can be conflicts.

It's very rare that two Greens would conflict with each other, and generally it'd only happen when they try to modify the same thing in different ways. So mod A would try slowing down tech research by raising costs, while mod B would slow it down by reducing the science output of certain buildings; there'd be no actual conflict, but it's unlikely that the user would want both to happen at the same time. Or maybe two mods add new civilizations, but both want to use a certain city's name in their mod; if the modder was smart about how he declared the text keys this wouldn't be an issue, but too many try to follow the vanilla game's formats. Or maybe one mod wants to change the Watermill from +2 food to +10% food, while another wants to change it to +3 food; the actual effect would depend on the load order.

YELLOW: Mostly safe; it's unlikely that any other mod will conflict, and when they do conflict one will often replace the other entirely. XML deletes fall into this category, as do many InfoAddict-style UI mods. XML modifications that effectively delete an entry by setting the previous effect to 0 are just as problematic as a true Delete, so those would often go here as well.
In the case of modifying existing Lua, it's often the case that two or more mods "conflict" by both trying to modify the same file in the same way; there are a few fairly common UI fixes that have found their way into several major mods by now.
Some of the more advanced user-made Lua functions can slide a mod into this category, if they do things that extend beyond that function. For instance, while most user-made Lua is fine for compatibility issues, a Nuke Interception function that kills nuclear weapons outright (triggering off RunCombatSim) would have the potential to screw up any other functions triggering off RunCombatSim or EndCombatSim.

You can often use two Yellows together; a quick check can usually be done to see if one mod would conflict with others, by looking at the files involved, but there's no transitive property; A and B might work together, and B and C might work together, but A and C can conflict. So this sort of thing must be done on a case-by-case basis.

RED: Forget about it. These mods MIGHT be compatible with each other, but it's very unlikely. These include:
> Total Conversion content mods that completely remove the existing game's content.
> Large-scale content mods that add entire new eras, rearrange the tech tree into a pyramid, add an entire Religion system to the game by overhauling the Policies, etc.
> Anything that edits certain commonly-modified core files, such as AssignStartingPlots.lua, ActionIcons.lua, or InGame.xml. AssignStartingPlots would be changed by any mod that adds new resources or adjusts resource rarities, while ActionIcons would be changed by any mod that adds new Build actions (especially new Improvements).
> Anything that edits the art definitions, especially those related to unit models. (Therefore, mods that add new 3D models for custom units will be incompatible with something like R.E.D., which rescales the existing models.)
It's extremely unlikely that two Reds (or a Red and any number of Yellows) could work together without major modification.

----

So given this sort of classification, you could basically use one Red and a bunch of Greens without any problems. Most of the mods listed in tyhar's post would, in my opinion, fall into the Green category.

The thing is, you can't go by whether the game seems to work; most of the conflicts won't be immediately apparent. It just won't do what you expected it to do, but there often won't be a crash or anything. For instance, take the example mentioned earlier:
Mod A changes the Watermill from +2 food to +10% food. It does this by changing the Building_Yields table entry for Watermill food from +2 to +0, and then adding a new entry in the Building_YieldModifiers table at +10.
Mod B changes the Watermill from +2 food to +3, through an Update on the Building_Yields table.

If A loads before B, then you get a building that gives +3 food and +10% food. If B loads before A, you get just the +10% food, as if B never happened.
If A had used a Delete instead of changing the +2 to +0, then you'd get just the +10% no matter which order they loaded in, because the Update wouldn't trigger if A came before B. So in this particular case, a Delete would actually be better than the Update, in terms of compatibility.

 

[whoward69]

And then the "mod technology" changes all the time so some things that were red may become green. For example, I have a solution to the ActionIcons.lua issue - 5 of my mods all add builds/improvements, but none of them conflict as they all handle extensions to ActionIcons.lua via a 6th mod that extends the functionallity with a shared mod database table.

 

[Spatzimaus]

Right, or how InGame.xml was originally used in nearly every Lua mod, but was eventually replaced with InGameUIAddin commands. So yes, things can change.

For instance, AssignStartingPlots could EASILY be reworked by the developers to reduce incompatibilities; most of the issues stem from the two subroutines that decide how many units of resources are placed in a single deposit of each. As a result, adding a new strategic requires making new versions of those, which are then incompatible with any other mod (or map script) that only alters those values.
All they'd need to do is have the game use the XML quantities as the default for when the value isn't set in those routines, and you'd be done. But until that time, any mods that adjust resource quantities or add new resources are incompatible.

 

[whoward69]

GREEN: Completely safe to mix and match. Generally, this applies to mods that add new XML rows or update existing rows, or use the SQL equivalents. No deletions allowed.

I'd also remove from "green" any mods that directly manipulate the (auto-increment) ID field, but include any mods that only delete from tables that contain relationships (usually these don't have an auto-increment ID field), for example, removing promotions from units, removing resource requirements from units, etc.

 

[Spatzimaus]

I'd also remove from "green" any mods that directly manipulate the (auto-increment) ID field

Correct, but outside of the total conversion mods (which are Red already) and mods that delete content (Yellow), there's really no reason to ever do this in the first place. The auto-increment function is incredibly useful. To me, manually setting IDs is sort of like using InGame.xml for your Lua; you CAN do it, but it's something we learned a while back not to do.

but include any mods that only delete from tables that contain relationships (usually these don't have an auto-increment ID field), for example, removing promotions from units, removing resource requirements from units, etc.

Generally I'd agree. Modifications to secondary (non-ID) tables won't necessarily crash anything, and as I explained in the Watermill example I gave earlier, a Delete in a secondary table is actually BETTER for compatibility than an Update. But there are some flaky things that can happen if you mess around with these; for instance, deleting a single tech prerequisite might not screw anything up, but if you delete ALL of the prerequisites for a tech, then it becomes available in the Ancient era. So if a tech requires A and B, and the first mod removes the dependency on A (so that it only needs B) and the second mod removes the dependency on B (so that it only needs A), then combining the two mods would make it available on turn 1, which would REALLY screw up the AI.
Similar problems can happen if you remove the AI table entries for a unit, etc., so while not as dangerous as a Delete in a primary (ID-listed) table, they're not completely safe either.