Mods, ArtDefs, Icons, etc.--Research & Workaround

[PlotinusRedux]

[EDIT: This entire project has been rendered unnecessary by the Window patch]

So, I finally got my DLL injection working with Civ6 from the time it starts, despite steamapi.dll's attempts to prevent it, which let me see exactly when the game is loading files and reloading or not reloading them.

As I suspected, most of the files people are having to manually copy to the game's directory to work with mods are only loaded once, before the game even gets to the Main Menu.

Specifically, all the .xml files under Base/Assets/UI/Icons; some (though not all, but including the all important units.artdef) of the .artdef files; and of course the XML files dealing with the main menu itself.

Possible Workaround

This means I could use my DLL injection, which is redirecting the CreateFile() API (for those that don't know, CreateFile() despite its name is also the Windows function to open existing files) to trap all the calls to open files and redirect them to the files under enabled mods when those exist, preventing people from having to copy files to their base install.

The problem is, since the files are only loaded at start up, any time you changed the mods that were enabled or wanted to load a save game with different mods enabled, you'd have to restart the game to get the new files loaded. That's not ideal, but it's what people are already having to do after manually copying the files to the base directory, so it would at least be an improvement.

Another problem would likely be, Windows Defender seems happy to allow my DLL injection without complaint, but I imagine it would set off all kinds of alarms for real anti-virus software, as it's exactly the kind of thing a virus would be doing, and there's no way for the anti-virus to know that in this case it's beneficial rather than harmful. Though I could work around that by altering the .exe itself--I did that with my Alpha Centauri patch and no one ever reported any anti-virus alarms.

Without knowing how long it will be before Firaxis addresses this, I may go ahead and do that--the hard part (getting the DLL injected) is already done.

Research Notes

I played around with @Deliverator 's Moar Units mod to see what I could do just with the .modinfo file itself. Adding his Civ6.dep file under <Components><ModArt>, I can see it really is loading that file from the mod directory. Unfortunately, after that, it isn't reloading Units.artdef, nor is it loading his Moar_Units.artdef file defined in his Civ6.dep, even if I add them to <ImportFiles>, etc. Since his .dep file is referring to Moar_Units.artdef which isn't being loaded, and the logs are showing no errors, I can only conclude the code is incomplete and they got as far as loading it then did nothing with it (probably because they realized it would require reloading .artdef files they're going to have to rework other code to allow).

Sadly, a number of .artdef files *are* reloaded when you create a game--Districts.artdef, Buildings.artdef, etc. But none of the Unit*.artdef files and no .artdef files you've added.

For anyone interested, here are the files that are loaded at each point, with the first set being files that would require a restart to redirect after a mod change, and the second set being files that could be changed without reloading:

Files loaded on startup before getting to the main menu: http://qskoz.com/Civ6/Civ6Loaded_BeforeMainMenu.txt

Files loaded during game creation: http://qskoz.com/Civ6/Civ6Loaded_DuringGameCreation.txt

Note Mods/MoarUnitsTest/Base/Civ6.dep is in that last set, though it seems to be ignored.

 

[PlotinusRedux]

I've got @Deliverator 's Moar Units mod working without copying files to the base directories through redirection by my injected DLL.

Testing that brought up an interesting point, though. For files that overwrite existing files in the base directory, I don't need to know where to fake them being, because I'm matching them against the files already in the base directory by filename.

However, Deliverator's mod at least also includes files that needed to be faked as being in the base directory that weren't already there, like Units_Moar.artdef--for those I had to also redirect the FindFirstFile() and FindNextFile() calls so I can tell the game new files exist where they don't so it will know to load them. (I was hoping when it loaded his Civ6.dep file it would be smart enough to try to load Units_Moar.artdef on it's own, but no such luck.)

For things like .artdef, I know where to fake them being just by the file extension, but for things like MOAR_Units_GameplayText.xml, .xml files can go anywhere.

Fortunately Deliverator mirrored the base directory structure all the way up to Base--i.e., MOAR_Units_GameplayText.xml is in MOAR_Units_Assets\Base\Assets\Text\en_US.

I don't expect everyone has mirrored the base directory structure all the way up to Base, but as long as they have included enough of the directory structure for me to uniquely identify what directory under base to use--for instance, "\Icons\" is enough for me to know where to fake them because there's only one Icons directory under Base; there are 2 en_US directories, but "\Text\en_US" is enough for me to distinguish it from "\Subtitles\en_US\"; etc.

If anyone knows of a mod that has files the user is supposed to copy that *are not* under a directory with the same name as the directory under Base, and which aren't replacing an existing file, let me know, as my procedure won't work for them.

BTW, the method I'm using to determine if a mod's file needs redirection is by parsing the <Files> element of its .modinfo file, then scanning the mod's directories for .xml, .lua, .artdef, .dep, .sql, etc. Any such files I find that *aren't* included in the <Files> element I assume were meant to be copied to the base directory and thus require redirection.

A more elegant solution would be to add a new element to the .modinfo files listing files to redirect, but I wanted this to work without mod authors having to change anything, especially since this is a temporary work-around Firaxis will hopefully soon make unnecessary.

Right now my DLL is just reading the mods.sqlite database to see what mods are enabled, and if 2 mods both require the same file to be redirected, whichever one I read last wins.

I've got a front end that's currently displaying all the installed mods with all their components, <Files> files, non-Files files, letting you enable and disable mods directly from it, letting you click on <ImportFiles> to see a difference window from the base files, etc. It's currently detecting simple mod conflicts (importing or replacing the same file), but I having figured out how I want to display that info yet. I've got my 3-way auto-merge code written, but I need to test it more before I trust it to merge conflicting mods unsupervised to see if it works on most existing mod conflicts and can at least fairly reliably report when there is a conflict it can't resolve. I also still need to write the UI for manual 3 way merge, but I've got a good start on that just from the 2-way difference window.

 

[Horem]

This sounds very intresting.

 

[Deliverator]

This is some very interesting and clever work! Is the aim of this to have everything load from the mod folder without any additional manual steps?

 

[PlotinusRedux]

@Deliverator yes, I've got that much working already, though for mods that involve files the game only loads at startup (the icon xml's, civ6.dep, units.artdef, etc.--most of the ones people are having to manually copy now), you'll either need to enable the mods you want before launching (which you can do through my mod viewer), or restart the game after enabling them in the game--just when you're changing which ones are enabled, of course.

The ultimate aim, though, is to have it also auto-merge conflicting mod files when possible without any additional manual steps--so if you've enabled 2 mods that both contain the same icon xml files, it can see that and do a 3-way merge on-they-fly so the game reads a file containing the icons from both mods without the user even having to know it's happening. Otherwise, we're quickly going to have a lot of incompatible mods, as lots of mods need to add icons, entries into units.artdef (or at least Civ6.dep to point to new .artdef files like you did), etc.

It's sharing code with my mod manager which is more a modder's tool to easily see exactly what each mod has changed in imported/replaced files, what mods conflict with what others, do 3-way merges with manual conflict resolution to create combined mods, act as a mini-mod-buddy for creating and maintaining .modinfo files, etc.

 

[Mynex]

@Deliverator yes, I've got that much working already, though for mods that involve files the game only loads at startup (the icon xml's, civ6.dep, units.artdef, etc.--most of the ones people are having to manually copy now), you'll either need to enable the mods you want before launching (which you can do through my mod viewer), or restart the game after enabling them in the game--just when you're changing which ones are enabled, of course.

Would the restart be required if the modinfo had the mod enabled already?
i.e. -
<EnabledByDefault>1</EnabledByDefault>
<EnabledAtStartup>1</EnabledAtStartup>

I do that in my mods because I figure anyone installing my mods wants them enabled... and if they don't they can manually disable it.

Just curious.

 

[PlotinusRedux]

[Edit: I wrote the below before realizing they had just released a new patch, now I need to do some testing with it to see if anything relevant has changed.]

(Of course, below and in this thread in general I'm only talking about mods that contain files that need to be loaded at startup--i.e., ones users have to manually copy to the base directories today, generally Icon*.xml, Civ6.dep, *.artdef, UI elements and lua scripts like the main menu that get displayed before the game loads mods at all, and scripts that need to run in the GameplayScripts context every time a game is reloaded, not just when a game is first created. Translations could also count, since the game currently doesn't like to reload them when a game is reloaded unless they were loaded at startup. Mods that don't contain such files would continue to work as they do today.)

@Mynex, the restart would only be required when enabling a mod that wasn't already enabled when they started Civ6.

However, the *first* time someone runs Civ after installing your mod, it isn't already enabled, because it doesn't yet exist in mods.sqlite--it has to hit the code that scans the Mods directory for changes and updates the database first, which it unfortunately does right *after* it finishes loading all the Icons*.xml files.

I've got some ideas for a workaround for that, but I haven't decided on one or implemented it yet:

(A) would involve having my mod viewer always run before the game and be the place people enable and disable mods rather than using the in-game screen. If it detects a mod that hasn't been installed into mods.sqlite yet, it could either (1) know the mod should be enabled because the user selected it, or because it parsed the EnabledByDefault element; (2) actually update mods.sqlite itself--I'm parsing all the info needed to do so, but that wouldn't be very future proof if they add more elements to .modinfo later; or (3) shell Civ6 in a hidden window to get the mod installled, then kill it (Civ6, not the mod, that is).

(B) would involve having the injected DLL rescan mods.sqlite any time the game has updated it to look for mods now enabled that weren't enabled on startup and which contain files that need to be redirected at startup, and inform the user that Civ6 needs to restart to implement the changes--and then maybe go ahead and restart it for them. That probably needs to be part of any solution, as it will also take care of mods they manually turn on or off in the game.

(C) A combination of (A) and (B).

I also still don't have a good solution for loading games with a different mod set than the ones currently enabled--the only solution I've got so far is to require users to manually enable and disable mods to match a saved game before they load it.

What I really need is a way to force the game to reload the things it loads on startup again after startup, but that isn't simple, and I suspect that's the reason Firaxis is taking so long to improve the modding system, they've got to rework that code to make reloading possible which probably means changing a lot of hard-coded stuff.

The mod viewer in (A) would basically be a slimmed down version of my Mod Manager with only the parts useful for a user--i.e., without the parts for modders like viewing file differences, manually-aided mod merging, etc. It would list all the installed mods with version, title, teaser, and author information, and whether they were enabled or not; let the user enabled, disable, or delete them; warn if 2 enabled mods had a detectable conflict (in terms of <ImportFiles> or start-up redirected files, I'm not going to parse the <UpdateDatabase> files and try to figure out what database changes might conflict at this point); allow them to group mods together to be enabled or disabled all at once; allow clicking to shell to a website if the .modinfo contained one; and let users install new mods by selecting a .zip, .7z, .gz, etc. file which it would then unpack to the mods directory for them (for users who have trouble figuring out how to unpack mods to the right place). I've got most of that coded in my Mod Manager already, I'd just need to put a simpler UI in front of it.

BTW, they got rid of <EnabledAtStartup> in the last patch and replaced it with <DisabledAtStartup>. I think all that element does is turn the mod off every time the game starts, which you really only want to do for the tutorial (the only .modinfo that uses it), so everything without <EnabledAtStartup> was defaulting to 1 for it; and since it makes more sense for a default value to be 0 than 1, they changed it to <DisabledAtStartup> so the default value could be 0 if the element isn't present.

 

[PlotinusRedux]

After a quick review of the new Poland_Jadwiga DLC, it looks like most of what I was going to do with this project they've already fixed:

I see a new <Icons> element;
It looks like <ModArt> is finally working;
<GameScripts> now apparently work on reload.

I'm got some testing to do, but this entire project is probably no longer needed.