Civ6 Modding Evaluation

[S3rgeus]

While developing the Wheel of Time mod for Civ5, we've been considering migrating our work from Civ5 to Civ6, now that the modding tools have been released.

I'm in the process of evaluating how well Civ6 will work for WoTMod and will be using this topic to post updates about the work I'm doing on that evaluation. I'm aware there will be pre-DLL-source limitations to modding Civ6 that puts WoTMod's more sweeping changes out of reach for now, but I want to understand what modding looks like in Civ6 well enough to make a good decision one way or the other.

Mostly this will end up being some kind of journal about me learning to mod Civ6, coming from Civ5. Some of the information may be useful to others, but I figure I might as well share what I find.

Step 1: a simple mod that adds a new unit that's only a simple gameplay variant of an existing one. Nothing fancy, would be super simple for me to do in Civ5.

In Civ6 this also turned out to be relatively straightforward:




I had a bit of a false start with setting all of the files in the mod to be loaded at the FrontEnd, instead of In-game, which caused some database errors. But that meant I went through database.log and some basic debugging of that kind of problem which is a good thing too. When I had changed the actions to load In-game, but had an error in my XML (referenced a row in the Types table that I had commented out) it also seemed like starting a new game immediately bounced back to the main menu. If all database errors (or even just all foreign key constraint failures) cause this behavior, then that's great. Makes it much easier to know that you've got something wrong in your XML files rather than finding confusing knock-ons in-game of that data not having loaded correctly.

Having explored a bit of Civ6's structure, I'm liking Modifiers as a concept. Seems nice and modular. A skim of NycholusV's two topics on the subject was a good primer.

Thankfully SQLite Personal can still open the game's debug gameplay database, like it could in Civ5. Unfortunately I couldn't find any database files for the icon or localization databases, so if you know those are available to view somewhere then feel free to let me know in my question topic about it. I did notice via Procmon that Civ6 tries to load a localization database from '<Civ6 install dir>\Base\Assets\Text\Localization.sqlite' if it's present, but there's no file there on my computer. (It seems to try the install directory before the directory in My Documents for several of the SQL databases.)

And that's all for now! I'll be back later this week with some other simple variants of the above. Adding a new building and the like seems quite similar to a unit. I might try out some Modifiers on those.

After that, I'll be creating a simple test civilization to try out bringing uniques and the like together. Then it's time for some unit graphics work, possibly even some 3D model import exploratory work.

 

[Leyrann]

Before I'm continuing reading the post - OMG A WHEEL OF TIME MOD YOU HAVE ALL MY SUPPORT TO MAKE ONE FOR CIV6!!!!

 

[counterpoint]

Hey folks, I'm one of the other devs on the WotMod... thanks in advance for any modding thoughts you share!

Glad this is working out so far!

 

[S3rgeus]

I've been away for most of this week, but have been spending some more time on this today. Adding a new building/district that is pure gameplay seems fairly trivial, basically the same thing as the unit that I got working above.

However, I have tried to add a custom icon to a new district. As expected, this is fraught with challenges. I'm learning a ton about the Firaxis Asset Editor and how it interacts with ModBuddy. How XLPs, ArtDefs, and BLPs all fit together with DDS files. There's basically a ton of metadata that Asset Editor should generate for us from the DDS files, but convincing it to do so is quite challenging. Based on some of the work done by ReichAdler it seems that manually authoring these files will let you get the rest of the way to actually importing icons into the game. But authoring these metadata files is amazingly labor intensive and they're clearly intended to be generated, rather than hand-authored. (So I'm trying to get the full generation pipeline up and running.)

At the moment, trying to load up any new DDS files into Asset Editor as textures leads me to a "file out of date" error (requesting that I "sync the file before exporting"). This is plainly a Perforce-related error, where the Asset Editor is expecting to be able to query the status of the newly imported files and, predictably can't access Firaxis's Perforce server from my machine. (I'm assuming this was to prevent devs from accidentally working on outdated files and having to scrap their work to rebase off the latest versions.) I'm currently using a decompiler to poke around Asset Editor's source and trying to convince it to not use a Perforce server. Failing that, if I can find out what address it's querying for the server at, then I can set up a Perforce server to trick it.

Overall I'm liking the look of Asset Editor. I'm convinced that I'm just using it wrong, because it's very close to working on my machine and it looks like it creates a quite straightforward import process once it's actually working. The capabilities of Civ6's art pipeline are also well exemplified by Deliverator's work on custom units. Getting to that same stage (completely new unit model with full custom animations) in Civ5 took more than a year and this import pipeline already looks more stable than Civ5's is now.

 

[esvath]

Fully interested in where this is going!

 

[S3rgeus]

I'm reaching the end of my rope with this icon problem! I see other mods have solved this, so clearly I'm doing something wrong when trying to import new icon textures. I can get a new district into the game, but the icon doesn't show up. (The one on the tech - haven't tried to add an icon for the "choose production" menu.) A variety of changes also seem to lead to crashes in one of the following situations: while on the DoM screen, when opening the choose production menu if the new district is a valid option, or browsing to the new district in the Civilopedia. (The last one seems to happen 100% if the district doesn't require a tech.) It would be great to be able to get more information out of these game crashes to point me in the right direction, because they're all fairly mysterious atm. (Even just the exception message would probably go a long way.)

I'm going to be banging my head against this some more tomorrow, but in case anyone can resolve my undoubtedly simple oversight before then, I've attached what I've got so far to this post.

 

[counterpoint]

any luck?!

 

[Wolfdog]

I'm reaching the end of my rope with this icon problem! I see other mods have solved this, so clearly I'm doing something wrong when trying to import new icon textures. I can get a new district into the game, but the icon doesn't show up. (The one on the tech - haven't tried to add an icon for the "choose production" menu.) A variety of changes also seem to lead to crashes in one of the following situations: while on the DoM screen, when opening the choose production menu if the new district is a valid option, or browsing to the new district in the Civilopedia. (The last one seems to happen 100% if the district doesn't require a tech.) It would be great to be able to get more information out of these game crashes to point me in the right direction, because they're all fairly mysterious atm. (Even just the exception message would probably go a long way.)

I'm going to be banging my head against this some more tomorrow, but in case anyone can resolve my undoubtedly simple oversight before then, I've attached what I've got so far to this post.

In case you are still looking into this I had a quick look and from what I can see (bear in mind I only have experience in adding unit icons) and it looks like there is no mention of Scrollery.artdef in your Mod.Art.xml file. Also you may need another artdef like the one here for unit icons.

<?xml version="1.0" encoding="UTF-8" ?>
<AssetObjects::ArtDefSet>
    <m_Version>
        <major>3</major>
        <minor>0</minor>
        <build>215</build>
        <revision>207</revision>
    </m_Version>
    <m_TemplateName text="UserInterfaceBLPs"/>
    <m_RootCollections/>
    <m_BLPReferences>
        <Element>
            <xlpFile text="ScrolleryIcons.xlp"/>
            <blpPackage text="UI/Icons"/>
            <xlpClass text="UITexture"/>
        </Element>
    </m_BLPReferences>
</AssetObjects::ArtDefSet>

 

[S3rgeus]

any luck?!

Sorry for the delay, I've been super strapped for time to sit down and work on this properly.

In case you are still looking into this I had a quick look and from what I can see (bear in mind I only have experience in adding unit icons) and it looks like there is no mention of Scrollery.artdef in your Mod.Art.xml file. Also you may need another artdef like the one here for unit icons.

<?xml version="1.0" encoding="UTF-8" ?>
<AssetObjects::ArtDefSet>
    <m_Version>
        <major>3</major>
        <minor>0</minor>
        <build>215</build>
        <revision>207</revision>
    </m_Version>
    <m_TemplateName text="UserInterfaceBLPs"/>
    <m_RootCollections/>
    <m_BLPReferences>
        <Element>
            <xlpFile text="ScrolleryIcons.xlp"/>
            <blpPackage text="UI/Icons"/>
            <xlpClass text="UITexture"/>
        </Element>
    </m_BLPReferences>
</AssetObjects::ArtDefSet>

That was definitely something I was missing! Thanks, Wolfdog! I didn't realize there were two distinct sections in the Mod.Art.Xml and figured the Artdefs were referenced in the project itself (and therefore the modinfo) and that would be enough.

I'm now getting errors about my new Landmarks artdef not loading correctly, which is a big step up from before where it would just silently not work. I'll be combing through that this evening to see if I can find the root of the problem.

I've added a new artdef for the icons as you suggested and have added a similar reference to that. According to the art log that one *does* load correctly, but I'm still not getting my icon in game. More investigation required!

 

[StormingRomans]

You're XLP isn't quite right and also, based on my experience, you need two XLPs. So, .... in your current XLP change the package name to 'UI/Icons', and then create an exact copy and give it a package name of 'UI/UI_Icons' instead. Basically, you need to use the same package names as the game currently uses (although the file name for the XLP can be anything you want).

In you Mod.Art.xml change your reference to 'UI/ScrolleryIcons.blp' to 'UI/Icons.blp' and then add another one immediately under it to 'UI/UI_Icons.blp'. Should look like this:

<Element>
    <libraryName text="UITexture"/>
    <relativePackagePaths>
        <Element text="UI/Icons.blp"/>
        <Element text="UI/UI_Icons.blp"/>
    </relativePackagePaths>
</Element>

Two things to remember for these BLPs

1. The package names need to correspond to one's that the game already uses
2. In your Mod.Art.xml you reference the output BLP (not the XLP) and you use the package name (not the file name)

That should be it ... you don't need ArtDef files for icons in my experience. The XML files all look good to me.

One thing I've found is that ModBuddy sometimes doesn't always generate the dependency file when you build your mod, which means your BLPs are basically useless as the game doesn't know about them. The workaround I use is to use Asset Editor to create an empty Units ArtDef, and add that into the Mod.Art.xml file

1. ArtDef references go into artConsumers elements
2. BLPs into gameLibraries elements

<Element>
    <consumerName text="Units"/>
    <relativeArtDefPaths>
        <Element text="My_Empty_Units.artdef"/>
    </relativeArtDefPaths>
    <libraryDependencies>
        <Element text="Unit"/>
        <Element text="VFX"/>
        <Element text="Light"/>
    </libraryDependencies>
    <loadsLibraries>true</loadsLibraries>
</Element>

There doesn't appear to be any references to ArtDefs in your Mod.Art.xml. Are you're adding custom graphics for buildings, districts, etc? If so, there's a whole load of entries you need in here.

This is what's worked for me.

 

[StormingRomans]

Just looked at this a bit more, I've not done a custom district but I have done custom buildings that re-use existing building graphics. I'm guessing they won't be that different.

You'll need 2 artdefs - one for the 'Landmark' and one for the 'District' - I can only see one in the ZIP. As a starting point, I'd say the easiest thing to do would be to copy the settings for one of the existing districts from the game's Landmarks.artdef and Districts.artdef.

Like I said in the previous post, there'll be loads of places in the Mod.Art.xml that references to these artdefs need to go. For my custom buildings, I figured it out by opening the Civ6.dep file in Notepad++ and just looking to see which elements these artdefs were in, and then added them to the corressponding one's under the 'artConsumers' element in the Mod.Art.xml

 

[S3rgeus]

It lives!







My prevailing mistake was the lack of "(Mod Art Dependency File)" in the "UpdateArt" action in the project! Once I added that (after many hours of tearing my hair out when nothing else was working), then the icons started to work!

The district 3D model took an additional bit of fiddling, but I was able to author it completely in the Asset Editor from the information here. This is straight up reusing the Campus model. I haven't managed to create a new "landmark" that reuses the existing assets (which would look identical to this) - that turns up invisible, but there are many useful things I can try now that the game is actually loading my stuff.

Thanks for the guidance as well, StormingRomans! I've tried out a few of the things you recommended and it doesn't look like you need the second XLP for the icons. (Works fine for me without it now.) I'm going to be experimenting with naming and the like as well.

I've also attached the mod files for the above as well. Note that the Landmarks.artdef is not actually referenced at the moment, because Districts.artdef references DISTRICT_CAMPUS from the built-in landmarks directly.

I'm supremely happy that this is working after so long not getting anywhere. I'm going to try and get the custom landmark definition (reusing existing assets) working next. Then I'll see about some new 3D model work on a district.

EDIT: Scratch that, fixed the landmarks issue! Custom district here we come! (New files attached with the working landmark.)

 

[counterpoint]

cool!

 

[counterpoint]

so where we at? how are you feeling with all of this?

 

[S3rgeus]

so where we at? how are you feeling with all of this?

Sorry for the delay, I've been quite distracted! Overall I'm really liking CiVI's art pipeline - it actually works and seems to be usable for some fairly comprehensive changes, based on both the tweaking I've done here and the other work I've seen going on on the forum. The modifiers system is really modular and great for mix-and-match mods, but I'm still feeling like we'd need source access to start working on WoTMod in truth in CiVI. The more I've been thinking about where it looks like CiVI is going in terms of mod capabilities, I'm thinking it's the better fit for us once we've got the source we need. It's a bit of a gamble to shift over to designing against CiVI though, since we don't know when that will be available.

It's a quandary I've been thinking about a lot for the last month, but still don't have any hard answers to. (I'm hoping Firaxis releases the source and makes this much less of a difficult decision. If you guys are listening, we love you and gameplay source - or even an announcement of when the source is coming - would help us out tremendously!)

I've yet to see any other major technical blockers for us and a lot of cool art-related wins.

This coming weekend I'm going to dive into creating a small scenario with some custom objectives and see whether I hit any roadblocks in assembling a set of simpler gameplay mechanics. (Most of the tinkering I've done so far has been art related, since that's where CiV fell down.)

 

[counterpoint]

ok, cool, I guess! Glad to hear there aren't any major evident roadbloacks, at least. Hopefully they announce when it will be released so we can make this decision instead of forever kicking the ball down the field....

good luck on the scenario! hope it works out.

 

[S3rgeus]

All right, I've spent some time tinkering! I couldn't find much guidance on the forums on how to set up a new scenario, so I was mostly discovering stuff as I went. I bought the Poland DLC to have an example to work from that I knew functioned correctly. (Firaxis does seem to have implemented their own DLC through their modding pipeline, which is cool!)

The Poland scenario was very helpful. There's still some stuff that I'm not doing right with my scenario map. There are two civs, but bizarrely the human player seems to only be able to choose the AI civ from the set up menu. (The "Human" civ is disallowed because they're a "duplicate".) All of the major civs are marked as Human in the Poland scenario, so I'm missing something there. But it's fine for now, you can get into the custom map (I like the new WorldBuilder!) by selecting England and the menu limits you to those two choices, at least!

Having spent some more time looking into the Modifier system, I'm generally a big fan. But I can see where the limitations are that we'll run up against trying to define entirely new rules. Once you get to the "Effects" level, it all appears to be inside the DLL (so no new "Effects" until source access).

The new AI stuff, despite only having taken a cursory look at it, seems like it might actually be super powerful. It looks like it should be possible to do a lot more than I expected there.

I also had to spend some time this afternoon actually playing CiVI to get a better handle on espionage, since I'm making that a primary component of the new scenario. The general idea behind the scenario is to create an espionage-focused war between two players - capturing cities and disrupting your opponent via spies instead of troops. We'll see how possible that is - I'm still getting my head around the Operations system and what can be triggered with it.

I was caught out again by lack of error messages from CiVI itself. DLL source access would have let me attach a debugger and see where things were going wrong, but as it stands now it took me 2 hours to work out that one of my gameplay lua scripts happened to have the same file name as one of the lua scripts in the base game. Apparently this manifests as the game hanging during first load. That wasn't fun to debug and had me tearing my hair out again, undoing seemingly unrelated changes until I finally tracked down that change as the culprit. Would have only taken a couple of minutes with a proper error message!

So most of this was to get the concept of "Scenarios" going correctly. I've been poking around in the base game databases to find how things fit together and it's all making more sense. There is some serious flexibility in some of this stuff, which is really cool. And as I mentioned above, I quite like the new WorldBuilder so far. I'll continue tinkering on weeknights this week (will post an update or two here), but I'm also traveling all next week, back on the 19th.

I would also massively appreciate if there was a command line arg or something that let me skip the logos and the "Continue" screen on boot. I appreciate at least some of that is loading time, but I die inside waiting for those screens to go away and switching to other windows often means I forget and get sidetracked looking into something else rather than what I actually booted the game to check.

Another thing that would be great if anyone could confirm is if rebuilt mods are reloaded from the main menu. If I didn't need to even close the game, that would save me a ton of time. I'll try out some changes to see if I can work this out myself, but until I have a better handle on how the changes I'm making affect the game, it's difficult to confirm one way or the other since I don't know if my changes will work in the first place.

Another cool PSA, that I should probably post in a separate thread, is that .civ6map files are SQLite databases! You can open them up in SQLite Personal or some equivalent program and inspect their contents (including the ones from Firaxis, which show some cool metadata options). It's difficult to interpret some information like plot indices ("Huh, I wonder which plot 237 is"), but it's a good quick debug view of the map ("Does it have any Districts in it?" and similar questions can be answered quickly).

Plus, the lack of a scenario tutorial here on CivFanatics is probably a niche I could fill after I've gone through this process since the scenario creation tools within CiVI seem to be pretty powerful. And as I mentioned at the beginning of this post, I've yet to see a tutorial on this here!

 

[counterpoint]

All right, I've spent some time tinkering! I couldn't find much guidance on the forums on how to set up a new scenario, so I was mostly discovering stuff as I went. I bought the Poland DLC to have an example to work from that I knew functioned correctly. (Firaxis does seem to have implemented their own DLC through their modding pipeline, which is cool!)

The Poland scenario was very helpful. There's still some stuff that I'm not doing right with my scenario map. There are two civs, but bizarrely the human player seems to only be able to choose the AI civ from the set up menu. (The "Human" civ is disallowed because they're a "duplicate".) All of the major civs are marked as Human in the Poland scenario, so I'm missing something there. But it's fine for now, you can get into the custom map (I like the new WorldBuilder!) by selecting England and the menu limits you to those two choices, at least!

super cool! Nice to see that you're learning so much.

Having spent some more time looking into the Modifier system, I'm generally a big fan. But I can see where the limitations are that we'll run up against trying to define entirely new rules. Once you get to the "Effects" level, it all appears to be inside the DLL (so no new "Effects" until source access).

The new AI stuff, despite only having taken a cursory look at it, seems like it might actually be super powerful. It looks like it should be possible to do a lot more than I expected there.

Could you remind me what the "modifier system" refers to, specifically? I'm sure we went over this months ago. Which of the elements is it, just so I'm getting the big picture. Is this the way units/graphics are adjusted in real-time or something?

I also had to spend some time this afternoon actually playing CiVI to get a better handle on espionage, since I'm making that a primary component of the new scenario. The general idea behind the scenario is to create an espionage-focused war between two players - capturing cities and disrupting your opponent via spies instead of troops. We'll see how possible that is - I'm still getting my head around the Operations system and what can be triggered with it.

Interested to see how the espionage-war shakes up. I've already written about the issues with it, so i'll be interested to see if you're able to make things defendable in a meaningful way without gimping the mechanic.

(the rest of it)

hmmm, I wonder why they would go with more limited "feedback" systems that you're describing here.

happy modding! Good luck with all this...

 

[Deliverator]

I would also massively appreciate if there was a command line arg or something that let me skip the logos and the "Continue" screen on boot. I appreciate at least some of that is loading time, but I die inside waiting for those screens to go away and switching to other windows often means I forget and get sidetracked looking into something else rather than what I actually booted the game to check.

You can skip the logos at least by overwriting logos.bk2 with the blank bik file from the first comment here.

Another thing that would be great if anyone could confirm is if rebuilt mods are reloaded from the main menu. If I didn't need to even close the game, that would save me a ton of time.

From experience anything relating to the main gameplay data rather than the Front End/Config data will reload each time you start a new game so you can just go back to the main menu and restart once you've built out your changes. The things that don't seem to reload are the Front End/Config database changes and cooked assets (.blp files) - these require you restart Civ 6 completely.

 

[S3rgeus]

Could you remind me what the "modifier system" refers to, specifically? I'm sure we went over this months ago. Which of the elements is it, just so I'm getting the big picture. Is this the way units/graphics are adjusted in real-time or something?

The modifier system is a gameplay system, rather than graphics. It's about how CiVI hooks up bonuses to actions in game. It's a lot more flexible than CiV in that it allows generic associations between any modifier bonus and any modifier trigger condition.

For example, in CiV, there's a table for having builds generate specific yields. This lets you set up a building to generate +X Culture per turn. Or +X Science per turn. Etc. In CiVI, there's an "add yield modifier". The building modifiers table associates it with specific buildings to get the same effect. But there are also government modifier tables and unit modifier tables. So now since they added yields for buildings, you can add yields to governments or units. So most bonuses are applicable to a lot more game entities.

(I don't know if the above specific example with yields actually works this way in CiVI, since I haven't tried it myself, but that's the general idea.)

Interested to see how the espionage-war shakes up. I've already written about the issues with it, so i'll be interested to see if you're able to make things defendable in a meaningful way without gimping the mechanic.

Yep, I'll definitely be referring to your post about espionage difficulties!

hmmm, I wonder why they would go with more limited "feedback" systems that you're describing here.

It's not so much that they opted into it, it's that they didn't do more work to provide the verbosity that would make these things clearer. It's mostly the nature of the beast when working with applications that you can't actually debug.

happy modding! Good luck with all this...

Thanks! Unfortunately I haven't had much time on the weeknights this week and I'm flying off tomorrow (we'll be quite a bit closer in physical reality!). I'll be back on Monday the 19th.

You can skip the logos at least by overwriting logos.bk2 with the blank bik file from the first comment here.

Awesome! Thanks, Deliverator! I'll definitely try that out!

From experience anything relating to the main gameplay data rather than the Front End/Config data will reload each time you start a new game so you can just go back to the main menu and restart once you've built out your changes. The things that don't seem to reload are the Front End/Config database changes and cooked assets (.blp files) - these require you restart Civ 6 completely.

Cool, that makes sense. InGameActions getting reloaded when entering a game from the main menu sounds quite sensible. I'll keep this in mind!