Mod switcher

[tdb]

It's great that this game is as moddable as it is, but one major shortcoming of the system is the inability to easily combine mods. I got fed up with hunting and merging changes by hand before I even started it. What we need is a mod switcher and since there are apparently none available yet, I decided to take up the job.

The basic idea is to store each mod as a set of changes instead of complete files. I will store these in a single XML file. It will be possible to convert a "standard" mod to this format with the program. The main window will have a list of all available mods with checkboxes to enable or disable them. When you click the "Apply" button, it will compile modified versions of all relevant files in the CustomAssets directory based on the activated mods.

Last updated 2006-07-04

Homepage: http://civ4.tdb.fi
Download: http://civ4.tdb.fi/downloads/civ4modder-0.5-win32-bin.zip

Changes since last version:
- Add support for more xml files (thanks Impaler)
- No conflict for two mods adding a file with exactly same contents
- Handle containers/objects with duplicate elements correctly
- Remove the previous entry when recreating a mod
- Bugfixes
-- The conflict resolution dialog actually has some effect now
-- Don't treat CR and LF as binary when determining need for base64 encoding

Notes:
- You will need to install the gtkmm runtime environment in order to use this program. You can get it from http://www.pcpm.ucl.ac.be/~gustin/win32_ports/
- Remember to specify the directories in the Preferences dialog before creating/using mods

Known problems:
- Recreating a mod doesn't remove the old instance from the list. Restart the program to clean it up.
- There are some XML files that are not supported yet. This is because the program needs some basic knowledge about which elements are containers and what's the key element in definitions. Please report any non-working things and I'll add support for them.

If you find any problems that are not on the above list, please report them with as much accuracy as you can. I need a detailed description on how to reproduce the problem and the versions of the involved mods at least. If the problem occurred when applying mods, it may be helpful to send the .Civ4Mod files for all enabled mods.

Todo for 1.0

- Add a feature for direct merging of a bunch of mods
- Support all of the most used XML files

Maybe:
- Split python files into functions and diff each one separately
- Mod dependencies
- Add support for temporarily disabling context for .py files (useful for imports, but will modders use it?)

Todo for 2.0

- Built-in mod editor
-- Edit units, civilizations and such with a nice interface
-- Edit python files with changes highlighted (removed lines in red, added in green)
-- Import / export dds files

How to install the gtkmm runtime environment

Download these two files (These files do not reside on my server so I cannot guarantee the links will stay valid):
http://gladewin32.sourceforge.net/modules/wfdownloads/visit.php?cid=10&lid=84 (This should result in gtk-win32-2.6.10-rc1.exe after the redirection and mirror selection)
http://www.pcpm.ucl.ac.be/~gustin/win32_ports/binaries/gtkmm-runtime-2.6.2-1.exe

Install the first one (gtk-win32-2.6.10-rc1.exe) and reboot when prompted.
Install the second one (gtkmm-runtime-2.6.2-1.exe).
You should now be able to run civ4modder.

Civ4Modder for dummies

After installing the program, the first thing to do is specify the directories in the Preferences dialog. Civ4 directory should be set to the directory the game is installed to ("C:\Program Files\Firaxis Games\Sid Meier's Civilization 4" by default). The program expects to find an Assets directory with the original datafiles in this directory.

Custom assets directory should generally be an Assets directory under some mod directory. Something like "C:\Program Files\Firaxis Games\Sid Meiers Civilization 4\Mods\MyCombinationMod\Assets" if you've installed the game into the default directory. You can also use the CustomAssets directory in My Documents, if you want to get the combination mod enabled permanently.

With the directories set, you can start combining mods. This is a two-step process. First you need to create difference files for all the mods you want to combine. Select Create Mod from the Mod menu to bring up the mod creation dialog. For the mod directory, you should specify the directory containing the mod's .ini file. If such a file was found, the name and description fields will be pre-filled for you. Click the Ok button when you're happy with the value to create the difference file.

When you have created a bunch of difference mods, you can use the checkboxes on the list to enable/disable particular mods. Press the Apply button to create a combination mod in your custom assets directory. If all goes well, there won't be any conflicts between the mods. If there are, you will (for now) need to disable one of each conflicting pair.

 

[BeefontheBone]

Best be careful to make sure two or more mods aren't trying to alter the same bits of files, but this is a quality plan, particularly if it'll help run more than one mod which alters the same files.

 

[Coopa]

Yeah a good idea. I think the main problem would come if someone changed the stats of the same thing. Fine if it merges the document but I'm sure you can see where the errors will come (e.g both mods change requirements for gunpowder, stats for swordsmen)

Twil be a very good program if it works well

Good Luck

 

[Psyringe]

When mods get more complex, even a utility that just lists the files that are changed in *both* mods you want to merge will be a godsend.

It may be spossible to automate the processing of changes to the same file, at least to a degree. Merging files with just added units or civilizations should be doable, but should produce a warning message as some routines of the mod *may* rely on certain units appearing on certain places of the list.

But as I said, even a program with minimal functionality in this regard will be useful. Go for it, and see how far you can push it.

 

[tdb]

I think I will actually include the XML structure in the changes. That will give maximum robustness against other changes in the files, like game patches. For python files it'll be something similar to the GNU diff/patch utilities, and graphics and other binary files are treated as whole files (but till checked for conflicts).

 

[woodelf]

You could be a HERO! Good luck on your endeavour. After merging 3 mods myself recently with mixed results I can't wait for your (hopeful) success.

 

[Lehawk]

Idea

Scan the original files for changes in the mod's files and create patch files. This is how version control software works. The patch files would be a special type of mod that you can put together in a special mod folder based on which mods the player choses. Then run the game using that mod folder, after changing the configuration file to use it.

Of course if patch files collide(on line by line basis), you'll have to find a way to make it work or just not let the player choose them together.

 

[tdb]

Lehawk: That would require me to keep duplicates of the original files somewhere... Granted, there's only 12MB of XML files in the game, so it wouldn't be that much extra space. But I think it will work just fine if I include the XML structure in the mod, so it can say things like "Change the strength of the Swordsman unit to 7". I'll keep that idea in mind though and re-evaluate it once we have an actual game patch from Firaxis.

Rufus: Did you read my the my second post to this thread? I already noticed that this is in the wrong forum and asked for it to be moved to the correct one.

 

[tdb]

Just a quick status report, and a bump in hopes of attracting more attention.

I'm making good progress with this. The program can already process XML and graphics files and create a single mod file out of them. Currently I'm working on the part which will merge the mod with the existing datafiles.

I will definitely release a first version for testing this week. It will not have everything I've planned to add (like conflict detection), but it will work.

I also have a message to the modders out there: If you use non-ASCII characters, please make sure that the affected files have encoding="latin1" in the XML descriptor tag. The XML lib I'm using has utf8 as the default encoding and freaks out if it sees a sequence of bytes it can't interpret with the active encoding.

 

[tdb]

Okay, I just hit a problem... It appears that the game doesn't look for CustomAssets directory in its installation directory. I had originally planned to apply the mods there, so now I need an alternative strategy.

I've figured out two possibilities:
1. Add a configuration option for it, or
2. Apply them as a compilation mod in the Mods directory

Option 2 has the drawback that the game doesn't remember the active mod across restarts. Furthermore, option 1 kinda includes also option 2, since you can specify a directory like Civ4Directory/Mods/MyCompilationMod/Assets. It does require more work from the user though.

What do others think of these choices? Does anyone has better ideas?

 

[Skippa]

Can't you just use option 2?

EG. Lets say I have 3 mods configurations. I could just make Mod 1+2 and Mod 1+3 and save them to the folders Mod1_Mod2 and Mod1_Mod3 respectively.

True the game doesn't rememeber but one can easily load the mod at startup or just cut and paste the files to the CustomMods folder (I do realize that this option would be counter to the whole idea of making the program in order to make things simpler ). I just would rather have a program that could be used to make many different mods (an not overwrite the last one I made) since I like to play many different mods. I think that if you could make the option of placing the files in the mods folder or in the CustomAssets folder that would be best but it seems that that isn't possible right now.

 

[Shadowlord]

I'd say just provide a way to merge mods and put the merged one in the Mods folder. If the user wants to put it in CustomAssets, they can go into the Mods folder and grab what they need - or, your program could look for "%userdata%\My Games\Sid Meier's Civilization 4" - If it's not there, ask the user what their Civ IV folder is named (Some people on IRC said the folder name in the My Games folder differed if your Civ IV folder was named differently).

 

[tdb]

I could make it have multiple mod sets, one of which is the permanent one. That'll come later though, I think I'll start with only one set.

 

[tdb]

I'm proud to announce that I am just about ready to release the first version of this program. The only thing left to do is to port it to Windows and verify it works there too. It's a matter of hours now until I put the file up.

 

[Supa]

If you want to merge two mods, it's probably because they change small things.

Until now, I haven't uploaded any XML files with my units, only an installation guide listing all the changes to be made.

And I listed them as an PHPBB install guide.

--Open--
blabla.xml

--Find--
Importantline

--Before, Add--
Beautifuladdition

--Save--

Like this it's easy to manually merge small mods and it opens the way for an automated software (like EazyPHP).

 

[Plankhead]

If you want to merge two mods, it's probably because they change small things.

Until now, I haven't uploaded any XML files with my units, only an installation guide listing all the changes to be made.

And I listed them as an PHPBB install guide.

...

Like this it's easy to manually merge small mods and it opens the way for an automated software (like EazyPHP).

But what about some of the larger mods that will eventually appear, like massive packs of units and civs? Suppose you want to add other mods to them?

 

[tdb]

"Easy" is such a relative term. An average mod adding one new civilization modifies about 10 XML files. To add 10 new civilizations, you will need to do 100 modifications. If we assume one modification can be done in 30 seconds, you will spend almost an hour merging the mods. Furthermore, this is repetitive, boring, and somewhat error-prone to do. With this program, it will only take about two minutes.

It will also potentially make the life of mod creators easier. You no longer need to keep track of all the changes you've made - once you're content with the mod, feed it to this program and it will find out the changes for you. In a future version, I'll also add an editor which allows you to review your changes and easily detect and remove anything that you don't want in the final mod (like if you've given yourself god-like powers to test your modern-age UU).

 

[Supa]

That's the problem. Larger the mod is, higher is the probabilities that the creators changed many things, making an automatic process likely to fail.

If the creators have been a little creative with the Python files, it would be hard to merge, especially with the .py files.

But if a mod, even large, just add civs and units, it would change nothing the XML structure. My installation guides will still be good and an automated program would probably work with them.

Merging "ready to play mod" without list of changes is just a pain in butt. It's the very reason I haven't installed the new civs by CivArmy yet.

It's for these reasons that I give no "Out-of-the-box" mod to play with my modifications but rather a guide to install them. That way, everyone can see immediatly what I've done and integrate my mods, or units, civs or whatever I could make easily in their mods.

Edit: I'm maybe a slight off-topic here. I'm quite tired so, if it is the case, excuse me

 

[tdb]

That's the problem. Larger the mod is, higher is the probabilities that the creators changed many things, making an automatic process likely to fail.

My goal is to make the automation so robust that it does not fail. It will either successfully apply the mod or report that it can't be applied due to conflict with another mod. In the latter case, not even applying by hand with the instruction would help because of the conflict. The user would have to decide which mod's changes to keep. I might add this kind of conflict-resolving functionality too.

Undoubtedly, getting the algorithm robust enough will involve a lot of trial-and-error testing. But that's where the community comes in. If there are any bugs in the program, we'll be sure to hit them sooner or later, given the large number of different machines and installations people have.

If the creators have been a little creative with the Python files, it would be hard to merge, especially with the .py files.

Nope. My program creates a set of differences very similar to the standard GNU diff tool. This has been proven to work, no matter how complex the files. Of course, being overly creative with the .py files (like swapping two pieces of code because it "looks better that way") will drastically increase the chances of getting a conflict with another mod.

Merging "ready to play mod" without list of changes is just a pain in butt. It's the very reason I haven't installed the new civs by CivArmy yet.

This is precisely the problem I'm trying to address with this program. The user can download a mod which is ready to play as-is, but he/she will also be able to easily combine it with other mods. This doesn't even need any kind of knowledge of XML

 

[tdb]

I hit some major problems with the Windows port. I'm using Gtkmm for the user interface and it appears that there aren't any prepackaged binary versions that are fully compatible with MinGW's version of GCC. Thus, I will have to compile a lot of stuff from scratch. Unfortunately, it's rather late here (3:30am) and I need to sleep. I will continue in the morning.

Meanwhile, if someone else wants to try his or her luck, here are the sources: http://databeaver.net/~tdb/civ4modder.tar.gz
It requires my mspstreams library, which can be obtained from http://mikkosoft.databeaver.net/libs/streams
Gtkmm can be fount at http://www.gtkmm.org