It's great that this game is as moddable as it is, but one major shortcoming of the system is the inability to easily combine mods. I got fed up with hunting and merging changes by hand before I even started it. What we need is a mod switcher and since there are apparently none available yet, I decided to take up the job.

The basic idea is to store each mod as a set of changes instead of complete files. I will store these in a single XML file. It will be possible to convert a "standard" mod to this format with the program. The main window will have a list of all available mods with checkboxes to enable or disable them. When you click the "Apply" button, it will compile modified versions of all relevant files in the CustomAssets directory based on the activated mods.

Last updated 2006-07-04

Homepage: http://civ4.tdb.fi
Download: http://civ4.tdb.fi/downloads/civ4modder-0.5-win32-bin.zip

Changes since last version:
- Add support for more xml files (thanks Impaler)
- No conflict for two mods adding a file with exactly same contents
- Handle containers/objects with duplicate elements correctly
- Remove the previous entry when recreating a mod
- Bugfixes
-- The conflict resolution dialog actually has some effect now
-- Don't treat CR and LF as binary when determining need for base64 encoding

Notes:
- You will need to install the gtkmm runtime environment in order to use this program. You can get it from http://www.pcpm.ucl.ac.be/~gustin/win32_ports/
- Remember to specify the directories in the Preferences dialog before creating/using mods

Known problems:
- Recreating a mod doesn't remove the old instance from the list. Restart the program to clean it up.
- There are some XML files that are not supported yet. This is because the program needs some basic knowledge about which elements are containers and what's the key element in definitions. Please report any non-working things and I'll add support for them.

If you find any problems that are not on the above list, please report them with as much accuracy as you can. I need a detailed description on how to reproduce the problem and the versions of the involved mods at least. If the problem occurred when applying mods, it may be helpful to send the .Civ4Mod files for all enabled mods.

Todo for 1.0

- Add a feature for direct merging of a bunch of mods
- Support all of the most used XML files

Maybe:
- Split python files into functions and diff each one separately
- Mod dependencies
- Add support for temporarily disabling context for .py files (useful for imports, but will modders use it?)

Todo for 2.0

- Built-in mod editor
-- Edit units, civilizations and such with a nice interface
-- Edit python files with changes highlighted (removed lines in red, added in green)
-- Import / export dds files

How to install the gtkmm runtime environment

Download these two files (These files do not reside on my server so I cannot guarantee the links will stay valid):
http://gladewin32.sourceforge.net/modules/wfdownloads/visit.php?cid=10&lid=84 (This should result in gtk-win32-2.6.10-rc1.exe after the redirection and mirror selection)
http://www.pcpm.ucl.ac.be/~gustin/win32_ports/binaries/gtkmm-runtime-2.6.2-1.exe

Install the first one (gtk-win32-2.6.10-rc1.exe) and reboot when prompted.
Install the second one (gtkmm-runtime-2.6.2-1.exe).
You should now be able to run civ4modder.

Civ4Modder for dummies

After installing the program, the first thing to do is specify the directories in the Preferences dialog. Civ4 directory should be set to the directory the game is installed to ("C:\Program Files\Firaxis Games\Sid Meier's Civilization 4" by default). The program expects to find an Assets directory with the original datafiles in this directory.

Custom assets directory should generally be an Assets directory under some mod directory. Something like "C:\Program Files\Firaxis Games\Sid Meiers Civilization 4\Mods\MyCombinationMod\Assets" if you've installed the game into the default directory. You can also use the CustomAssets directory in My Documents, if you want to get the combination mod enabled permanently.

With the directories set, you can start combining mods. This is a two-step process. First you need to create difference files for all the mods you want to combine. Select Create Mod from the Mod menu to bring up the mod creation dialog. For the mod directory, you should specify the directory containing the mod's .ini file. If such a file was found, the name and description fields will be pre-filled for you. Click the Ok button when you're happy with the value to create the difference file.

When you have created a bunch of difference mods, you can use the checkboxes on the list to enable/disable particular mods. Press the Apply button to create a combination mod in your custom assets directory. If all goes well, there won't be any conflicts between the mods. If there are, you will (for now) need to disable one of each conflicting pair.

Best be careful to make sure two or more mods aren't trying to alter the same bits of files, but this is a quality plan, particularly if it'll help run more than one mod which alters the same files.

Yeah a good idea. I think the main problem would come if someone changed the stats of the same thing. Fine if it merges the document but I'm sure you can see where the errors will come (e.g both mods change requirements for gunpowder, stats for swordsmen)

Twil be a very good program if it works well

Good Luck

When mods get more complex, even a utility that just lists the files that are changed in *both* mods you want to merge will be a godsend.

It may be spossible to automate the processing of changes to the same file, at least to a degree. Merging files with just added units or civilizations should be doable, but should produce a warning message as some routines of the mod *may* rely on certain units appearing on certain places of the list.

But as I said, even a program with minimal functionality in this regard will be useful. Go for it, and see how far you can push it. :)

I think I will actually include the XML structure in the changes. That will give maximum robustness against other changes in the files, like game patches. For python files it'll be something similar to the GNU diff/patch utilities, and graphics and other binary files are treated as whole files (but till checked for conflicts).

You could be a HERO! Good luck on your endeavour. After merging 3 mods myself recently with mixed results I can't wait for your (hopeful) success.

Idea

Scan the original files for changes in the mod's files and create patch files. This is how version control software works. The patch files would be a special type of mod that you can put together in a special mod folder based on which mods the player choses. Then run the game using that mod folder, after changing the configuration file to use it.

Of course if patch files collide(on line by line basis), you'll have to find a way to make it work or just not let the player choose them together.

Lehawk: That would require me to keep duplicates of the original files somewhere... Granted, there's only 12MB of XML files in the game, so it wouldn't be that much extra space. But I think it will work just fine if I include the XML structure in the mod, so it can say things like "Change the strength of the Swordsman unit to 7". I'll keep that idea in mind though and re-evaluate it once we have an actual game patch from Firaxis.

Rufus: Did you read my the my second post to this thread? I already noticed that this is in the wrong forum and asked for it to be moved to the correct one.

Just a quick status report, and a bump in hopes of attracting more attention.

I'm making good progress with this. The program can already process XML and graphics files and create a single mod file out of them. Currently I'm working on the part which will merge the mod with the existing datafiles.

I will definitely release a first version for testing this week. It will not have everything I've planned to add (like conflict detection), but it will work.

I also have a message to the modders out there: If you use non-ASCII characters, please make sure that the affected files have encoding="latin1" in the XML descriptor tag. The XML lib I'm using has utf8 as the default encoding and freaks out if it sees a sequence of bytes it can't interpret with the active encoding.

Okay, I just hit a problem... It appears that the game doesn't look for CustomAssets directory in its installation directory. I had originally planned to apply the mods there, so now I need an alternative strategy.

I've figured out two possibilities:
1. Add a configuration option for it, or
2. Apply them as a compilation mod in the Mods directory

Option 2 has the drawback that the game doesn't remember the active mod across restarts. Furthermore, option 1 kinda includes also option 2, since you can specify a directory like Civ4Directory/Mods/MyCompilationMod/Assets. It does require more work from the user though.

What do others think of these choices? Does anyone has better ideas?

Can't you just use option 2?

EG. Lets say I have 3 mods configurations. I could just make Mod 1+2 and Mod 1+3 and save them to the folders Mod1_Mod2 and Mod1_Mod3 respectively.

True the game doesn't rememeber but one can easily load the mod at startup or just cut and paste the files to the CustomMods folder (I do realize that this option would be counter to the whole idea of making the program in order to make things simpler :D). I just would rather have a program that could be used to make many different mods (an not overwrite the last one I made) since I like to play many different mods. I think that if you could make the option of placing the files in the mods folder or in the CustomAssets folder that would be best but it seems that that isn't possible right now.

I'd say just provide a way to merge mods and put the merged one in the Mods folder. If the user wants to put it in CustomAssets, they can go into the Mods folder and grab what they need - or, your program could look for "%userdata%\My Games\Sid Meier's Civilization 4" - If it's not there, ask the user what their Civ IV folder is named (Some people on IRC said the folder name in the My Games folder differed if your Civ IV folder was named differently).

I could make it have multiple mod sets, one of which is the permanent one. That'll come later though, I think I'll start with only one set.

I'm proud to announce that I am just about ready to release the first version of this program. The only thing left to do is to port it to Windows and verify it works there too. It's a matter of hours now until I put the file up.

If you want to merge two mods, it's probably because they change small things.

Until now, I haven't uploaded any XML files with my units, only an installation guide listing all the changes to be made.

And I listed them as an PHPBB install guide.

--Open--
blabla.xml

--Find--
Importantline

--Before, Add--
Beautifuladdition

--Save--

Like this it's easy to manually merge small mods and it opens the way for an automated software (like EazyPHP).

If you want to merge two mods, it's probably because they change small things.

Until now, I haven't uploaded any XML files with my units, only an installation guide listing all the changes to be made.

And I listed them as an PHPBB install guide.

...

Like this it's easy to manually merge small mods and it opens the way for an automated software (like EazyPHP).

But what about some of the larger mods that will eventually appear, like massive packs of units and civs? Suppose you want to add other mods to them?

"Easy" is such a relative term. An average mod adding one new civilization modifies about 10 XML files. To add 10 new civilizations, you will need to do 100 modifications. If we assume one modification can be done in 30 seconds, you will spend almost an hour merging the mods. Furthermore, this is repetitive, boring, and somewhat error-prone to do. With this program, it will only take about two minutes.

It will also potentially make the life of mod creators easier. You no longer need to keep track of all the changes you've made - once you're content with the mod, feed it to this program and it will find out the changes for you. In a future version, I'll also add an editor which allows you to review your changes and easily detect and remove anything that you don't want in the final mod (like if you've given yourself god-like powers to test your modern-age UU).

That's the problem. Larger the mod is, higher is the probabilities that the creators changed many things, making an automatic process likely to fail.

If the creators have been a little creative with the Python files, it would be hard to merge, especially with the .py files.

But if a mod, even large, just add civs and units, it would change nothing the XML structure. My installation guides will still be good and an automated program would probably work with them.

Merging "ready to play mod" without list of changes is just a pain in butt. It's the very reason I haven't installed the new civs by CivArmy yet.

It's for these reasons that I give no "Out-of-the-box" mod to play with my modifications but rather a guide to install them. That way, everyone can see immediatly what I've done and integrate my mods, or units, civs or whatever I could make easily in their mods.

Edit: I'm maybe a slight off-topic here. I'm quite tired so, if it is the case, excuse me :)

That's the problem. Larger the mod is, higher is the probabilities that the creators changed many things, making an automatic process likely to fail.

My goal is to make the automation so robust that it does not fail. It will either successfully apply the mod or report that it can't be applied due to conflict with another mod. In the latter case, not even applying by hand with the instruction would help because of the conflict. The user would have to decide which mod's changes to keep. I might add this kind of conflict-resolving functionality too.

Undoubtedly, getting the algorithm robust enough will involve a lot of trial-and-error testing. But that's where the community comes in. If there are any bugs in the program, we'll be sure to hit them sooner or later, given the large number of different machines and installations people have.

If the creators have been a little creative with the Python files, it would be hard to merge, especially with the .py files.

Nope. My program creates a set of differences very similar to the standard GNU diff tool. This has been proven to work, no matter how complex the files. Of course, being overly creative with the .py files (like swapping two pieces of code because it "looks better that way") will drastically increase the chances of getting a conflict with another mod.

Merging "ready to play mod" without list of changes is just a pain in butt. It's the very reason I haven't installed the new civs by CivArmy yet.

This is precisely the problem I'm trying to address with this program. The user can download a mod which is ready to play as-is, but he/she will also be able to easily combine it with other mods. This doesn't even need any kind of knowledge of XML

I hit some major problems with the Windows port. I'm using Gtkmm for the user interface and it appears that there aren't any prepackaged binary versions that are fully compatible with MinGW's version of GCC. Thus, I will have to compile a lot of stuff from scratch. Unfortunately, it's rather late here (3:30am) and I need to sleep. I will continue in the morning.

Meanwhile, if someone else wants to try his or her luck, here are the sources: http://databeaver.net/~tdb/civ4modder.tar.gz
It requires my mspstreams library, which can be obtained from http://mikkosoft.databeaver.net/libs/streams
Gtkmm can be fount at http://www.gtkmm.org

You know, this is increasingly looking like the sort of stuff CVS (Concurrent Versioning System) and other code repositories do. They have a feature which enables multiple persons to make changes to the same piece of code and update the code intelligently. In case of conflicts they can ask from the user what to do with particular line(s) of code. So, it is possible to make a program which can do the same thing to mods. Of course, some mods are innately incompatible and cannot be combined because they might, for example, change the rules of the game in different ways.

Cheers,
Uluk

There are dozens of free document merge programs available via the web, why write a new one?

You know, this is increasingly looking like the sort of stuff CVS (Concurrent Versioning System) and other code repositories do. They have a feature which enables multiple persons to make changes to the same piece of code and update the code intelligently. In case of conflicts they can ask from the user what to do with particular line(s) of code. So, it is possible to make a program which can do the same thing to mods. Of course, some mods are innately incompatible and cannot be combined because they might, for example, change the rules of the game in different ways.

Pretty much yes. I've actually replicated some of the behavior of GNU diff and patch in my program to handle python files.

There are dozens of free document merge programs available via the web, why write a new one?

Because this one has a user interface and other features specifically designed for Civ4 mods.

Take a look at what the GNU diff utility thinks of the CIV4LeaderHeadInfos.xml from FinMod (in the attached file). Looks rather sub-optimal, no? What do you think would happen if some other mod first added a leaderhead between Kublai Khan and Louis XIV and you then tried to apply that patch?

My program solves this problem by looking at the XML structure of the file. It treats the elements ar leaderhead definitions and not just some piece of text. When it sees Kekkonen in FinMod's file, it looks at the original file and notices it's missing. Thus, it stores the definition in the mod file. When it next sees Kublai Khan, it can find it from the original and notices they're exactly the same definition. Thus, there's no point in storing that.

You know, I'm getting so much negative responses here that I'm wondering if you actually want this program. I can just keep it to myself and let you merge your mods by hand or whatever diffing utility you like. I can take critique constructively, if there's actually something to construct with.

You know, I'm getting so much negative responses here that I'm wondering if you actually want this program. I can just keep it to myself and let you merge your mods by hand or whatever diffing utility you like. I can take critique constructively, if there's actually something to construct with.

Hint: Being thick-skinned is a definite plus when dealing with people on the internet. ;)
Some will appreciate your work and some won't, no matter what ... that's just the way it is.

I know... I've had this kind of "why reinvent the wheel" comments countless times before. As for the why, I like coding. It's a magnificent feeling when you finally see your project working.

BTW, I just got the program successfully compiled on Windows. Now I just need to kill some bugs before I release it.

First version released (download link in the first post)

Please report any problems and suggestions to this thread.

Maybe this should be moved to the "Utility programs" section now? Or should I create a new thread there?

The link for the download is bad

The link for the download is bad

Sorry, my mistake. It's fixed now.

You know, I'm getting so much negative responses here that I'm wondering if you actually want this program. I can just keep it to myself and let you merge your mods by hand or whatever diffing utility you like. I can take critique constructively, if there's actually something to construct with.

Now don't get me wrong tdb. If done properly, this could be great. That CVS thing just hit my mind because of a recent lecture at the uni I attended. And reinventing the wheel isn't necessarily that bad, it's a learning experience! In fact, I believe that a large amount of work people do includes some form of reinventing. Nobody's going to pull new stuff simply out of their hats, it has to be based on something.

Nevertheless, if you are going to make a program which practically every civ player can use, a simple document merging program is not going be enough. It has to have some civ specific functionality as well. Keep up the good work, tdb.

Cheers,
Uluk

I'm going to integrate some kind of mod editor with this. I'm not yet sure how complex. At first it will probably be a simple text editor, but in the future I might create a point-and-click interface with icons and everything.

If you have any ideas about what features this program should have, just tell me and I'll see what I can do.

I have no idea about the computer lingo you guys are using yet I knew the importance of it to read the whole post. :D Must be doing something right then. :lol:
I kind of follow what you are trying to do and making it more Civ specific (compared to other generic mergers) is wonderful. Seems like that idea is warranted considering all the people this series/game positively affects.
Next, the simple idea you had of a.)showing which lines of code conflict and b.) giving the user the option to choose, would be a fundamental place to start, IMH(Ignorant)O. After that, the next logical (but difficult) step could be prompts explaining to the non-'coders' what those conflicts were.

If you could do those three,...
1. Multiple Mod Merging (MMM... good)
2. Conflict prompts using 'coder' language
3. Lamen conflict-explanations
... what more could a Civer ask for? :goodjob:

People seem to bring up the topic of combining mods regularly, so would someone please make this sticky and/or move it to the Utility Programs forum?

There's a new version coming out today that will fi most of the problems mentioned in the first post.

I'll try to post a new thread in which we can keep track of good and bad combo's of mods. I was searching for something like this myself when i stumbled upon your prog (making me very happy and anxious to see the new version)

I will post a link towards this thread to introduce more people to your work.

Keep it up:goodjob:

Aye, this is of great importance! This is the tool we need... if only you can make it work right *crosses fingers*

Here it is - the new and shiny civ4modder 0.2! The most important new feature is undoubtedly the conflict checker. Other improvements include on-demand loading of the mod files for faster startup and the ability to select teh mods you want with the checkboxes in the list.

Download link has been updated in the first post.

Aye, this is of great importance! This is the tool we need... if only you can make it work right *crosses fingers*
Rest assured, with the help of you people I can make it work right. If you come up with some feature you'd like to have, there's a 99% chance that I can code it.

This is an absolutely superb idea. I've been looking for something like this so I can combine my different mods and easily add a few features that others have developed. No, I don;t have anything useful to say -- I just wanted to encourage further development :)

I'm not familiar with what a run-time environment is. And since others may agree, can you tell a little about what it is, why we need it, and a little more explanation on how to configure it?
Thanks for your modding contribution. I'm sure it will be a big hit!

I'm not familiar with what a run-time environment is. And since others may agree, can you tell a little about what it is, why we need it, and a little more explanation on how to configure it?
Thanks for your modding contribution. I'm sure it will be a big hit!

Basically, it's a collection of libraries to provide some user interface stuff the program needs. It's a lot easier to use (programming-wise) than WinAPI, and also is portable. Since I'm doing the actual development on Linux, a portable UI library is a must.

I've added detailed instructions for installing gtkmm to the first post.

Ok, i'm still applying the civ4modder, found the first bug, being human error...:rolleyes::blush: No serious consequences though.

I tried combining two mods and place the newly created modfile into the mods-map in the game's installation directory. I Started the game, went to load the new mod and found an XML mod and an art mod and I thought DOO how could i be so stupid. So now i treid the same thing over again but created the new mod-file in a new folder within the mods-folder. I thought this would be it and tried to load the mod. Loading the mod went ok, but when i tried to find the new civ (the netherlands was on of the two initial mods) i couldn't find them. So obviously something had went wrong.

When i wanted to redo the whole thing, i couldn't find the original mods... because civ4modder had somehow cleaned out the entire modsfolder.
So now i'm reinstalling the mods.

STILL not working for me!
I'm trying to combine the most recent versions of lost unit mod(by Frontbrecher) and the Netherlands(by Tremo). The Civ4modder creates a new modfolder (which i named Multiple) in the modfolder in the installation dir. In which i can see the new (two) leaderhead folders etc, and the new units (14 in total of which one is UU for netherlands) It looks to be ok, uploading the mod into the game also seems to go just fine. But when i try to find the new civ, it isn't available.

Could some one tell me what went wrong?

Okay, the civ4modder's CustomAssets directory definitely nees some more documentation.

The program uses that directory as the combination mod's assets directory. You'll generally want to use one of these choices (adapted to your setup):
C:\Documents and Settings\Username\My Documents\My Games\Civilization 4\CustomAssets
C:\Program Files\Firaxis Games\Sid Meiers Civilization 4\Mods\MyCombinationMod\Assets

Your second attempt didn't work because the game looks for an Assets folder in the mod folder.

Also, when applying the mods, the program deletes the entire destination directory to ensure you'll only get those mods you have selected. So that's why your Mods folder disappeared.

Wait a seccond, does this app combine python files/custom art or is it XML only?

I don't see how combining custom pyton files could ever end well.

Creating new mod went ok, i understood and implemented the assets part of your explanation.

Now loading the combi-mod in CIV4 gave me a lot of XML load errors. The lost units mod and the Netherlands mod in total give 14 new units, each new unit generated a XML error. It clearly has difficluty with new units.

The new civ was playable so no problems there.


is there a quick fix? or should i forget about new units...:sad:

But we're clearly getting a great enhancement to the game with this prog!!! :goodjob:

EDIT: Got some error specs (Win.)

First error: failed loading xml file XML\Units/CIV4UnitInfos.xml. [FXml.cpp:133] Error parsing XML file reason: Element is incomplete according to the DTD/Schema. line 159,83

then another err about the unitinfos file

From then on, all new units give an error as in this example:
Tag:UNIT_AMERICAN_NAVY_SEAL in info class was incorrect (current XML file is: Civilizations/CIV4CivilizationInfos.xml

GenericKen: It does combine python files, as long as two mods don't modify the same lines.

Roetghoer: Adding new units and civilizations should be ok. I've successfully combined FinMod, Netherlands, Carthage, Celts and Turks. Could you put the failing file somewhere I can download it from and see if I can reproduce the same result?

I tried it again today but it stilll didn't work.
So I renewed all the mods to current versions. Still using Lost Units mod and the netherlands.

Now the modder doesn't give me XML errors anymore BUT the civ isn't playable anymore.

Both mods are available on this site.

I'm quite certain I installed gtkmm the right way but to be sure i'll redo this as well.

@tdb: Which file would be usefull? the entire combimod? or just the unitinfos.xml


EDIT:
refreshed EVERYTHING including game, mods, gtkmm, ALL! (Probably unnecesary BTW)
Messed with folder settings.
Settings are now as follows:
1.E:\Program Files\Firaxis Games\Sid Meier's Civilization 4
2.C:\Documents and Settings\Rutger\My Documents\My Games\Sid Meier's Civilization 4\MODS
3.E:\Program Files\Firaxis Games\Sid Meier's Civilization 4\Mods\MyCombinationMod\Assets

! Now the modder produces the (much desired at this time) conflict warning !
Conflict between mods 'Lost Units' and 'Netherlands' in file 'XML\art\CIV4ArtDefines_Unit.xml'

I feel like we're getting somewhere, I WAS able to combine the FexFX (large-long worlds) and Netherlands mod. All other (Dual) combo's with Netherlands we're conflicting

If you feel the Prog is mature enough we should experiment with multiple mods and give results in a sole-purpose 'Conflicts Thread'. This would, hopefully, give modders some more insight into conflicting XML's etc.

OK First of all; it looks like i'm bashing this prog, i'm NOT i think this is THE missing link and think every modder should be able to see where his/her mod conflicts with other productions! This prog makes things happen :goodjob:

But at the moment I'm a bit puzzled by TDB's proclamation of the succesful combo of turks carthage and netherlands as they all give conflicts on my modder. I don't seem to be able to combine any civs at all. (Conflict between mods 'Netherlands' and 'Jecrell [Carthage]' in file 'XML\Art\CIV4ArtDefines_Civilization.xml')

It doesn't look to me like you're bashing my product. It looks like you're experiencing problems which are due to either a bug in the program or incomplete documentation. It's perfectly acceptable to complain about either of those things, especially with detailed input as yours.

Now that I think of it, that five-civ combine was before I implemented the conflict checker... Now I need to remove Netherlands to get the other four combined.

I did get Netherlands (version 1.06) and Carthage (1.2) combined though. Which versions of those did you have? I'd like to see the .Civ4Mod files produced by the program from those two mods, if you can put them somewhere.

As for the Conflicts thread you suggested... It's a good idea, but I think we can't fully trust this program just yet. Manual verification is still needed as the program may contain bugs. Of course, there are some clear cases... Like Netherlands and Turks both containing the file Art/Interface/Buttons/actions_builds_leaderheads_specialists_atlas.dds. The graphics files are the hardest ones to combine (although possible, and I intend to do that one day) so modifying them should be avoided at all costs, instead adding new files and referencing those from the XML.

(quick) Using Carthage 1.3 and Netherlands 1.06 No Match! (Leaderhead infos)

BTW i don't have a place to store files publicly but if u PM an email adres i'll try to send some conflicting modfiles. Or u could point me to a free public host

I'll check myself with that updated version of Carthage as soon as I can. If I still don't get the error here, I'll try to figure out some place where you can upload the files.

This program ROCKS!!

I have been merging mods left and right (it seems I have had a few problems if the Mod contains Graphics files but this is rather easy to do manualy so its no big deal). I have been posting links to you over at Poly and trying to get people to adopt moding standards that will make their mods more compatable and mergable.

Mainly were looking at having a CvCustomEventManager.py file that overrides every event with a call to its parent the normal EventManager. Moders simply insert their mods as a single line call under the apropriate events. The original intent was to make manual cut and pasting easier but it also works with your merging program. If we can get moders to use a common template of Custom python files with all the moding consisting of simple additions to thouse templates and their extra files then we can virtualy guarantee no Python scripting will trigger a collison or logic problem. People could install the template directly to their Civ4 files as they do nothing to the game on their own, when you compile merge files all your doing is getting the additions to the template which will be a few imports and a few calls. Pre-complied merge files could then be downloaded and plugged in almost effortlessly by a person with the template installed.

Package the whole thing together with some good Documentation and Tutorials and we could realy give the Mod fans and the Mod makers a great interface tool. Im working on the templates as we speak and will be singing your praise at Poly, keepup the good work on your program I cant wait for next version. One sugjestion, could the "Description" field perhaps display the first line of the mods readme file so authors can elaborate on what their mod dose. :goodjob:

Standarization! that should speak for itself. I think Take2 and firaxis should have anticipated on this 'problem/inconvenience' beforehand by making a standard on their own or at least facilitate in the developement of such a service.

BTW Love your quote Impaler :goodjob: Dune fan myself

Sorry for the lack of updates on this front. I've had a load of other things to do. Rest assured though, I don't intend to drop this project anytime soon.

Roetghoer: Was your attempt after the 1.09 patch? I noticed that it changed the iLimitedWarPowerRatio of about every leaderhead. Consequently, two mods containing leaderheads will always conflict if they were made for 1.00. Unfortunately, it's practically impossible for my program to take this kind of things into account. One solution for this would be that modders start distributing their mods in the Civ4Mod format. That way the distributed mods will only contain the actual changes and conflicts due to game patches will be reduced to a minimum.

Impaler: I'm planning to parse the mod name and description from the ini file. And yeah, graphics files are a problem... I might figure out something for them, but the best solution would be if modders would create new graphics files instead of modifying existing ones. Thanks for the supporting comment too! :)

All attempts on fresh game using all new versions and the v1.09 patch of the game.

The mods are made against 1.00. That means they contain XML files that are copied from 1.00. Now, as the 1.09 patch changed some things, the data files in the mods no longer match and my program detects the changes.

What you would need to do is create the Civ4Mod files using 1.00, then patch the game, and apply the mods.

I've been trying to create a full CustomEventManager shell that can be installed to the users Python directy and used as an atachment point for a whole variety of Mods which add new Python based Events to the game.

The idea is for Mods makers to insert their mod onto the template with only a few lines of additiona code. Modswitcher should then be able to merge these together. But I am getting collison warnings presumable because the added lines are in the same locations in the files. I would hope that modswitcher could deal with 2 seperate files trying to add a single line in the same spot in the original file. For example...

## Sid Meier's Civilization 4
## Copyright Firaxis Games 2005

from CvPythonExtensions import *
import CvUtil
import CvEventManager
import sys
#INSERT IMPORTS
import someplayermademod

And another mod using the same template instead adding

## Sid Meier's Civilization 4
## Copyright Firaxis Games 2005

from CvPythonExtensions import *
import CvUtil
import CvEventManager
import sys
#INSERT IMPORTS
import someothermod


I figure as long as these are simple one line inserts to an otherwise identical file they should be mergable. Am I doing something wrong? Dose modswitcher not have this ability? Is their some kind of work around that could be used? If its just an issue of which line should come first then lets just default to alphabetical order, these single lines of Mod call should function in any order. Here are the files and the original they are derived from.

CvCustomEventManager is an empty shell that wraps around EventManager to alow Mods to plugin to all the events in the game without needing to touch the original EventManager. It can be placed into your Civ4/Assets/Python directory without altering the game in any way. CvEventInterface overwrites the file of the same name under Assets/Python/EntryPoints. It performs the wrap and will not effect the game either (as it simply wraps a shell that dose nothing), but do be shure to back up your original files anyway. With both of these in Civ4 directory Mod switcher will see the Mods that use the template as additions to this file rather then a new file. The TechConquest and Realfor mods are 2 popular mods that use Python Event triggers and were released with CustomEventManagers of their own. I have altered them to use my extended EventManager. They should compile and run correctly if you have the first 2 files properly instaled. The merging process gives a collision in CvCustomEventManager though.

V. Convenient!

this is a holy idea tdb.:goodjob:

I have been wanting to combine the Realism mod along with the Composite mod by exeter.

i cannot make it work, probably because my computer literacy only extends to headbutting my keyboard.:cry:

I would just like it to be a simple thingy, where upon you could cut and paste mods into this umbrella structure, it would work great for the digital dunces of the world, like myself.:crazyeye:

I think an option to skip Mod file creation or do it in one big batch and proceed strait into final merging,is what he is looking for, just drop ready to play mods into a Directory and compile the whole thing into a ready to play mod. It shouldn't be very complex to do, have a 4th file pathway under preferences to a directory of Mods then just check each sub-directorys name to see if it has a matching merge file, if not compile it and when everything is ready merge them.

On another note I have been doing some brainstorming on the Tecnique you use to record XML changes, the heirarchical nature is definatly more efficient. I began to wonder if similar gains could be seen if Python was also interprited in a similar fashion. For python the number of indents on a line determine the heirarchy so it can isolate changes to a particular function or loop within a function.

On a second note I have been wishing for a runtime Collision resolution ability, even if its as primitive as "MOD A over-ride" "MOD B over-ride" buttons poping up repetedly untill all the collisions are resolved. Eventualy some kind of slick interface cound be developed with lots of options and maybe even textfields you can manualy cut/paste/type out the resolution to your hearts content.

On a third note I have found that Mod switcher will generate a collision if 2 mods try to add identical content. I took a simple mod and made 2 identical merge files by compiling it twice and trying to merge them togeter. They imediatly colided on the art file each of them tries to add to the game even though they both obviosly encode the same file. Their should be a check involved when 2 Mods attempt to modify something, if both are essentialy in agreement on the modification or try to add identical content then a collishion dose not need to be thrown. Many popular mods borrow and extend the content of other mods so these types of false alarm conflicts will be common and very inconvenient for people if this isn't adressed.

Lastly I have noticed an erronious result on a Mod which adds sub-elements in an XML file, several additional lines and 2 levels of nesting are added to an element which is normaly empty in the UnitInfos document. The merge file is picking up and making a record of the changes but the output Mod lacks is only adding a closing tag.

The original file has simply <ProductionModifiers/>

I modify this too this (its fully functional in the game so its not a formating problem)
<BonusProductionModifiers>
<BonusProductionModifier>
<BonusType>BONUS_IRON</BonusType>
<iProductonModifier>100</iProductonModifier>
</BonusProductionModifier>
</BonusProductionModifiers>

giving this block of data inthe merge file

BonusProductionModifiers&gt;
&lt;BonusProductionModifier&gt;
&lt;BonusType&gt;BONUS_IRON&lt;/BonusType&gt;
&lt;iProductonModifier&gt;100&lt;/iProductonModifier&gt;
&lt;/BonusProductionModifier&gt;

And when I compile I only see

<BonusProductionModifiers>
</BonusProductionModifiers>


Try it yourself and see if the same thing happens too you

can i get a hold of a civ4modder for dummies? ireally need a step by step from the beginning. i've never edited python or anything really, i've just edited my first config file like yesterday.

']I think an option to skip Mod file creation or do it in one big batch and proceed strait into final merging,is what he is looking for, just drop ready to play mods into a Directory and compile the whole thing into a ready to play mod. It shouldn't be very complex to do, have a 4th file pathway under preferences to a directory of Mods then just check each sub-directorys name to see if it has a matching merge file, if not compile it and when everything is ready merge them.

I'll do something to that end in a future version.

']On another note I have been doing some brainstorming on the Tecnique you use to record XML changes, the heirarchical nature is definatly more efficient. I began to wonder if similar gains could be seen if Python was also interprited in a similar fashion. For python the number of indents on a line determine the heirarchy so it can isolate changes to a particular function or loop within a function.

That should be possible, at least on the function level. Parts of functions get more tricky.

']On a second note I have been wishing for a runtime Collision resolution ability, even if its as primitive as "MOD A over-ride" "MOD B over-ride" buttons poping up repetedly untill all the collisions are resolved. Eventualy some kind of slick interface cound be developed with lots of options and maybe even textfields you can manualy cut/paste/type out the resolution to your hearts content.

Good idea. I'll figure out some way to do it.

']On a third note I have found that Mod switcher will generate a collision if 2 mods try to add identical content. I took a simple mod and made 2 identical merge files by compiling it twice and trying to merge them togeter. They imediatly colided on the art file each of them tries to add to the game even though they both obviosly encode the same file. Their should be a check involved when 2 Mods attempt to modify something, if both are essentialy in agreement on the modification or try to add identical content then a collishion dose not need to be thrown. Many popular mods borrow and extend the content of other mods so these types of false alarm conflicts will be common and very inconvenient for people if this isn't adressed.

I'll see if I can do something to this too...

']Lastly I have noticed an erronious result on a Mod which adds sub-elements in an XML file, several additional lines and 2 levels of nesting are added to an element which is normaly empty in the UnitInfos document. The merge file is picking up and making a record of the changes but the output Mod lacks is only adding a closing tag.

I hope to have fixed this in the version I'll post in a few minutes. (Edit: 3am. I'm pretty tired. If you want further explanations, ask and I'll explain when I have the energy for it)

can i get a hold of a civ4modder for dummies? ireally need a step by step from the beginning. i've never edited python or anything really, i've just edited my first config file like yesterday.

I added such a thing in the first post. I hope it's clear enough.

Version 0.3 is up! See first post for changelog and updated download link.

Hi Tdb,

I tried the new version but it gave me a compatibility issue.

All settings were correct, then tried to create mod MegaModv1.1 ( http://forums.civfanatics.com/showthread.php?t=148486 )

It was nearly finished when it crashed on the UnitInfo's and WinXP wanted to report an error which would include a file on compatibility. I'm not running anything else then XP firewall and firefox so shouldn't be a problem there.

I can regenerate this error over and over, would you have any use for the compatibility file or the WinXP error log?

The WinXP error log is not much use to me. I don't know what the compatibility file is, but you could send it just in case. Also, if you could zip all the Civ4Mod files involved and send them to me, that would save me a great deal of work. You can put the files in http://www.filefactory.com for example.

hi all
im having a problem,
it seems that i can add every mod that doesnt contain the unitinfo xml file
the promlam crashes ...
i wanna mostly it happaned when i treid to use a civilzation mod - like civarmy civ pack with another mod like mega mod for instance....

anybody can help?

The WinXP error log is not much use to me. I don't know what the compatibility file is, but you could send it just in case. Also, if you could zip all the Civ4Mod files involved and send them to me, that would save me a great deal of work. You can put the files in http://www.filefactory.com for example.

Actually the compatability error occured on a fresh civmodder without any Mods created. Whilst i was creating the first mod (megamodV1, link is in earlier post) civmodder crashed and gave the error report.

I'll send you the compat.file
EDIT: its attached to this post (small txt)108623

Hm, seems like I have released a broken version... I will try to fix the crashes and post a new version after that. Meanwhile, please use version 0.2 instead.

I think I got it fixed... At least I was able to create a diff of MegaMod as well as various other mods without crashes.

However, I'm at my parents' place for christmas and I don't have my Windows development box here. So unless someone else volunteers to compile the program for Windows (I'll give the source on request), you'll have to wait until after christmas.

In any case, merry christmas for everyone!

Did u upload it? or is it v0.3 as is?

It seems to be working for me, the incorporation of the expanded elements I mentioned now seems to be working properly though it inserted the name space tag to the element and the lines were a bit odd looking.

<BonusProductionModifiers><BonusProductionModifier xmlns="x-schema:CIV4UnitSchema.xml">
<BonusType>BONUS_OIL</BonusType>
<iProductonModifier>100</iProductonModifier>
</BonusProductionModifier></BonusProductionModifiers>

CIV dosen't mind the fact that it looks funny, its still valid XML. Nice work tbd I will keep checking out this latest build .It can crash on me but so did the last build, the patern seems to be that if I have created a Mod without restarting the program I will get a crash when attempting to merge. As long as I just close and reopen the program and then merge first I seem to be fine.

MERRY CHRISTMAS TO ALL!!!

hi tdb

i hope youll be able to fix it soon ill be waiting! :)

but anyway - where can i get v 0.2?

oh sorry i saw now you added link for 0.2..

thanks!

well
i treid to merge a mod - it did load the mods with no problem like in version 0.3, but when i treid working the mod, it did not worked properly - it gave some errors on the xml files i think somthing about the units.

hope you can make it work...

thanks.

That was most likly a genuine collision ware the 2 mods attempted to change the same value of a Unit, Modswitcher cant desided which change should be used, hopefully a system of resolving these collisions will be added in the future. For the meantime you just need to make shure to dont mix mods that have such collisions.

EDIT - I had another idea that might be of use in a future version, Dependencies!!
As we know lots of Moders are borrowing and extending content created by others, say for example someone adds Saltpeter as a resorce and then someone else makes this Resorce a Pre-requisite for Greniders. If this second Mod gets applied without the first Civ4 will throw an XML error. To avoid this the User should be prevented from merging mods without also merging all of that Mods Dependecies as well.

When you create a merge file you should have an option to enter Dependencies (atleast one possibly more), only other Merge files you already have can be assigned as Dependencies. A new Dependecies element in the merge file holds the information and when you see your checkbox list anything with a Dependency is indented and underneath its parent in heirarchical fashion. Checking a mod with a Dependency will automaticaly check its upward chain of parent Dependences, likewise unchecking the parent will uncheck the child mods. If a Mods Dependency cant be found (cause you removed the Dependency somehow) then a red X is shown in the check box and the Mod cant be selected or merged.

']It seems to be working for me, the incorporation of the expanded elements I mentioned now seems to be working properly though it inserted the name space tag to the element and the lines were a bit odd looking.

Yeah, I have some problems getting nice looking input from the XML lib I'm using... But as you mentioned, it's still valid XML. Just not human-readable.

Dependencies are indeed a good idea. However, for that feature to be efficient, mod makers are required to distribute their mods in the Civ4Mod format, which not everyone may be ready to do. Don't get me wrong here - it would be great if mods were distributed in this way since the format is superior in many ways. Much less compatibility problems when the game itself is patched.


i treid to merge a mod - it did load the mods with no problem like in version 0.3, but when i treid working the mod, it did not worked properly - it gave some errors on the xml files i think somthing about the units.

You may be referring to the problems caused by the 1.09 patch changing certain values for all units, which broke merging 1.0 mods unless you had a difference file made with 1.0. The other possibility, as Impaler said, is a genuine conflict between the mods.

so i see now.
what you guys are saying -that - if i want to merge a mod that contains
similar files - like in unitinfo, i cannot merge the mods...mm..
bummer...hope someone will have a magic solution. :)

You can merge them IF they dont alter the same element values.

So for example one persons feels Archers are too strong and changes their power to 2. Another Moder feels their too weak and changes the power to 4. If you tried to merge thouse mods you would get a collision. Now if one Mod had affected cost to build and the other had changed its power then their would be no conflict (except ofcorse you would likly screw the game balancing of the 2 seperate mods). Later version might give you some kind of popup screen that asks you to choose between conflicting data.

ok cool. so i guess ill be looking forward for the next version :)

it's a coll idea. but i think i can be done much more simply. i have read in the german civforum that one hase just copied the link on the desktop and wrote at the end of the destination-line ("Ziel") *-mod= modname*
so you can make some links and when you want to play a specific mod, just start the link it belongs. it runs perfectly for me :)
see on pic:

This program is for combining multiple mods together not simply selection which Mod you want to load at startup (which is easily done by adjusting the Config file, which is what I suspect the program your showing us dose). It will eventualy eliminate the hugly tedious task of cutting and pasting together XML and python files. If you have spent any time doing this you would realize the complexity and time savings involved. Now we just need to work all the bugs out...:crazyeye:

Hey Tdb,

Have you launched the modswitcher on any german forums? Germany has a lot of CIV modders and gamers who could probably be of great help. And i don't think you would have to be afraid of the language because a lot of Germans are perfectly able to speak English :)

P.S. currently at my parents' home, forgot to bring CIV :cry: I think i'll keep Christmas short this year :lol:

@impaler
aah...i have not read it propperly enough :)
i thought it is a programm like the one for ctp2 where i can simply choose the mod i want and start. but when it is able to mix 2 mods then it sounds cool :)

I'm back at my own apartment with my Windows dev box. Expect bugfix version 0.3.1 to be up later today.

Roetghoer: No, I haven't. And unfortunately, I'm not sure if I have time to keep track of two forums... However, feel free to distribute the program at other forums as long as you give me proper credit for it and post a URL to this thread so users will be able to get the latest version.

Even with the bugyness of the current version I have been semi-succesful in using it to modularize the Mod I am currently working on. I'm adding 9 buildings (some are my design some I borrowed from other mods) and I keep each one in its own stand alone mod folder I compile and merge them into customAssets as I test and debug each one. I find that when ModSwitcher is adding a new element to the game it works very well. On the other hand certain types of alterations are being lost someware butting the compiling or merging.

I can consistently get a crash when I try to compile this New Trait Unit Changes which changes a number of units (navel units, missionaries, settlers, workers, recon) specialunit field. Specificaly it always crashes after the Indian Fast worker or atleast thats the last thing displayed in the console.

I also notice inconsistent merging of files that have alterations more then one level deep, for example when I tried to add more unitcombat types to the aggressive trait they dont end up in the final output though new traits are successfully incorporated. Likewise truncation of elements in a mod

<Element>
<subelement>VALUE</subelement>
<subelement>VALUE</subelement>
<subelement>VALUE</subelement>
</Element>

<Element/>


Dose not seem to work. Hope this feedback is usefull for you I absolutly love this program :D

I think I have fixed that too in 0.3.1. You'll have to wait a couple more hours to get your hands on it.

0.3.1 uploaded. See first post for link.

Hi,

It has some difficulty with the Lost Units mod, both in combination with other mods as well as stand alone through civmodder. Game itself crashes on XML initialisation.

EDIT: LostUnits standalone:
http://www.filefactory.com/get/f.php?f=f44e5758f8a18253a940cfcc
and A combo with the Netherlands:
http://www.filefactory.com/get/f.php?f=0ed509eab4c76e91c699b747

Both gave me init failures of the game, most definitely the Lost Units failed.
Combo of Greenmod and Netherlands went just fine
I hope it's usefull.

Game 1.52
civmodder 0.3.1

Hm... That might be due to the patch again... I'm using Lost Units in combination with a load of other mods with 1.09 and it works fine. I'll investigate this problem when I get my game patched to 1.52.

Ya, Lost Units is causing some sort of problem at XML load. I had it with 3 other mods and it kept crashing, dropped it and the other 3 loaded just fine.

hi all

what i am able to do with modswitcher is only to combine simple modswith nounits or bulding - its basiclly just copy files from one mod to another and when its detect two files of the same type - it wont continue (mostly unit files buldings and civilization files)

any new version will come out that would be able to do full combine with mods?

i want to thank you for making this. i have now spent all night combining mods and have 80 playable civilizations with over 200 leaders in my mod


thanks, you rule

I can't get it to work. In the OP you said when two files conflict I should just delete one. So I do that, hit refresh, and then try to apply it again and the same message comes up saying that the same files are conflicting even when one of them is deleted.

Has anyone else had this problem?

Good Job, very useful.

seems like this project is at a hault, what happen?

Yes, are you still working on this project? I never really play with a single mod and have put hours and hours into combineing various mods and recombining certain mods with certain other mods and have ended up with many copies of the same data in diferent combinations. This program would simplify things so much.

And just an idea, but what if the program could compile a list of the units, buildings, civs, ect in each mod and allow you to pick out parts of each mod and put them together in a new mod. Granted this would be a little more work and might go along with the editor part of the prog as you would have to worry out tech prereqs and such for the stuff you bring in.

Well anyway, hope you haven't abandoned this program, it looks to be a very cool addition to the CIv world. :>

BUMP!

I hope that someone continues with this

Bump x2!

I started a petition for a program like this! Pu-leease don't let it die! Its just starting to rock!

Yay, keeping the dream alive. The thing is, a lot of released mods are really components - individual mods that only change one small aspect of the game. I imagine that lots of people want to use more than one of these component mods (I know I do). It's not just a case of comparing and editing XML files either - there's python files, .dll files, and so on. Is there anyone who could continue on with this fine project (even if that means starting from scratch)?

I would love to continue it (hopefully not from scratch), now that the SDK exists it might even be possible to incorporate the whole thing into the DLL. Currently though my mod work takes me elseware, I will want to revisit this when I am more skilled or when the original author releases their source.

I'm back!

I guess I just kinda disappeared back there... I got some other things to do and subsequently forgot all about Civ4. You can thank Impaler for reminding me of its existence. Now I've got some more free time though, so I could continue playing it and working on this program as soon as I finish Dungeon Keeper 2.

Impaler: I can give you the code if you want to try turning this into a dll. Getting the relevant parts out of my code shouldn't be too hard, but I have no idea how hard it is to integrate it with Civ4.

Woohoo! Good to have you back! And Impaler, great to see you here too! Now let's get this show on the road!

Oh this is a plesent suprise and so soon as well!

have been doing some thinking on diffing and merging several months ago and was toying around purely on paper with some merging algorithms and how to encode the information in XML, basicaly trying to re-invent what you had done. I also had some ideas conserning dependency, I myself have been burned many times by patches that change the XML some method of designating a change file as coming from a particular version might help.

Incorporating this into the DLL is a very bold dream at this point, it may not even be possible if the relavent comands are not in the DLL (I have already run up against problems that cant be resolved because they come from the Exe). The geneal plan was for the player to go to the "Load a Mod" page be able to select multiple radio buttons and have theses merged and imediatly launched into, in addition the new mergeing should be saved in some mannor so it can be reused quickly at a later date.

If incorporation directly into the DLL is successfull I would still like to see an outside standalone utility program without any Civ4 specific features developed for the general public. What do you think of licensing it under the GPL?

Go ahead and mail me the code if you like, or even better upload it to CivFanatics.

Releasing the code as GPL might be a good idea. It would also guarantee some degree of contingency in the case I disappear more permanently. I'll have to create a nice and easy source packaging system for myself first though.

A word of warning at this point: I'm developing the program on Linux, and it depends on a couple of my custom libraries. Getting all these compiled on Windows is not trivial.

My 2 cents:

I don't think that incorporating the mod combiner into the civ shell is really necessary. It sounds like a lot of work, and 3rd party program that did the merging and either created output files or just merged on a temp. basis and launched civ 4 would be fine.

As they say, every cloud has a silver lining - after a long pause with some project, I'm able to look at problems with new angles. I just had a great idea for enhancing the XML diffing engine, making it more generic, robust and easier to maintain.

I'll probably finish Dungeon Keeper 2 next weekend, so you can expect a new version of civ4modder in two or three of weeks. I hope to get the source released at that time as well.

Brilliant news :b

Grats on beating DK2! I always lose interest about halfway through (often due to a red a-money beat down).

Have you ever played Majesty? V. similar, v. good.

Oh, and get to work!

EDIT: I doubt you really need to run it all through the game. A stand alone program worked (and works) just fine with Morrowind. In fact, I betcha there would be more utility in a stand alone program that simply compiled mods, especially since you could then upload the result.

Finished DK2 on Sunday (or actually Monday, like 2am), and after having to do all kinds of things this week I've finally got my hands on civ4modder code again. I should get something ready sometime next week unless something unforeseen happens. Next up on the list are mod priorities (for conflict resolution) and quick-merging functionality.

Hm, that Majesty game looks good... I'll have to try it... I have a soft spot for simulation-strategy games.

As for integration with Civ4, I'm not going to do it in any case. I don't need it (I build the mods once, then play a game for the next two weeks) and I don't have time to learn the DLL modding system (I have a full-time work and a model railway). Another strong factor is that I'm playing the game on Linux and getting a working cross-development environment for Windows programs is difficult at the very least. If someone else wants to try though, I'll offer support as well as I can.

WooHoo, glad to hear your back to work on this, I'll have to keep an eye on this thread. :D

I'm making pretty good progress, the new XML diffing and merging system is almost ready. I've also written a great deal more XML definitions for the program (due to the new system forcing me to do so), which will make it detect more changes.

So far I've hit one problem - namely, the file CIV4DiplomacyInfos.xml contains Response "objects", which however have no key that could be used to identify them. Thus the program is currently ignoring the file. I know how this can be fixed, but chances are the fix won't be in version 0.4.

I'm not sure if I can get both mod priorities and quick-merging done this week (I'll be unable to work on this during the weekend), but I'll try.

Thanks for the update!

I'm having a slow day at work and decided to throw up my own website for this program and the upcoming mod database.

http://civ4.tdb.fi/

Ouch. I downloaded and extracted GreenMod in order to chop it up into smaller mods. Then I hit the file CIV4PlotLSystem.xml. It contains stuff like this:


<ArtRef Name="art:crop">
<Attribute Class="Scalar">NIF:Art/Structures/Improvements/Farm/crops_w.nif::2x1</Attribute>
<Attribute Class="Improvement">IMPROVEMENT_FARM</Attribute>
<Attribute Class="Bonus">NO_BONUS,BONUS_ALL,!BONUS_WHEAT,!BONUS_POTATO</Attribute>
<Attribute Class="Scalar">bApplyRotation:1</Attribute>
<Attribute Class="Scalar">bZMode_Ground:1</Attribute>
<Attribute Class="Scalar">bCutTrees:1</Attribute>
<Attribute Class="Scalar">bForceGround:1</Attribute>
<Attribute Class="Scalar">bImprovementWorkedOnly:1</Attribute>
<Attribute Class="Scalar">bIrrigatedOnly:0</Attribute>
<Attribute Class="Scalar">szBatchGroup:farm_crop</Attribute>
<Scale>0.5</Scale>
</ArtRef>


Not only is that lacking any specific key, it's using XML element attributes as well. Couldn't they have at least been consistent... Nowhere else have I seen attributes been used. It will be some extra work to incorporate support for them in civ4modder.

I had suspected you worked off a "key" element whichin each element to determin if its modified or an addition. It should still be possible to do a comparison on on element that lacks a key, but it will require that all elements be searched for an identical match (and the search needs to be able to full hierachical down to bottom most element), initialize a boolean array to check off child elements that have been found so they can be excluded from further searching and thus save lots of time. Unless people start moving element around it will remain resonably efficient because elements will remain in the same order so the first comparison will get a match, exclude that element from searching, proceed to second element, match ect ect...

The Name="art:crop" would serve as a good key if you can use atributes as keys (I dont see why not, though it is more work like you said)

On a related note, the list of key values should be exposed to the user, idealy on an XML document itself with <ElementKey> listing the element name and the name of its key child element which is used to identify it. This would help make the program more flexible for non Civilization hierachy. The user would switch this file around to be able to work on any set of XML files.

Nope, there are multiple elements with the same Name value, but different contents. In fact, the Name attribute seems to be more like a class than a name. A quick grep reveals 16 elements with Name="art:crop" and 68 with Name="art:building".

As part of the revised XML diffing system, I did externalize the element list. It's currently a plain-text file (easier to parse), but I might turn it into XML. There are still some rules hardcoded in the program, due to the *Infos.xml files behaving in a consistent way with regard to element naming. With a few lines of code, I can match all those.

If you look at almost any xml file, it's pretty obvious that all "objects" (as I call them) have such a key. I'll be releasing some information about the algorithm along with the source once version 0.4 is ready.

I have a question for modders that is appropriate for this thread. Do you modders ever play with the sdk? Are changes to the main dll a part of your mod? Or are people who use the sdk and people who make mods generally separate themselves by generally doing one or the other.

The reason I ask is because I would think that some modders would want to change the source code if they want to make a really big mod. This is why I brough this topic (http://forums.civfanatics.com/showthread.php?t=174793) up. I think it would be nice if a mod switcher was part of a modified version of the main game (i.e. the source code). That way modders could include changes to the source code as a separate code file that is part of the mod. For more information see this thread (http://forums.civfanatics.com/showthread.php?t=174793). I would like to know what others think of this idea. Is using the SDK common in modmaking? Is this idea important to modding? Is it just too complicated for most modders (because they are not programmers?)

I don't think it's necessary to incorporate the mod combiner in the game itself in order to use plugins. It should be easy enough to look for DLLs in a certain directory of the mod and use them as plugins.

The new diffing system and mod prioritizing looks pretty good. I'll polish it off tomorrow and compile a Windows version on Friday. Quick-merging will have to wait until next version.

Hm, more interesting problems, this time with Python files. Modified Special Domestic Advisor and Exotic Foreign Advisor both contain the file DomPyHelpers.py. They have much of the same content, and in fact the file in EFA file is a subset of the one in MSDA. However, civ4modder compares mods against the base set, not against each other, so it will see a conflict as both mods try to add the whole file.

I see two possible solutions to this problem:

1. Turn this file into a mod of its own, and make both advisor mods depend on it. This will be pretty easy on my part, but will require a bit of extra work from the mod makers.

2. Process Python files a bit like XML, splitting the files into classes and functions. If two diffs add the same things, they won't conflict. This won't require anything from modders, but will be a rather major thing for me, and may cause unforeseen difficulties in the future. (This solution has actually been suggested earlier.)

This won't affect the release schedule of 0.4, you'll still get it tomorrow.

Version 0.4 released! Download link updated to first post.

Thanx tdb,

I installed version 0.4 and the library required and started to work. I use some EFA, MSDA mods and other non-game altering mods.

I set the preferences, and expected the mod directory that I set in the preferences to be the directory when I opened to look for a mod to create via CREATE MOD menu option. However it started in the standard mod dir.

I had to change the customassets folder name that comes with most mods to assest for your program to work. Is this the way to do it, or should it also work with the customassets dir-name.

BTW when I close the modder and reopen it, the mod I just created is not displayed, clicking refresh doesn't solve it, only when I click in the listbox, the mod-properties are displayed and if I say "save" the mod is displayed again.

It looks like there are some small glitches in the finishing in this gem. Or i'm doing something competely wrong ;)

BTW, where do you store the diff-info??

HTH

Thanx tdb,

I installed version 0.4 and the library required and started to work. I use some EFA, MSDA mods and other non-game altering mods.

I set the preferences, and expected the mod directory that I set in the preferences to be the directory when I opened to look for a mod to create via CREATE MOD menu option. However it started in the standard mod dir.

Yeah, I guess that configuration option can be pretty confusing... The mods directory is where the difference files get stored. I should probably change it to something else. Storage directory perhaps? Do you have suggestions?

I had to change the customassets folder name that comes with most mods to assest for your program to work. Is this the way to do it, or should it also work with the customassets dir-name.

That's as expected. AFAIK, Civ4 only looks for an Assets directory under mods. If this is wrong, I can make it also detect CustomAssets.

BTW when I close the modder and reopen it, the mod I just created is not displayed, clicking refresh doesn't solve it, only when I click in the listbox, the mod-properties are displayed and if I say "save" the mod is displayed again.

Weird. Sounds like some kind of glitch in updating the window. Are you saying that the mod properties window is displayed right away when you click on the list? It shouldn't be until you select Mod Properties from the menu. Does this trick make all mods show up or only that one?

What versions of GTK and Gtkmm do you have?

It looks like there are some small glitches in the finishing in this gem. Or i'm doing something competely wrong ;)

Indeed, that's why the version number begins with a 0. Any feedback on bugs or possible enhancements is welcome.

BTW, where do you store the diff-info??

As previously noted, into the directory labeled "mod directory" in the preferences. By default this will be a directory called "mods" under civ4modder's directory. (Pay attention to the "working directory" setting if you're making shortcuts.)

Yeah, I guess that configuration option can be pretty confusing... The mods directory is where the difference files get stored. I should probably change it to something else. Storage directory perhaps? Do you have suggestions?
Ah, that explains it. Mayby you can have two preferences one for your store of separate mods to read from and one for Modder-mods (doesn't sound right though), mod vault, modlist. Your default choice of the mods-dir is offcourse a valid one, but it would be nice if we could overide it.

That's as expected. AFAIK, Civ4 only looks for an Assets directory under mods. If this is wrong, I can make it also detect CustomAssets.

I mostly use mods as use-allways mods in the custom assets dir. Most modmakers have their mods tailored to this so the dir-tree starts with custom assets. The game also looks in this directory for mods to use. I don't know however if you select a standard mod ingame how the game handles customassets.


Weird. Sounds like some kind of glitch in updating the window. Are you saying that the mod properties window is displayed right away when you click on the list? It shouldn't be until you select Mod Properties from the menu. Does this trick make all mods show up or only that one?

What versions of GTK and Gtkmm do you have?

GTK 2.0 and GTKmm 2.4 (2.8.8.1. via version info) just downloaded them today from your link.

This is what I get when opening modder,I get this screen
http://img138.imageshack.us/img138/1145/modder16wq.jpg

The version number is empty again, I inserted 0.95. If I reinsert it again and close it again, the mod is displayed correctly, like

http://img97.imageshack.us/img97/9205/modder37kv.jpg

Another related thing, whenever I hit 'Verversen' (Refresh) everything looks like the first picture in my previous post. I now have more mods inserted, they all do the same

Hm, looks like a problem with reading the mod file headers. Can you send me the .Civ4Mod file for that? EFA should have few enough changes that the file fits here as an attachment.

Noticed their wasn't any Building support on the element.txt I'll add it to the file and upload it here, if you think it looks good you could use it in the next update.

I'm realy enthusiastic about this tool it could be on par with the SDK in its significance to the Moding community.

EDIT: I was also thinking of a method to bundle diff files together as a quick way to get around dependencies, allow people to place folders containing Diff files in the Mods directory, the interface would then allow only the whole folder to be checked or unchecked, thus the user can create bundles of diff files that they know should always be compiled together and thus avoid dangling references. Yet they are still able to breakup their work into the smallest possible blocks such as individual units, techs or buildings so other can extract and re-use the bits with a minimum of work.

There's probably plenty of other stuff missing as well from the elements.txt file. Any help is appreciated. If you can, please send patch files (as generated by diff -u).

Good idea there with the folders. I could show them as an expandable group in the list so the individual components could be switched on or off as well, if the user so desires. (I'd very much like to have only the new resources from GreenMod...)

And thanks for showing so much interest in this program :) It's always nice to know my work is appreciated.

Excellent program! Does the last update (23/6) save the "changes" as whole xml files? Won't that make it impossible to select what mods you want to include and exclude?

Will you make it Multilanguage. For example english/german/french etc.?

Here's the updated elements.txt

I added, BuildingInfos, SpecialBuildingInfo, TraitInfos and PromotionInfos. I made some small additions to other areas ware they were lacking. Externalizing that list was a great idea, I see now that most of my trouble in the past versions was due to my habit of activating nested elements that arn't normal used in the game and thus weren't on the lists. The new warning "Such and such element has Children maybe its not a value element" is very handy for catching these ommisions, pervious versions just skiped such elements causing changes to not be recorded. Oh also when I compiled my New Traits Mod I got an 88% reduction in size :goodjob:

Also I see the program succesfully added new value Tags that I added with SDK work, I am shure if I add support for my my complex nested tags on elements.txt these will work as well and shouldn't cause any trouble for people not using them. Anyone have requests for additional documents which need support?

Good idea there with the folders. I could show them as an expandable group in the list so the individual components could be switched on or off as well, if the user so desires. (I'd very much like to have only the new resources from GreenMod...)

Actual that would defeat the point, the idea was that everything in the Folder needs to be compiled together or not at all because of cross dependency and dangling references. So for example I create a new Civ and reference 2 new leaders on the Civ, the 2 Leaders can be seperate Diff files of their own so I have 3 files and put them all in the folder. If you compiled the Civ without the leaders you would have a dangling reference on loading. Now if a person knows what their doing and wants to replace one of the leaders its rather easy for them to remove one leader file and replace it with another, then they can compile and manualy edit the dangling reference away. The folder level checkbox helps keep newbies from making errors and helps experts modularize their content to the greatest extent possible.

Such and such element has Children. Maybe its not a value element? appears for like every mod I try to merge! That includes the mercenaries and immigration mods!! What should I do??

Impaler: Thanks, I'll take a look at that in a few days. I'm currently visiting a friend for midsummer (have been since Friday), and will be building my model railway a few days after I come back, but I should have time to continue working on this program from Tuesday on.

I guess I misunderstood your idea with the folders then... It's certainly possible to make it handle them as unsplittable units as well.

Fachy: What are the mods in question? I also need to know the name of the file that produced the error (should be visible over the progress bar). You could try grabbing the updated elements.txt from Impaler's post above, if the files he reported to have added ring ny bells.

Immigration mod: Unit Formations / Civ4FormationInfos.xml
Mercenaries mod: Units Combats / Civ4PromotionInfos.xml
Fachy's (updated) Reglions mod: Civ4Defines / GlobalDefines.xml
Btw this last mod has no pyhton at all!

For some reason it also deleted my CustomAssets folder! (in my documents) I didn't lose any data, but it's acting weird...

Another very rediculous problem I just discovered.. it's showing conflicts between just about everything!

-Tech cost mod (which decreases some tech costs as they become more known) is contradicting with the training barracks mod (something totally irrelevant)
-Enhanced tech is contradicting with enhanced conquest
-Junit religion won't go with the revolution mod!!

Btw, how am I supposed to merge something like "CvModName.py" when merging 2 mods? Should it include both names?

I will further expand the elements.txt that should clear up the "element has children" problem your getting and allow you to diff more mods.

The merge conflicts are probably due to the EventManager Python files, almost all Python based Mods change this file and will generate conflicts their, theirs not much I can do their untill people re-design their mods using a better registration based eventmanager which puts the mod logic is a seperate file and uses a special eventmanager that automaticaly imports the file. The CCP has such an EventManager (made by SimCutie) which should allow easier merging of Python Mods by hand and with ModSwitcher.

But as it implied, the problem wasn't in that file.. rather in other xml files (at least)

Hm, looks like a problem with reading the mod file headers. Can you send me the .Civ4Mod file for that? EFA should have few enough changes that the file fits here as an attachment.

I included the EFA en MSDA .Civ4Mod files

Fachy: I suppose CvModName.py is a completely new file, i.e. doesn't exist in the base game? Two mods adding the same file will always cause a conflict. As Impaler noted, there's not much I can do for that.

As for deleting the CustomAssets directory... If you specified it as the civ4modder custom assets directory, that's normal behavior. I've even noted it in the first post. The program has to get rid of any old files so they won't cause problems.

Michelangelo: Thanks, I'll take a look at those next week.

I thought the main advantage if your program was to merge different data into one file ^^

Eventualy (next version or 2) it should be possible to compare the diffs when their both trying to add the same whole file. If they match then theirs no conflict and you can pick either copy to use. Alternativly if you wanted to move more of the processing towards diffing and away from merging you could at diffing run an MD5 sumcheck or other such arbitary number generator on the file and use that as a keys to identify it, the two mods add a file with matching name and key then again we can assume no conflict and visa-versa. These algorithms are widly avalible and shouldn't be hard to incorporate.

The program works by merging the changes made by two or more mods, as compared to the original data set. Adding a file is the same as changing everything in it, so it's a bit more complicated. There are basically three different levels in this:

1. Both mods have exactly the same contents in the file. Detecting this is fairly trivial (like Impaler suggested) and the file from either one can be used. I'll probably add this soonish.

2. The file in mod A is a superset of the one in mod B, so the one from mod A could be used without ill effects. One such case is the DomPyHelpers.py in Modified Special Domestic Advisor and Exotic Foreign Advisor. This requires some additional logic to detect, basically generating a diff from B to A. If there were no removals, the file in A can be used. This will probably come after version 1.0.

3. The files are different. You used CvModName.py as an example, which I believe contains the name of the mod in some variable. If two mods have this same file, how is the program supposed to decide which one to use? Implementing an algorithm that really understands what the script does is really difficult, and won't be seen in this program (unless someone else supplies it).

But currently it isn't effective if it fails to merge CustomEvenManager or even Modname files!

Go ahead and describe an algorithm for merging new, different files from two mods and I'll implement it. Meanwhile, either alter the mods so that the files have different names, or just decide which mod shall take precedence by clicking one of the buttons when the conflict is reported.

In my opinion, this problem should be brought to the attention of modders and better practices should be introduced and applied.

The program is currently quite effective for some mods, e.g. new civilizations. The version number starts with 0 for a reason.

I'm for one am very glad you're working on this project. :goodjob:
I also have to agree with you that the modding community should make a big effort to allow the mods to be combined with others, this would also mean a great deal for this mod-switcher.

As you have mentioned before the EFA en MSDA are sharing a common file DomPyHelper.py IIRC. How do you advise using your program merging these two or should I try manually to combine these mods into 1 and use this new mod instead?

It seems that actually MSDA has removed one function when compared to EFA (or EFA has added that since MSDA copied the file). To combine, you first need to add this function into class DomPyPlayer:


def getStateReligion (self):
return self.player.getStateReligion()


Then just select MSDA when the program reports the conflict. I haven't tested this myself but it should work.

Would someone want to take up the task of communicating this kind of things to mod makers, or maybe even fixing up mods if needed? I do some of that myself, when I'm starting a new game, but my games tend to go on for weeks.

Thanx, I'll try that

I've been trying to get it to work, but I don't seem to be doing well. :(

I've added the function you described. I found it blanked out with ## so it was easy enough to add it.

If I select both mods (EFA and MSDA) I end up with only MSDA whichever mod I select in the conflict popup. (MSDA is the first in the list)

If I select one mod at a time it works perfectly.

I hope this helps you in resolving this. If you need more info, let me know.

It's possible that they both modify some file in ways that can't be trivially combined. I'll take a look at this in a few days.

For some reason, it's not letting me use large mods like the Sevomod -- I get the usual windows error. Is this just caused by load on the system (though my computer is pretty new), or is it an actual problem?

Also, with most other mods that I try to load, I get an error message that says, "Value element CommerceChangeOriginalOwners has children. Maybe it isn't really a value element?", or some other value element in place of CommerceChangeOriginalOwners. Then, when I click OK, the program stops trying to load the mod component. When I play the mod component, though, it works fine. Is there any way to work around this?

I would also like to say that this is a great idea for a program, and I think that many people (myself included) will find it very useful.

What's "the usual windows error"?

This:
http://img522.imageshack.us/img522/1447/civ4moddererror8jb.th.jpg (http://img522.imageshack.us/my.php?image=civ4moddererror8jb.jpg)

Ah, Windows and its wonderfully verbose error messages. I'll see if I can reproduce the problem and fix it.

As for the value element error, the reason is that that particular element is not supported. Please tell me the mod and file names and I'll add support for them.

I get the windows error message when I try to load Sevomod or Total Realism.

For value elements, I had problems with UnitFormations in Civ4FormationInfos.xml in the Immigration Mod (by TheLopez) and CommerceChangeOriginalOwners in CIV4BuildingInfos.xml in the Shinto Mod (by Dual). Now that I try some other mod components, they usually work, so I guess I just tried the wrong ones.

Thanks in advance.

tbd I have been adding more stuff to elements.txt and am able to solve most error messages with such, but I am unable to figure out a particular class of XML files, particularly thouse that have multiple children directly under the root element. Civ4GlobalDefines is the most obvious of these but their are others. Anything with the format

<Root>
...<InfoObject>
...<InfoObject>
...<InfoObject>
</Root>

Is generating a "Root element has Children maybe its not a value"
When a file is structured

<Root>
...<Infos>
......<InfoObject>
......<InfoObject>
......<InfoObject>
...</Infos>
</Root>

I have no problems with these, most major files are structured this way and the program handles them fine once I add support for the various InfoObjects. If you can explain or provide an example of how to add elements.txt support for the GlobalDefines style file I should be able to apply it to any other such files which give me trouble.

Also here is the latest element.txt containing additional supported XML files

Impaler: Make the root element a container. The toplevel type is just a trick to skip the root element. For an example, see the text definition files.

I noticed this when diffing your version against my original:


@@ -94,12 +217,15 @@
LeaderHeadInfos Trait object TraitType
LeaderHeadInfos UnitAIWeightModifier object UnitAIType
LeaderHeadInfos ImprovementWeightModifier object ImprovementType
+LeaderHeadInfos Trait object TraitType


Any particular reason for adding the Trait element again? Or did you just miss the first definition?

Also, I'd appreciate if you kept the style similar to my own. I.e. add a space after # and drop CIV4 from the file names. (No offense, just a hint for how to be even more useful)

Couldn't help it to try some more.

It looks like when a conflict arises civmodder always uses the first in the list, regardless which mod you select, and any files following the file that causes the conflict will be ommitted and only the files from the first one in the list will be used.

I moved the Dompyhelper entry in the diff to the end, and then the next file causes the conflict. When I move that one to the end as well, all files exept those two get merged.

HTH

Turns out that the prioritizing code had a bug in it. The fix will be in the next version.

Arrrgh. CIV4PlotLSystem.xml is getting more and more difficult. Towards the end of the file, I found LProduction elements, which are similar to the ArtRefs in that the don't have any identifying key. However, they're contained in the root element, along with LNode elements, which _do_ have a key (and which contain the ArtRefs). If I treated the whole root element as a blindset, each LNode would be considered as one unit. However, this would be sub-optimal since mods change or add ArtRefs but leave others in the same LNode untouched. Treating LNodes and LProductions differently produces its own set of problems, since the existing code is not too well suited for that.

Yet more problems come from the fact that the damn file uses CSV lists for things (example from a diff with GreenMod):


@@ -3050,7 +3738,7 @@
<!-- input=PlotBuilder_Productions/Node_6x6_03.bmp -->
<LProduction From="Node_6x6">
<Attribute Class="Improvement">NO_IMPROVEMENT,IMPROVEMENT_ALL,!IMPROVEMENT_FARM</Attribute>
- <Attribute Class="Bonus">NO_BONUS,BONUS_ALL,!BONUS_DYE</Attribute>
+ <Attribute Class="Bonus">NO_BONUS,BONUS_ALL,!BONUS_DYE,!BONUS_CANNABIS,!BON US_COTTON</Attribute>
<To Name="Node_3x2">
<Translate>-2.000000,-1.500000</Translate>
<Rotate>90</Rotate>


Now if I wanted to make cannabis and cotton separate mods (and I'd very much like to), I'd have to process the CSV list for additions. And since this is inside an element with no key, I have no way of connecting these additions to the original element when applying the mod.

And it doesn't even end there. If several mods were to change this file, the applying process would fail horribly. You see, mods are applied in sequence, incrementally. The way I had thought blindset to work was to compare everything in the elements to produce a set of additions and removals. But! If mod A removes something and adds something else, and then mod B tries to change the same thing, it doesn't see the original anymore but will add its own, producing a duplicate. Not good.

In short: Arrrgh.

Any ideas are welcome, my supply is starting to run out.

As for SevoMod and Total Realism... I'm unable to install them with Cedega to try them (damn nonstandard installers, what's wrong with good old zipfiles?), but it is likely their size will cause problems. Civ4Modder needs to keep the mods entirely in memory when it's doing something with them, and there aren't any checks for memory allocations succeeding. At some points the memory requirement might be more than twice the size of the (uncompressed) "standard" mod.

I'll try with my Windows box at some later time - I'm currently experiencing some hard drive problems with it. Meanwhile, if you could tell me how much RAM your system has, that might help me reproduce the problem.

The Shinto mod is missing some element definitions, I'll add them in the next version. I'll probably be using that mod myself too.

Impaler: I put the files in elements.txt in alphabetical order for easier maintenance and added some informational comments at the top. Please use this one as a base for future additions.

http://tdb.fi/~tdb/elements.txt

viper275: Shinto mod builds with this file, so you might want to try it while waiting for the next version. The Shinto buildings won't show up on the world map due to CityLSystem.xml not being processed (see my above rantish post for reasons), but otherwise it works fine.

My computer has 512 MB of RAM, so that could explain the problem with Total Realism and Sevomod. Thanks for helping with the Shinto Mod, any ideas with the Immigration Mod?

I've updated the elements.txt in the above URL to work with Immigration mod.

Impaler: You had made some mistakes with your updates.

ReligionInfo Civ4ReligionInfo toplevel
ReligionInfos ReligionInfo set
ReligionInfo GlobalReligionCommerces list
ReligionInfo HolyCityCommerces list
ReligionInfo StateReligionCommerces list

This one has the wrong toplevel element (a typo I suppose). The ReligionInfos element should be container and ReligionInfo an object. However, the typo prevented this line from working and the hardcoded rules for Info stuff took over. Set and list elements can only contain value elements!

LSystemInfos LNode container
LSystemInfos LProduction container
LSystemInfos ArtRef list
LSystemInfos To list

This just won't work. Containers and objects rely on each sub-element being uniquely identifiable from the element name or key. See also a few posts above.

UnitFormations EventMaskList list
UnitFormations UnitMaskList list
UnitFormations PromotionMaskList list
UnitFormations Position list
UnitFormations UnitEntry list

Position should be container. UnitEntry is basically a container too, but can potentially cause problems since it can occur multiple times. I'll have to see if I can figure something out - there might be an easy-ish fix for it. The masks should be sets, since each value is unique and order doesn't matter (I suppose).

These are fixed in the above elements.txt as well.

Yup, the multiple UnitEntry problem was easy enough to fix. You'll see the result in the next version. Unfortunately this doesn't help too much with the LSystem files, since it relies on the number and order of entries with the same name staying constant, and adding resources requires adding stuff to those files.

I managed to edit EFA and MSDA slightly to make them combine successfully, but it depends on one code change too, so I'll give you the edited versions at the same time as the next version of the program.

I tried CivMore, which is 180MB packed and 300MB unpacked (and happens to contain some things I want). As I suspected, when it got to the final saving phase, the program was using 705MB of RAM. SevoMod is even larger, so no wonder if it crashes with only 512MB of RAM.

New idea for future: "bridge" mods.

This would allow features that are only present when certain two (or more) are selected, but would allow each of those mods to be used separately as well.

Example: Mod A adds a new civilization. Mod B adds a new leader trait and. Mod A would like to use this trait, but would also like to maintain independency. Now the author of mod A could create a bridge mod that automatically gets enabled when both A and B are enabled, and changes the traits of mod A's leaders.

What do you think?

I'm not familiar with the concept of Bridging, its sounds like a kind of "soft" dependency "use if its their but not if it aint" which sounds good in practice but I worry how easy this will be to configure. Mod makers are lazy and tend to make one of two types of mods, mod components which are intended to be used with other things and big modpackages. I see ModSwitcher being used by the makers of the components as a vehicle for easier merger and greater use of their component. The package makers will use it to save themselves time, but as these people already spend vast sums of time hand merging I would expect them to still be willing and able to do some "stitching" nessary to combine components. So for example the new Trait comes as one component in a diff file and the Leader as another file. The package maker merges them and then edits the leader to use the new trait. If a bridge mod is more work then stitching they will probably skip it as their only concerned with their imediate modpackage.

I'm not shure what the best method may be for achiving the goal but it should reflect the typical user/moder and how the creator->consumer flow actualy works in most situations.

I just invented the concept, more or less. The idea is to "smooth the edges" when combining mods. Of course, my vision might be different from how people actually use the program, so the idea might or might not be useful. In my vision, mod makers would ultimately create relatively small mods and release them in the Civ4Mod format, ready to be plugged in to Civ4Modder. Players could then choose exactly what they want to have in the game.

Could people areadly do something similar by the following steps

Diff 2 seperate mods from the Firaxis Original Assets

Merge the mods but use an empty directory as the Base upon which to merge (am I correct in thinking that would give you all files back with just the added elements in the file).

Create a copy of the merged mods and edit the copy with linkages between the various elements (such as give a new leader from one mod the new traits from the other mod).

Diff the edited files from the un-edited files to record the changes as yet another Diff file, this one being a "Bridge" and the first 2 (or more) diffs being its dependencies.

Now some other guy downloads these 3 files and opens up the UI which lists the Bridge as an option requiring the various dependent files. The merging process is now 2 step (hopefully this can be automated as its rather straitforward) firsts all non-bridge mods are merged with the normal Firaxis original code (from properties) as the base, then all Bridges are merged using the results of the first merge as the base. This resulting final merge overwrites the first and the end user gets a rather seamless end result out of it all. Is this what you had in mind?

Something like that. It isn't really necessary to use an empty base though - applying a mod and then reading the result back in will produce the same mod. In fact, it would only cause problems when creating the combined mod, since the program would see all files as "new".

As for the merging process, mods are always applied one by one, i.e. there will be as many "phases" as there are enabled mods. Bridge mods would automatically override all of their dependencies without having to ask the user about conflicts.

Version 0.5 released. Be sure to also check http://civ4.tdb.fi/?page=mods.

Downloaded it, and tried it..........................

It works now for the EFA and MSDA :goodjob: (downloaded from your site). The ones I create myself don't. I'll take a look at it what you changed but I have the idea that it has to do with screwed up formatting in regard to the original, while not actually changing anything substantial.

The problem with the invisible modnames (probably due to the incorrect reading of the version number) still exists.

I'll try to combine some other mods now myself.

Of course they work, since I made them work ;) It was pretty simple really, if you know how diffs work. Then again, if you don't... I can give a more in-depth explanation later, if you want.

Unfortunately, I wasn't able to reproduce the invisibility problem on my computer (tried both Windows and Linux). The program uses exactly the same code for updating the mod list both on startup (or refresh) and after changing its properties. This would point to something being wrong with the mod file, but that evidently isn't the case as your files work here. Thus, I'm pretty baffled here...

Does the mod information show up in the properties dialog? Are you using any non-ASCII characters? Do you have other computers you could try it on? Perhaps with different versions of Windows?

I guess I'll have to create a special version of the program that shows some debugging information...

Does the mod information show up in the properties dialog? Are you using any non-ASCII characters? Do you have other computers you could try it on? Perhaps with different versions of Windows?

Everything shows exept the version number.

This is what I get when I open the properties of a mod.
http://img89.imageshack.us/img89/6610/properties5ft.jpg

If I change the version to anything i've tried, normal characters including numbers, f.i. "1" "v95" "0.95".and I click on save, close the window, I can see the mod. But when I go back to the properties screen or press refresh, the version number is gone again.

I would like to know a bit more about the diff's. I'm hesitant on making changes as I see a location indicator of the change (@@ .....@@). I don't know what happens if I remove / add something from a diff. No hurry though as for now I can just ask you to make it work for me ;)

That's wery strange indeed. I'll create the special debug version in the near future so we can get to the bottom of this.

Each line beginning with @@ starts a new diff block. The first pair of numbers is the starting line and line count in the original file and the second pair has the same information for the modified file. Civ4Modder ignores everything except the original starting line (even that can be wrong and it'll still work).

The block data consists of lines, prefixed with a character telling the type of the line. ' ' means a "context" line, i.e. it is the same in both files and is used to find the correct location. '-' is a removed line and '+' is an added line. When applying a diff, all context and removed lines must match to the original, otherwise an error is thrown.

Moving on to the problem with MSDA and EFA... If we take a diff of the CvScreensInterface.py file from both, we get the following (I've cut some unnecessary stuff):


$ diff -u Assets/Python/EntryPoints/CvScreensInterface.py Mods/ModifiedSpecialDomesticAdvisor/Assets/python/entrypoints/CvScreensInterface.py
@@ -1,7 +1,14 @@
## Sid Meier's Civilization 4
## Copyright Firaxis Games 2005
import CvMainInterface
-import CvDomesticAdvisor
+
+## 12monkeys - Modified Special Domestic Advisor - begin
+##--------------------------------------
+## import CvDomesticAdvisor
+import CvModSpecialDomesticAdvisor
+##--------------------------------------
+## 12monkeys - Modified Special Domestic Advisor - end
+
import CvTechChooser
import CvForeignAdvisor
import CvMilitaryAdvisor

$ diff -u Assets/Python/EntryPoints/CvScreensInterface.py Mods/ExoticForeignAdvisor\ v0.96/Assets/python/entrypoints/CvScreensInterface.py
@@ -3,7 +3,10 @@
import CvMainInterface
import CvDomesticAdvisor
import CvTechChooser
-import CvForeignAdvisor
+# Exotic Foreign Advisor Change 1/2
+#--------------------------------------
+import CvExoticForeignAdvisor
+#--------------------------------------
import CvMilitaryAdvisor
import CvFinanceAdvisor
import CvReligionScreen


After applying the first one, it looks the same as the file in MSDA:


## Sid Meier's Civilization 4
## Copyright Firaxis Games 2005
import CvMainInterface

## 12monkeys - Modified Special Domestic Advisor - begin
##--------------------------------------
## import CvDomesticAdvisor
import CvModSpecialDomesticAdvisor
##--------------------------------------
## 12monkeys - Modified Special Domestic Advisor - end

import CvTechChooser
import CvForeignAdvisor
import CvMilitaryAdvisor


Now, when the program tries to apply the second diff (from EFA), it first tries to find a line that says "import CvMainInterface". That's found at line 3, since MSDA didn't change it. However, according to the diff, the next line should be "import CvDomesticAdvisor". But since MSDA changed this, the program doesn't find what it expects and decides "nah, this block didn't fit here after all" and goes looking for more possible positions to put it in. There aren't any in the file, so it eventually throws an error.

And what's the solution? Remove the surrounding context from the diffs, leaving only the removed and added lines. Now the program will only look for that one line and replace it with another. Since the mods change different lines, they won't interfere with each other.

Thanx for the explanations. I'll probably need it when incorparation more mods.

Please contact me if you need some testing done for resolving the bug, or other bugs/features etc. I'll be happy the help.

Bug Update,

The version is only deleted when I restart or press refresh. If I check the preferences when the mod is visible in the modlist, then the version number is displayed.

Any new developments?

I was thinking that their should be a collection of pre-diffed mods made so people can easily download and merge them. I think it would be prudent to have a thread under ModComponents page with a number of the popular mod Components for download as well as a post onthe completed mods page with diffs of the popular compilation mods. People can then do their own add-ons much more easily.

I've been pretty busy actually playing the game, sorry :mischief:

I'll be adding the quick merge feature next and maybe fixing some of the existing problems listed in the readme.

There's one thing I'd like some feedback on: dynamic context size for python scripts. Way back at version 0.2 or so there was some talk about this. However, I'm a little sceptical about whether mod makers will use the feature, and it's about as easy to edit the diff by hand with the built-in editor. Basically, it would work like this:


import Module1
#!context 0
import MyNewImport
#!context default
import SomeOtherModule


The program would not generate context for the bolded lines, and thus would ignore any possible changes to the other imports when merging. Given the fact that mod makers probably won't use this, do you think this is worth implementing (1) before version 1.0 or (2) at all?

As for the pre-diffed mods... Did you notice I already put some up at http://civ4.tdb.fi/?page=mods? Feel free to submit additions.

Yes I know what its like to get distracted actualy playing the game ;)

by the way the link is broken, and I agree that feature is very unlikly to be used by mod makers so theirs little point in including it.

You mentioned that diffs are applied one by one. I had always though their was some means of doing them all at once. My thoughs were something along the lines of...

The first hunk of each diff is read and its line recorded, the line numbers are compared and the lowest line number/first diff pointer and linenumber are recorded in another location. The original files are copied untill the first hunk line is reached, then the hunk is writen in. The diff that it came from then advances to the next hunk and lowest-line-hunk is checked again and updated. When the lowest-line checking finds that 2 or more hunks modify the same line then it throws a conflict and we go into conflict resolution mode. This might make it easier to resolve sticky situations as all the relevent diffs are on-hand, it also allows for a 3+ way conflict to be resolved better as the user can be presented with all the conflicting data at once rather then just 2 at a time.

Seems like the forum decided to interpret the end-of-sentence question mark as part of the url. The correct one is:

http://civ4.tdb.fi/?page=mods

It would basically be possible to apply all diffs at once as you suggested. The result would be the same however, and it wouldn't make automatic conflict resolution any easier. It is also possible to collect all mods that conflict on a single file and ask the user to order them, but this will create complications if we also want to report all of the conflicting files for a single mod pair. I'll have to think about ways to include both of these.

I'm making progress with the mod database, and decided to publish a WIP version of it. Take a look at:

http://civ4.tdb.fi/?page=moddb/index

Not much mods in it yet and very sparse information available for those few that are, but it's a start. I'll continue adding features tomorrow.

tdb,

I checked out your website and created an account. Looks like a very good start. When I'm back from the hollidays I'll try and get more involved again and start uploading mods I use.

I noticed the settings-page not working (yet)

Yeah, settings and search are not working so far. The settings page will have stuff like default display settings and whatnot, I don't know exactly yet. Search should be pretty self-explanatory.

BTW, there's a thread specifically for the database project here:

http://forums.civfanatics.com/showthread.php?t=174590

I should have only posted a link to that thread here to keep discussion more on-topic, but I only remembered it after posting here. Kinda thread-jacked my own thread :p So now I kindly ask everyone to move database discussion there.

For some on-topic stuff, we're nearing version 1.0 with Civ4Modder. According to the roadmap in the first post, the only major feature missing is quick-merging. I haven't received complaints about mod creation errors after releasing 0.5, so I suppose the XML file support is in pretty good shape. If I don't receive any complaints, the next version will likely be 0.99 (beta1).

New problem to think about: SDK mods. They require putting the CvGameCoreDLL.dll into the main Assets directory. Should the program do this? Should it backup the original automatically? If so, how would it know the original from some other mod's dll?

New problem to think about: SDK mods. They require putting the CvGameCoreDLL.dll into the main Assets directory. Should the program do this? Should it backup the original automatically? If so, how would it know the original from some other mod's dll?

Actually, SDK mods don't need to be placed in the "main" directory, if I'm on your same wavelength. You can place them in your mods assets directory, just like you would "replace" any other file. You should never need to replace the original.

The real problem you'd have is combining two mods that both have DLL's. I'm going to say save yourself the trouble and just say combining two mods that both use DLL's will not work :P

Keep up the good work!

Hm, I tried that a while ago with the 3-square city radius mod, but got a CTD just like with the original DLL.

You're right, I'm going to say combining DLLs won't work. But there are many cases where only one mod needs a custom DLL.

I've noticed the subject of combining mods cropping up on this forum quite regularly. If you notice a thread asking about it, please go and post a link to this thread.

More things to think: Map scripts. I think they don't work from mod directories. Should putting the scripts to the PublicMaps directory be supported by Civ4Modder?

More things to think: Map scripts. I think they don't work from mod directories. Should putting the scripts to the PublicMaps directory be supported by Civ4Modder?

You can put maps into a PrivateMaps directory in the mod folder. Not sure how that works in your plan, however. Typically, the reason people put the maps in PublicMaps is because A.) The map or script is for the vanilla game or can be used for any mod (thus, it's normally just a stand-alone map, and doesn't come with a mod), B.) It comes with a mod, but the owner doesn't know about the PrivateMaps folder, or C.) It comes with a mod but the owner thinks that it should go in PublicMaps because it should be usable with other mods.

If you're creating a tool that allows people to combine multiple mods into one mod, would it be possible to take all the mods they're combining with maps in the PublicMaps and put them into the PrivateMaps of the new mod you're creating for them?

If you're creating a tool that allows people to combine multiple mods into one mod, would it be possible to take all the mods they're combining with maps in the PublicMaps and put them into the PrivateMaps of the new mod you're creating for them?

Yes, as long as the mods are consistent in the place for storing that script inside the directory hierarchy. However, if mods don't really have a PublicMaps directory, that can get a bit tricky. It's not exactly safe to assume that every .py file in the top level is a map script.

Yes, as long as the mods are consistent in the place for storing that script inside the directory hierarchy. However, if mods don't really have a PublicMaps directory, that can get a bit tricky. It's not exactly safe to assume that every .py file in the top level is a map script.

That's true enough. I guess for now your hands are tied, but I think putting in support for the PrivateMaps for now would be a good step forward so that when more modders realize that their mod isn't working with this, they'll start using the PrivateMaps folder.

Just so you know: I'm not dead. My interest has just been wandering elsewhere (like factoring large integers) for a while. I'll resume playing Civ4 and working on this project shortly.

Hey, I tried using this mod, and I wanted to combine the Revolution (http://forums.civfanatics.com/showthread.php?t=171127) mod with another mod. I like the Revolution mod because it's quite realistic. When I try to combine it, I get the error "Unable to apply diff" and I just wanted to know what the problem might be, so at least I could understand the error. Thanks!

That error arises when two mods try to modify the same Python file at the same or near enough locations. The program should tell you which file it failed at. Armed with that information, examine the diff produced for that file in both mods (through the mod properties dialog). It may be beneficial to copypaste one or both into notepad so you can examine them side by side. Remove any conflicting context and you're set. (Check my explanation about MSDA and EFA at post 174 (http://forums.civfanatics.com/showpost.php?p=4237462&postcount=174))

Ah, I've got a problem. It only comes up with an error window that reads "Unable to apply Diff".

so in thoery i can combine dales combat mod with sevo 3 mod

modified Dll aren't mergable, you would need to put the Dale Dll into SevoMod, but you would then lose any features that Sevos Dll provided, any it might not be stable to boot.

From what I've seen, the game tends to crash if you have the wrong DLL present. There are basically two approaches for solving that: either create a main DLL with plugin interface for sub-DLLs, or create one DLL which incorporates features for all mods and allows toggling them with XML. The Civ4 Core Community Project is doing the latter, AFAIK.

As a side note, I'm TRYING to get work done for this program. I'm just getting distracted a lot lately...

Just downloaded your program (thanks for creating such a useful tool, by the way!).

I unzipped it and it went into it's own "civ4modder" folder but when I try to run the 'civmodder.exe', I get an error message that "... libatkmm-1.6-1.dll could not be found".

I'm running Win XP-64 if that makes a difference.

Thanks for your help.

**EDIT** I just read your first posting and realized I had not installed the gtkmm runtime yet - as soon as I followed your instructions (DOH!), everything works fine. I did notice, however, when I went into the "About" menu, it still said version .4 (sorry, I work as a software tester and so notice picky details like this)

Good catch withthe about dialog. I'll make sure it's correct in the next version.

Hello guys. i get an error when trying to create diff files for some mods. i can merge so mods with no problem. I went to create mod, selected Regiments 3.0 and clicked ok, everything's working but about halfway through i get an error that says
"Value element Fader has children. Maybe it isn't really a value element?":confused:

Hi everybody,
English is not my thongue by a longue way so I will do short.

When trying to use Civ4modder, it reject 3 mods out of 4 with the following message : Value element "a xml files(never the same)" as children. Maybe it isn't a value element?
Then it don't load the mod in question.
I was thionking that maybe it was just a difficulty with warlords so I try with SevoMod for civ4 vanilla. The program crashed.

I had great expectations in this utility because I can mod xml my self but do not understant a bit of python and even less in core moding.

I cincerly hope it will be fixed someday.
Thanks tbd anyway: You are the only one who try to help dummies like me this way.

The elements.txt file needs to be updated to include references to the type of file in your mod, most major files are in but a few arn't included. Have a look at the way its currently done and try adding them, I've done this quite a bit myself and it works.

SevoMod is too large (I think) and crashes do to some kind of overflow, smaller mods will work though.

Thank You Impaler !

I did what you said and I was able to "create" the mods I wanted to add.
Now, a new problem occur (i'm not sure of the speling...) apear, when trying to "mix" the mods together, it give me an error message that say : "XML error in mods\JRandomNames.Civ4Mod: Document not well-formed."
We well know the other of this mod do it's job quite properly so i'm I don't know what to think.

I obtain this message with other mods too...

I think I will try to do my mods-mixing by hands... A new string at my bow and this kind of thing.

Well, thanks you a lot anyway

can someone please post some new links for the "gtkmm runtime environment" files ?

do you think you can give instructions on how to set this up with warlords?

Seems like I did the disappearance act again... I guess I need to explain myself.

Short version: I have some other things in my life now, and am not playing Civilization IV. I will pick it up again some time in the future, but I can't say when.

Long version: I know some people can focus on a single game for years. I've even talked (through IRC) with one person who said he has nothing else to do except playing World of Warcraft. However, I'm not one of those people. While I don't have any real attention disorder, I just can't keep playing only one game, no matter how good it is. Other things (games, programming, model railway) will creep up to fight for my attention. Eventually something happens in the game that makes me not want to play it right at the moment, such us losing by a few turns in Civ4 due to a neighbor declaring war on me while I try to win the space race. I start doing one of those other things, but by the time I've cooled down about the game, my interest has already shifted and the new thing now feels more interesting.

If someone wants to pick up the source and continue developing this program while I'm absent, feel free to. If you want your changes incorporated to any future versions I will create, post them as a patch file (diff -ur olddir newdir) along with an explanation of what you've done.


To answer some of the questions expressed here:

- "Document not well-formed" errors are most likely caused by invalid characters in the xml file. IIRC the XML library I'm using assumes UTF-8, so if you put Latin-1 characters in a file without an encoding header, you get that error. Try adding a proper encoding header or removing any non-ASCII characters.

- The gtk and gtkmm links in the first post still seem to be good.

- I still don't have warlords so I can't give any exact instructions, but if you tell what's wrong I can try to help.

It's nice of you to drop a notice. A lot of others don't do this and it can be a bit frustrating when a favorite mod / util / etc is stopped being developed before it's fully finished.

But hey, it's all voluntarilly. Anyaddition is greatly appreciated, at least by me. So basically no explanation of absence is required.

I just suffered a Civ-less period of about 4 months. Just finished a game back then that was very intense, and couldn't bring myself to go through it again.

Anyways, have fun with other things (Sid Meiers Railroads! perhaps??)

The link that leads to (gtkmm-runtime-2.6.2-1.exe). is not working. I went to the link to http://www.pcpm.ucl.ac.be/~gustin/win32_ports/ to look for (gtkmm-runtime-2.6.2-1.exe). but i can't find it. The only gtkmm files i found were different versions. Does it matter which version i use?

I followed the installation steps, installed both the requires programs and yet the program refuses to work and asks for a missing "libatkmm-1.6-1.dll". Then after i find it on the internet and place it in the civmodder folder it refuses to work again and asks for a missing "libatk-1.0-0.dll". After i place it on the folder it asks for another file(all of them are dlls with names staring with "Lib") and it keeps going like this till i gave up(after 22 dlls downloaded...)

Can anybody help me?

General Kohl,

This is probably not worth pursuing as this utility has been abandoned 1.5 years ago. The runtime environment it depends on has evolved and civ4 as well. Back then it worked to a certain extend but it didn't work on all mods. Probably now it doesn't any more.

I plan on using it to merge old Warlords mods =O

But if a new tool works better, i would be thankfull if you could provide me a link to it.

I think warlords is also too new for this utility see last sentence of the last post of tdb (#206)
IIRC the xml has been made modular by firaxis from Warlords on and this could be incompatible with this mod. But I'm leaping to conclusions here. I suggest if you want to still try it, try to find the versions specified in the opening post. Also tbd had a website with compatible mods and slightly edited mods for compatibility.

Good luck