The Modder's Guide to Civ4

[WildWeazel]

DON'T PANIC!

"Civ4 is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to XML, but that's just peanuts to Civ4." -Douglas Adams, and me


So you pre-ordered the game, you've seen every screenshot, and you've been scouring the forums for months. You finally got your hands on that game, and you just can't get enough of it. And if you're like me, there's only one thing on your mind: how am I going to mod this amazingly deep and complex game?

If you want to mod Civ4, this thread is for you. I'm digging into the modding capabilities of Civ4, and sharing all of it with you. I will post everything that I know in this very thread. And if I don't know it, I'll link to it.

Whether you're a conscript newb who has never played Civ before, a regular modder like me who hasn't quite mastered all there is to Civ3 yet, or a veteran programmer who just can't wait for the SDK, this thread is for you.

QUICK LINKS:
Post 2: Modding Overview
Post 3: Graphics
Post 4: World Builder
Post 5: XML Modding
Post 6: Python Scripting
---------------
Graphics
DDS Converter
jak08's DDS Skin Tutorial
RAD Game Tools

World Builder
Dale's World Builder Manual
BMP to WBS Converter
Dale's In-Depth Look at the WBS

XML
Civ4Wiki
W3Schools.com XML tutorial
XML.org Beginner's Guide
BrentRichie's XML Tutorial Part 1: Adding Civs
Part 2: Civ Info
Part 3: Leaders
XML Quick Reference by Max_x_fire
Adding New Units by sovarn

Python
Civ4Wiki
www.Python.org
GBM's Python Tutorial and Website
Locutus' Python Documentation
Civ4 Python API

[WildWeazel]

Modding Overview:


As you probably know, there will be 4 levels of modding in Civ4:

World Builder: This is the equivalent of the map editor in Civ3, but much more powerful. Here you can create custom maps and basic scenarios. You can edit terrain, place cities and units, and much more. The world builder will ship with the game.

XML: This is roughly equivalent to the rules editor in Civ3. This is where you can add units and improvements, edit the civilizations, and tweak just about every value in the game. XML can be written in notepad or an XML editor. XML is similar in appearance to HTML, and is fairly simple to learn once you understand the basic concepts. A great XML tutorial can be found at w3schools.com

Python: Python is a programming language that is used in Civ4 for the game interface, map generation, scripting events, and more. Gingerbread Man is building an excellent Python tutorial focused on Civ4. Several lessons are already available. Python can also be written in notepad. The Python Integrated Development Environment (IDLE), which allows you to run Python files, can be downloaded for free at www.python.org.

SDK: The Software Development Kit is where the hard-core modding will be done. Everything but the graphics engine and the core Civ engine will be available to modders. Programming will be done in C++. The SDK will not be available until early 2006.

Graphics: In addition to these forms of modding, there will also be a demand for graphics. Units, icons, and the user interface will all require new graphics as new mods and scenarios appear. Depending on what graphics are involved, they may require XML, Python, or even the SDK in order to be incorporated into the game.

Creating a mod
Once you find something you want to change, you can make a mod for it. When you create a mod, no matter how small, create a new folder for it inside the Civilization 4\Mods folder with the name of your mod. DO NOT overwrite the original files with your own. If you are making a lot of small changes to the game to fit your own preferences, you will probably want to play this mod every time, instead of the standard game.
Fortunately for you, you can tell Civ4 to run a certain mod every time it loads. There are a few ways to do this. The first is the Custom Assets folder. You can store files in here that you want to override (not onerwrite) the original files every time the game loads. When Civ4 loads, it first takes these files, then follows the designated file path. The Custom Assets folder is a good place for small changes such as textures and text that you want to use for all games.
If are creating a specific mod that you want to load automatically when Civ4 runs, you can direct the game to it from the main configuration file. In _Civ4Config.ini, in the main Civ4 folder, there is a line "Mod = 0". By setting this to a specific folder, such as "Mods\MyMod", you can make Civ4 load a certain mod automatically.
It is not necessary to copy all files into your mod folder- only those that you have changed from the originals. Like Civ3's scenario search folders function, Civ4 will look first in Custom Assets and take any files it finds. It then moves on the to the specified mod, if any, and takes whatever files that it doesn't already have, before getting all remaining files from the main folder setup.

CLEAR YOUR CACHE!!!!!11!!!1!!!1!1!11one1!!!
If your mod is not loading properly, (ie, the game says it loaded but nothing changes), you may need to clear your cache. Civ4 saves a lot of files in a cache folder and uses them instead of reloading the files, so sometimes changes don't appear as they should. Go to C:\Documents and Settings\USERNAME\Application Data\My Games\Civilization 4, show hidden files and folders, and delete the cache folder. You may need to do this periodically as you make changes.

Mod Structure
The Assets folder, which contains all files necessary for a complete mod, contains a massive 559 folders, and over 10,000 files. Fortunately for you, you don't have to duplicate it. Unlike Civ3, when you had to create the folder setup to mirror the original, Civ4 does this for you. When you run the game using a mod, it automatically creates a full folder setup in the mod folder. As you add more files to your mod, simply place them in the same sub-folder that the original was in. The basic unpacked* structure looks like this:

*If your Civilzation 4\Assets folder DOES NOT contain these subfolders, you will need to decompress the file art0.fpk with the PakBuild utility from Firaxis. art0.fpk is a huge compressed file similar to a RAR archive that contains most of the files and folders from Assets. After you unpack the file, move art0.fpk to a different location or rename it. This is because Civ4 will look for this file before looking in the Assets folder. If you do not move or rename the file, Civ4 will continue to load the default files within the package.

Basic guidelines
As with all modding (or anything in life for that matter), there a few guidelines to follow:
Back up your work. Before you start messing with anything, save a copy of the original (or the latest version of your own work, for that matter) in a place where it won't get deleted, overwritten, lost, or eaten. The only way to recover the original is to reinstall the game. And trust me, your hard drive doesn't want that. And whatever you do, don't come begging for somebody to send you an original file. It's called piracy. And Chieftess will eat you alive. You think I'm kidding, don't you?
Use the Mod folder setup. In other words, DO NOT replace the original files with your own work. First of all, if you did something wrong you might have to reinstall to fix it. Secondly, you may want to play the game as it originally was, instead of your modded version. And thirdly, you won't get any sympathy when you complain about your game being messed up. The Mod folders work the same way as they did for Civ3- just put whatever files your mod or scenario needs into its own folder, copying the setup of the main folder structure. For example, if you create a new XML file, put it in Mods\MyModName\Assets\XML\
Ask questions. If you don't know how to do something, or want to learn more, don't be afraid to ask. We're all learning this together (except for those beta-testers ) OTOH, some of us have been modding Civ3 for a while, and some of the same concepts will carry over. We're happy to help new modders, because that means (a) more people to help us make our mods and (b) more mods for us to play. No, seriously, we help because we enjoy modding and want other people to enjot it as well. So jump right in and start learning!
Have fun. This is the most important thing to remember. It's a game. It's meant to be enjoyed. You can gain a lot of computer-related skills by modding, but that's just a bonus. The real benefit is doing something you enjoy, seeing the product of your hard work and dedication, and working with some really cool people along the way. That's what C&C is all about.

[WildWeazel]

GRAPHICS

Graphics in Civ4 are more complicated than in previous games. Because of the 3D environment, most graphics consist of models and textures instead of single images or animations.

Units
We are not currently able to create new models and animations. Until Firaxis releases the SDK or a converter plugin for the Gamebryo engine, we have to use the models that shipped with the game. Gaias explained it to me in another thread:

Originally Posted by Gaias

What I was trying to infer with that statement was that even though I did export a basic primitive, it doesn't mean that it would actually work for all that is required to make a unit in Civ4. The plugin in question that I used with Milkshape 3d was a alpha opensourced project that was geared initially toward The Elderscrolls Morrowind (which also used the GameBryo engine). I say 'was' because said plugin is close to a year old and alpha notheless (all I could do was export the basic primitive into the game and nothing else, ie animations).

The other problem is that even though this instance did work the plugin was geared toward a different game entirely. Morrowind came out in 2002, and the authors of that plugin said explicitly that they were working on the NIF file format that was used in the game. Now I am pretty sure that the makers of GameBryo have upgraded their engine in that 3 year time. I am also assuming that Firaxis used a more recent incarnation of that engine. Just by an educated guess I would be lead to believe that even though both NIF file formats are in general the same, they may vary enough as to not be cross compatible. And that is the real problem.

Unless someone with good C++ and a knowledge in 3d applications wants to design a exporter plugin for one of the various 3d applications available to the public, we are just going to have to wait until Firaxis release their SDK with a NIF exporter. Until such time, things like units and building are not going to happen. Well at least not very easily one might assume.

While we can't make custom models yet, that doesn't mean we can't have new units. Units can be "reskinned" by editing or replacing the textures. With a DDS Converter, you can export the dds texture files to standard bmp images, edit them, and convert them back. A single model can be used with dozens of different textures to create a variety of custom units. jak08 wrote a short DDS Skin Tutorial.
Buildings
Leaderheads
Terrain
Buildings, Leaderheads, and terrain all use the same graphics format as units- a combination of NIF, KF, and KFN files, with DDS textures. Terrain is especially complicated. It encompasses a massive 1,217 files distrubuted among 168 subfolders. Terrain modelling in Civ4 is not for the weak.
Buildings and Leaderheads are somewhat limited at this point because buildings are not culture-specific, and LHs are not era-specific. Until the SDK is released, These graphics will be limited to simple skin replacements only.
Interface
The game interface is comprised mainly of DDS textures, making it the most moddable set of graphics at this point. The interface is run by Python.
Movies
Polyphemous discovered the program used to make bik movies for Civ3, and it turns out that Civ4 uses the same format. The Bink Tools program is a free download from RAD Game Tools.

[WildWeazel]

The World Builder

The World Builder (WB) is the part of Civ4 that is used to create new maps and scenarios. It can be loaded at any time during the game. Many people find the WB slow and clucnky, but fortunately there are other ways to edit the saves. An interesting thing about the World Builder Saves is that they are saved in ordinary text format. You can open, edit, and save them using the most powerful computer program known to man- NOTEPAD. The file is easy to understand, and for small changes it's probably easier to tweak the text rather than loading it in the World Builder. There is also a converter that creates maps from a series of bmp images.

Manual: Dale (who turned out to be another one of those elusive beta testers) has made a great World Builder Manual in pdf format.
Manual (7.1 mb)

Making Maps: There are 3 ways to build new maps. You can use the WB to make it in-game, you can manually set it up in Notepad, or you can use the BMP to WBS converter to generate maps using bmp images.

The World Builder Save (WBS): Dale is also working on a complete guide to editing the WBS in text format in this thread.

Unlock the full capabilities: If you have tried to use the World Builder, you have probably noticed that a lot of the functions have been disabled. This is to keep the World Builder from being a cheat machine, but you can't make good scenarios. Fortunately, Soren Johnson released the "cheat code": In the main ini file, which can be accesed through "Shortcut to _Civ4Config" in the main Civilization 4 folder, replace the line "CheatCode = 0" with "CheatCode = chipotle". This will unlock all the features of the World Builder, allowing complete scenario control.

[WildWeazel]

XML Modding

XML is short for Extensible Markup Language. It appears similar to HTML, but serves a different purpose. While HTML displays information, XML is used to organize information. The unique thing about XML is that there is noset, predefined system. When you use XML, you make up your own tags based on what your program needs to do. XML itself is not a programming language. It's a "metalanguage"- it's used to define new languages.

So what does this have to do with Civ4? Civ4 uses XML to organize information about the game. Instead of the complicated rules editor from Civ3, with obscure labels and buggy functions, Civ4 uses XML to arrange all of the necessary information so that it is easy to read and edit.

I have found 2 good online XML tutorials that I would recommend. They are much more in-depth than my simple introduction, and are very easy to follow.
w3schools.com: XML Tutorial
XML.org: XML Beginner's Guide

(If you want to jump right in and see some XML in action, keep reading. If you just want to know what this has to do with Civ4, skip this section.)

--------------------------------------------------------------------------
--------------------------------------------------------------------------

Spoiler for XML crash course:

That's all great, but what does XML look like, and how does it work? If you are comfortable with HTML, or even the VB code used in your posts, then learning XML should be no problem. It consists of tags like <b>...</b> that wrap around pieces of information to describe them. To see a good example of XML in use, look no further than your PMs! You can download PMs in XML format- there is an option in the menu where you delete them.

As an example, here is a PM that I sent to myself:

  <?xml version="1.0" encoding="ISO-8859-1" ?> 
- <!--  Civilization Fanatics' Forums;http://forums.civfanatics.com 
  --> 
- <!--  Private Message Dump for User Weasel Op; Oct 25, 2005 09:31 AM 
  --> 
- <privatemessages>
- <folder name="Inbox">
- <privatemessage>
  <datestamp>2005-10-25 09:31</datestamp> 
  <title>XML in Civ4</title> 
  <fromuser>Weasel Op</fromuser> 
  <touser /> 
  <message>Hey, did you know that Civ4 uses XML? :D</message> 
  </privatemessage>
  </folder>
  </privatemessages>

Now that probably looks awfully complicated, especially if you have never seen HTML, but don't worry. I'll explain how it works.

<?xml version="1.0" encoding="ISO-8859-1" ?> 
- <!--  Civilization Fanatics' Forums;http://forums.civfanatics.com 
  --> 
- <!--  Private Message Dump for User Weasel Op; Oct 25, 2005 09:31 AM 
  -->

That part just tells the program reading the file, and you the coder, what this is. The first tag informs the program that this is an XML file, written in version 1.0, and that it uses the Western European character set. The next 2 lines are comments- notice the <!-- and -->. Everything between those is a comment, and is ignored by the program. It's just there for you, so you know what this is.

Next comes the good stuff:

- <privatemessages>
- <folder name="Inbox">

The first tag that really does something is the <privatemessages> tag. This tag tells the computer that everything from this tag through the closing tag (a tag with a / before the name, in this case </privatemessages>) is an element. An element is a piece of information. It can contain other elements, text, or both. In this case, <privatemessages> begins an element that contains all of the private messages in the file. (There is only 1 here, but if I had saved my entire Inbox they would all be in there.)

Next is the <folder> tag. Notice that this tag does not only have a name, but a description. This description is called an attribute. An attribute is a little piece of information included in an element tag. It gives the element a characteristic, in this case the name "Inbox." This is an example of the flexibility of XML. "Inbox" could have been a separate element within "folder" like this:

- <privatemessages>
- <folder>
- <Inbox>

Here, "Inbox" is an element within "folder." However, "Inbox" is better as an attribute because logically, "Inbox" is a specific folder, not a part of a folder.

Are you getting the hang of it yet? Let's move on. This is where things start getting interesting:

- <privatemessage>
  <datestamp>2005-10-25 09:31</datestamp> 
  <title>XML in Civ4</title> 
  <fromuser>Weasel Op</fromuser> 
  <touser /> 
  <message>Hey, did you know that Civ4 uses XML? :D</message> 
  </privatemessage>

That's a big chunk of code, but if you look closely it folloes the same pattern. This is the PM itself, the actual information that I sent myself. Notice the <privatemessage> tag. This is NOT the same as <privatemessages>. That tag defined everything in it to be a group of private messages, but this tag defines itself to be a single PM. Look at the bottom line. See that closing tag? It has the same name, but a slash before it: </privatemessage>. This is the closing tag for <privatemessage> indicating the end of that element. Everything in between, including the tags themselves, are part of the element.

The second line gives us the date and time the message was sent. The <datestamp> tags wrap around it, indicating that it is also an element. Similarly, the next line is the title of the PM, and the line after that indicates who the message is from. Keep in mind that each of these elements are part of the <privatemessage> element, which is part of the <folder> element, which is in turn part of the <privatemessages> element.

The next line looks a little different. This tag combines the opening and closing tags into one. Since the line is blank, there is no need for two separate tags. Adding the slash to the end after the slash is like xml shorthand- it opens and closes the element with the same tag. The "touser" element is blank becuase this is my own PM. If it were addressed to someone else, it wouldn't be in my inbox!

The next element is the heart of it all. This is the actual text of the PM, enclosed in the <message> tags. Depending on the PM, this element could inclde several lines of text, VB code, etc. But the important thing to note is that it's all text. There is nothing magic about XML. There is no compiler, no functions, no special characters. Everything is saved, sent, and edited in text format only.

As I mentioned, the last line indicates the end of the <privatemessage> element.

Now let's look at the last piece of code:

  </folder>
  </privatemessages>

By now you can probably guess what these do. </folder> closes the "folder" element. Notice that the attribute "Inbox" is not included here. The attribute is a part of the element, not an element itself. In fact it can't be closed because it was never opened. It's just a characteristic of the "folder" element. If there's no more "folder", it can't be an "Inbox". The final line closes the <privatemessages> element.

Now you may be wondering, why are there 3 elements inside of each other like that? If you look back at the entire piece of code, you will see that the 3 major elements open and close together:

<privatemessages>
- <folder name="Inbox">
- <privatemessage>
.
.
. 
  </privatemessage>
  </folder>
  </privatemessages>

Why couldn't those all be one element? Actually, the only reason they appear that way is because I only saved 1 PM. If I had downloaded all of my PMs in one file, it would be different. <privatemessages> would still contain everything, because that is the root element. That just means that is the primary, basic element of the file. The whole file is essentially one element, <privatemessages>. Inside <privatemessages> would be two <folder> elements: <folder name-"Inbox"> and <folder name="Sent Items">. After the first </folder> tag, the second folder would then begin:

</folder>
<folder name="Sent Items">

The same would apply to <privatemessage>. Each PM would be a separate element, and after each closed the next would begin, until the end of the folder.

Now that wasn't too hard was it? Once you understand the basic concepts of how XML works, it's a simple process.

Now I know you're probably wondering, how does that help me mod Civ4? PMs have nothing to do with it. That's right, but you will be surprised at how similar it will be.

--------------------------------------------------------------------------
--------------------------------------------------------------------------

XML in Civ4

You can find detailed information about the XML used in Civ4 at Civ4Wiki. Civ4Wiki includes pages for each XML file, with an explanation of every tag. Wiki sites allow users to submit their own material, which means that if you find an empty or incomplete page, you can update it yourself.

Max_x_fire wrote an XML Quick Reference, which covers all 129 files and includes a basic description of what each file is for, in a neat Excel spreadsheet. If you don't know which files you need to modify, check this out.

BrentRitchie's XML Tutorials provide a more in-depth look at specific tasks dealing with XML:
Adding new Civs
Civ Information
Leaders

[WildWeazel]

Python Scripting

Python is a relatively new object-oriented programming language that is used for various things in Civ4, including map generation, event scripting, diplomacy, and the user interface. Python is available as a free download from www.python.org

Python is a portable language. It can be written once and run on any system that has a Python development environment installed. Python is also an interpreted language, meaning that it is compiled as it is run. This means that you can write the code and immediately execute it without the need to compile it into binary code like most other languages. The downside to this is that Python does not operate as quickly as other languages, making it unsuitable for the core game code.

Like any new language, Python takes time and practice to master. If you would like to learn how to use the Python language, Gingerbread Man (another beta-tester) has compiled (no pun intended) an excellent tutorial. The tutorial is simple yet thorough. GBM will have you writing Python code in no time.

Once you have a basic understanding of how the language works, you will probably want to start digging into Civ4. Locutus (also a beta-tester) explains the terms used in the Civ4 Python code. The Civ4 Python API, which provides additional information for programmers, is also available online.

The Python files for Civ4 are located in the Assets\Python folder. The files can be opened, edited, and save using Notepad, although Python editors are available. Remember when you edit these files, DO NOT overwrite or change the originals. Create a folder in the Mods folder for your creations.

[WildWeazel]

*SDK Programming*

[WildWeazel]

*Future home of modding FAQ*

[WildWeazel]

Okay, you can post now if you have any specific questions or comments about modding. I will address them if I can.

[Pascal]

Will it be possible to improve the World Builder using the SDK or Python?

[WildWeazel]

Good question. I never thought about that before. I assume that since the world builder will be an integral part of the game this time (you can even use it during a game to change things while you play), it would be part of the game itself, and would fall under the "everything but the graphics engine and the core civ engine" category. So I would say most likely yes, with the SDK.
I do know that Python is used for things such as map algorithms and game scripting, which would involve the World Builder.

[MarineCorps]

I was thinking of a thread like this. As we still have no idea how moddable this game is yet.

[Civrules]

I'll see how this thread evolves and it'll probably be stickied later.

[vbraun]

I will help when I can.

[Dale]

Civrules:

If I may suggest waiting till after release, there may be something more substantial to sticky for modding.

Dale

[WildWeazel]

This will be substantial after I get my hands on the game.

[WildWeazel]

XML modding introduction added. XML in Civ4 will be added when I get the game.

[Ekmek]

What about the leaderheads are they going to use the same flcs are civ3 (if you know) or we can discuss later

[WildWeazel]

I don't know for sure, but I assume they will use FLC files. They will probably be a different size though.

[Olorin0222]

There may be more FLC files as well, since CIV LH's seem to have emotions that Civ3 LH's did not have.