Best way to learn how to mod the SDK?

[Eusebio]

I've been trying to do some modding in civ4 myself, but I admit I'm finding some difficulties. Some of them are caused by my limited knowledge of python and C++ (and programming in general) but there are lots of places around the web where I can look for help in such cases. Some of the problems, however, are more specific to Civ4 code itself, I believe. For example, understanding how the code works, what each file and function does, how to correctly implement a new feature following the rules of the code, etc. So, I would like to know:

1- What's the best way to start learning how to mod this game in Python and C++? I know there are lots of tutorials in this forum, but is there a place containing some sort of documentation about these codes? The wiki has this page containing documentation about some of the C++ files, as well as Python, but the majority of the files aren't documented yet.

2- I am, right now, trying to create a mod for this game by editing the DLL files, do you think I should start with python and then jump to C++ instead?

3- So far, the tools I'm using to edit the code are: Codeblocks; VS2010; Notepad++. I tend to favour Notepad++ to study and edit the files themselves, though obviously compilation needs to be handled by VS2010 (I think there's a way to compile the game with Codeblocks, but I couldn't make it work yet). At this moment, I'm only using Codeblocks to search for functions declarations and things like that, Notepad++ does have some plugins to do this, but sometimes their findings are somewhat confusing. Is there another software suggested to use in modding the game?

 

[f1rpo]

I used to do a lot of full-text searches – on the DLL codebase, the Python, XML, game text folder(s). I still do that all the time really. To find out e.g. where some text fragment is coming from or what some XML variable or tag does exactly or how a particular function is (properly) used. I don't know if

limited knowledge of python and C++ (and programming in general)

is sufficient for that approach. Much of the code does not strike me as difficult to comprehend, e.g. a CvUnitAI subroutine like CvUnitAI::AI_settleMove – apart from some strange details like the PROFILE_FUNC call or getOwnerINLINE rather than just getOwner. That said, there are some important interactions that are difficult to glean from the code, such as AI_settleMove being executed for a single unit but utimately sending an entire group of units on a mission, or the allocate-uninit-reset-init/read-deallocate lifecycle of CvPlayer. This is where some (external) documentation would seem helpful. (Such broad-strokes documentation would likely also be valid for every mod, not just the original game.) Well, we don't have that, or only scattered across numerous threads. Could try Google searches like this one:

site:civfanatics.com civ 4 add new xml tag​

Stepping through an obscure section of the code in the debugger is usually illuminating (but it's tedious).

[...] do you think I should start with python and then jump to C++ instead?

My impression is that people start out with Python because they don't want to deal with compiling the DLL or don't know C++ at all. Implementing game rule changes through Python alone will often require elaborate workarounds for what would be straightforward in the DLL. This recent post seems like a pretty typical example. That's not to say that modding only or predominantly in Python can't make sense, but I don't think general lack of experience is a good reason for doing that, at least not once the DLL compilation hurdle has been cleared.

Is there another software suggested to use in modding the game?

For text search, I've been using a desktop search program named "Agent Ransack", which I launch through a right-click on a Windows folder. I've never heard of anyone else on this forum who uses that particular program (except maybe the one person I've recommended it to in the past); perhaps "Find in files" in Notepad++ is equally convenient.

Versioning software (Git is what most people use) takes a while to become comfortable with, perhaps better to put that off. Also helps more with finding bugs and sharing code with other modders than with understanding existing code. WinMerge is handy whenever it's unclear which parts of a file another modder has changed. (And for actually merging files.)

I do my C++ code editing in VS2010. (I guess I'll upgrade to a newer version of VS eventually.) Having the compiler integrated is a pretty big plus over a mere text editor: immediate feedback through error markers, auto-completion (Ctrl+Space), tooltips and "go to declaration/ definition/ call hierarchy/ header file" via right-click. I recommend adding some filters in the Solution Explorer for quick access to frequently needed files.

Edit: typos

 

[Leoreth]

From my own recollection, I share your perspective on what makes getting into DLL modding difficult. I agree with f1rpo's perspective that unless you are wary of setting up the DLL compiler, there is not much of a difference between C++ and Python modding in terms of difficulty. In fact, I would argue that only experience with the C++ code really makes you learn the API which is what you need for Python modding as well. And the difficulty of setting up the compiler is in my opinion overstated as some sort of cultural memory from the times where it was genuinely difficult. I think the tooling for that is pretty nice these days, although I may be biased due to experience.

The main problems with actually learning how the C++ code is the total absence of documentation and commentary and the fact that we are dealing with code that was written around 2004 (honestly not even up to date with the patterns of that time afaict) and not always using the best patterns. Often the answer to my initial "why is this so hard to parse" reaction turned out to be "because this is archaic C++ written by a not too experienced programmer, presumably under lots of pressure".

How do deal with that? Even though it this point I have probably touched every part of the DLL code at least once I still need to figure out where to implement a change sometimes. And like f1rpo points out, the best tool for that is backwards searching.

I consider a good IDE essential here, and imo a modern version is worth it. VS2019 has built in context menu tooling for "go to definition", "go to declaration", and "find usages" of a given variable or function (I remember VS2010 to be harder to use in that way...) and its full text search is fast and easy to use as well.

The second component is a basic working knowledge of the main components. I think the best starting point is CvInfos, which contains classes mapping on to everything that has a corresponding XML file. Often when I had no idea how a certain feature was implemented, I still knew about an XML tag associated with it (and if you don't, they are easy to find in the XML files). From there it was just a matter of finding its usages in the code. Likewise, for adding an entirely new feature it is often a good idea to identify a similar feature that is already exposed in XML, and use that as a pattern to emulate. If you can look up how a civic increases improvement yields, you are already a good way to implementing your feature that e.g. makes civics increase feature yields.

The other core classes that are worth looking into are CvGame, CvPlayer, CvTeam, CvCity, CvPlot, CvUnit. Those probably manage everything of note that can happen in the game. That also makes them very large and complex, so I am not suggesting to go through them and learn them by heart before you can make changes. But more to get a rough idea of which part of the game is probably located in which of these classes so you have a starting point for where to look.

Another one that might be worth it is CvGameTxtMgr, which is responsible for rendering a lot of the game information into text. I don't find the naming conventions in this one very intuitive but it is still helpful in the sense that you know that if something has to be displayed in the game, it is probably used in this class, which can be used to find its underlying code.

Lastly, the Python bindings are defined in the DLL (look for Cy[class]Interface). This is helpful to find if a C++ function is available in Python, to find its API/functionality, or to make a new DLL function available in Python. If you are more familiar with Python that is also a good entry point into the code.

 

[Eusebio]

Thanks a lot for these replies. I'll follow Leoreth advice in using CvInfos and CvGameTxtMgr as starting points. Aside from that, is there a place or site with some sort of basic documentation about the DLL, even if incomplete? For example, suppose I want to know how combats are handled by the game: which files should I look into? CvUnits, CvGame, etc? Or what if I'm trying to change the way movement of units work (for example, if a unit explores any unrevealed plot, it spends all of it's movement points for this turn)? Where are such systens handled by the code?
Basically, it's all about finding the correct place(s) in the code where a given mechanic (exploration, combat, promotions) is handled. Is there an advised way to do this? A link with some sort of documentation? Asking around the forum? Or going from a good starting point as Leoreth suggested?

 

[f1rpo]

[...] Basically, it's all about finding the correct place(s) in the code where a given mechanic (exploration, combat, promotions) is handled. [...]

Those are all in CvUnit, you could search the header for "combat" (spoiler: most relevant is probably resolveCombat), "move", "promo". Bad software design; imo the unit class should mainly represent the current state of one unit on the map - e.g. its location and remaining hitpoints, and the combat system and movement system should have their own classes. Well, with the classes that we have, it's still clear enough that CvUnit is going to be involved in combat resolution somehow - surely the units have to take damage at some point -, so that's at least an obvious starting point when searching for the pertinent code. And, on the bright side, a surgical rule change like this ...

(for example, if a unit explores any unrevealed plot, it spends all of it's movement points for this turn)

may take just one line of code to implement, so finding the right spot is really the main effort. That being said, proper user interface and AI support is often the (much) bigger task. In this example, both probably come down to the pathfinder, which gets used by both AI and UI (waypoints in Go-To mode). To get the pathfinder to work, it might suffice to implement the exhausted movement points through CvPlot::movementCost - a function called by CvUnit::move but also by the pathfinding code in CvGameCoreUtils. In contrast, if CvUnit::move just takes away all remaining moves, the pathfinder won't be aware - as CvUnit::move only deals with actual moves, not hypothetical ones.

UI code in the DLL appears in some pretty obscure places. Some in dedicated classes (e.g. CvGameTextMgr), some in the game state/ logic classes (e.g. CvPlayer::getGlobeLayerColors, all CvGame functions in CvGameInterface.cpp), some in the CvDLL...IFaceBase headers (implemented in the EXE).

Asking around the forum? Or going from a good starting point as Leoreth suggested?

I see nothing wrong with asking in the Quick Modding Questions thread. Having to wait for a reply there sure holds things up, so, if you can find the code you need through searching and inspecting the codebase or the forum, that would seem preferable.