Is there a high-level overview of CvGameCoreDLL?

[Vilx-]

I'm trying to get started with mod development (well, actually, to tweak Rom:AND), and since I like to go by a top-down approach in programming, I'd like to start with a high level overview of how all the pieces fit together.

So, the general idea I've gotten so far is: civ4bts.exe loads the CvGameCoreDLL, which then in turns loads & executes all the XML and Python files. CvGameCoreDLL then does all the actual game work, while calling back into civ4bts.exe for visual/sound stuff.

The next questions I have then are:

• What is the "surface area" between civ4bts.exe and CvGameCoreDLL? That is, what are the function/classes that civ4bts.exe uses from CvGameCoreDLL, and vice versa?
• Does civ4bts.exe also load any XML/Python files, or is that solely the domain of CvGameCoreDLL?
• Who is responsible for non-game logic (various menus before game)? Seems that it should be the domain of civ4bts.exe, however mods can influence the title screen, so I guess there's some collaboration. What are the functions on both sides that facilitate this?
• What does the whole life cycle look like? That is, what functions does civ4bts.exe call at various points in time? Also, this includes the main game loop - where is it defined (civ4bts.exe I presume?) and what functions does it invoke in CvGameCoreDLL?

I know this seems like a very broad list, and I don't expect that I'll get answers to all of these questions here, but anything is better than nothing, since it will save me time finding those things out myself. Thank you all in advance!

 

[Archid]

What is the "surface area" between civ4bts.exe and CvGameCoreDLL? That is, what are the function/classes that civ4bts.exe uses from CvGameCoreDLL, and vice versa?

If you look in the header files some function declarations are prefixed with DllExport, what this means is that the function is exposed to the outside world so is available for the exe to call ... So never break the signature of these methods!

Does civ4bts.exe also load any XML/Python files, or is that solely the domain of CvGameCoreDLL?

My understanding is that Python is loaded by the exe, but all the XML files are loaded by the DLL. Look at the LoadXxx() methods in CvXMLLoadUtilitySet.cpp.

Who is responsible for non-game logic (various menus before game)? Seems that it should be the domain of civ4bts.exe, however mods can influence the title screen, so I guess there's some collaboration. What are the functions on both sides that facilitate this?

The UI screens are defined in the Python files. Look at CvMainInterface.py which is the starting point for the main and city screens. The exe handles things such as pixel drawing, but a lot else is defined in the XML files in the Art folder. You really need to ask specific questions to get a response as to what can/cannot be modded.

What does the whole life cycle look like? That is, what functions does civ4bts.exe call at various points in time? Also, this includes the main game loop - where is it defined (civ4bts.exe I presume?) and what functions does it invoke in CvGameCoreDLL?

Build a debug DLL and start slapping some breakpoints around the doTurn() methods for the best way of determining what gets called when

 

[Vilx-]

Thank you!

DllExport - some of those are also because of Python, right? Is it a safe rule-of-thumb that DllExport in Cv* files are for the .exe, and Cy* files are for Python?

Python - I know that you can export functions from CvGameCoreDLL to be used in Python. But does CvGameCoreDLL not call Python functions? I guess what I really want to know is - are there any magical/sacred XML/Phython files that MUST be there, because the .exe uses them? Or are they all called from CvGameCoreDLL (so I can find all their uses there)?

Debug DLL and breakpoints - already on it. I've managed to find that the first entry (besides DllMain) is CvGlobals::setDLLIFace(). Or at least it's called VERY early. Then there's also CvGame class which seems to be at the bottom of things.

Thank you for pointing out doTurn().

 

[Archid]

DllExport - some of those are also because of Python, right? Is it a safe rule-of-thumb that DllExport in Cv* files are for the .exe, and Cy* files are for Python?

You are correct in the significance of the prefix, Cv* internal/exe & Cy* Python. This is convention rather than a requirement as far as I know, but I see no good reason to break it.

DllExport just says that on compilation they will be available to binaries outside the dll. If you are adding new functions you will never need to add DllExport as the exe won't know (or care) about new functions, and it is not required to expose a function to Python (the CyXxxInterface files do the exposure with the CyXxx classes doing the mapping to the internal CvXxx defined functions

But does CvGameCoreDLL not call Python functions?

CvDLLPythonIFaceBase.h defines the functions that allow the DLL to call python code. If you search for

gDLL->getPythonIFace()->

you can find plenty of examples of these being used.

are there any magical/sacred XML/Phython files that MUST be there, because the .exe uses them?

I have no idea, but I wouldn't go deleting anything in the Python\EntryPoints dir. There may be others that are critical, but I am not much of a UI modder and I suspect that there would be very little value in deleting a Python module. All the mods that I have seen that touch Python only ever extend it.

 

[Vilx-]

Golden. Thank you!

I don't intend to delete anything, but, as I said, I'm really trying to get my bearings here. Though it would be an interesting exercise to make a really really simple "Hello World" mod with only as little stuff in it as absolutely necessary.

 

[platyping]

Python files which are just for UI purposes, such as Pedia files or Advisor Screens, can be safely "replaced". All modders create their own mod folder, and they either "add" new things there, or write their own version to "replace" the similar file which is supposed to be used. Nobody ever goes into the default BTS folder and "delete" anything.

For instance, I can write my own Pedia and tell the system to load my Pedia files rather than the BTS Pedia files. There is no harm in renaming the new files so long as you rename all references to it in other related files.

Any non UI python files, such as CvEventManager or CvGameUtils, they are all necessary.

 

[Nightinggale]

Sounds like the real question here is "where is the design documentation?".
AFAIK it has never been released and I have yet to find useful attempts at creating a good overview description. Most of what I know has been learned the hard way

I really wish we had better documentation because quite a number of poor programming are results of people not knowing the code well enough. One really good example is network games. It is fairly easy to do once you know what to do, but with absolutely no documentation, people turn mods into singleplayer only.
http://forums.civfanatics.com/showthread.php?t=525430 <-- my description of how to handle network games. Result of code reading alone as I haven't found anybody else writing about it. (yes it's for colonization, but as far as I can tell, it's 100% identical to BTS network code).

One problem with DllExport is that a number of people are unaware of what it does and add it to their own functions because "it works in vanilla". If you are interested in modifying arguments or similar in a function with DllExport, do check if the function exist in vanilla and if it has DllExport there.

I haven't read the source for Rom:AND and don't know if it is affected by this issue. It's more like a general advice because the problem exist.

• What does the whole life cycle look like? That is, what functions does civ4bts.exe call at various points in time? Also, this includes the main game loop - where is it defined (civ4bts.exe I presume?) and what functions does it invoke in CvGameCoreDLL?

Somewhere (CvGame.cpp if I recall correctly) there is a function, which the exe appears to run in a loop. This works like the game loop and this one triggers doTurn() for everything if the next turn bit is set.

The exe is also responsible for executing incoming network activity. The previously posted link explains all you need to know about this.

Other than that, the most calls during gameplay would be from the graphic engine, which repeatedly calls functions like CvPlot::getFeature() to figure out what to draw.


There are also calls from the DLL, which are handled in the exe. Most noteworthy is the pathfinding algorithm.


For the most part, the exe can be completely ignored.

 

[Vilx-]

I know that there isn't a documentation. Sadly. Though at this point I wonder if there's a point of writing one anymore, since Civ4 is so old already? Then again, it's so different from Civ5, that many people prefer that over the newer game. I wonder, how long before it becomes obsolete? I don't spend any time in these forums, perhaps there are some regulars here who could make a guess?

 

[Vilx-]

Heh, mucking around with the code I found a tool called "dumpbin" which comes with Visual Studio. That revealed that the vanilla BtS CvGameCoreDLL has 2432 exported functions.

Rom:AND has 2728, but I don't know if the extra functions really need to be exported or not.

Anyways, writing a doc for that would be a challenge.

 

[Leoreth]

Somewhere (CvGame.cpp if I recall correctly) there is a function, which the exe appears to run in a loop. This works like the game loop and this one triggers doTurn() for everything if the next turn bit is set.

That's right, the method in question is CvGame::update().

I know that there isn't a documentation. Sadly. Though at this point I wonder if there's a point of writing one anymore, since Civ4 is so old already? Then again, it's so different from Civ5, that many people prefer that over the newer game. I wonder, how long before it becomes obsolete? I don't spend any time in these forums, perhaps there are some regulars here who could make a guess?

Civ4 will remain alive as long as there are mods that are keeping it alive, I'd say. The situation is still looking quite good in my opinion, with a good number of larger mods in active development and even more people working on smaller projects and art and so forth.

It's hard to predict how long that will last, but in my experience a mod with a decent base of players will maintain this base of players as long as it is actively maintained, new features keep getting added and the community has some sort of interaction here on CFC.

 

[Afforess]

A bit late to the conversation, but I documented a few of the C++ SDK files a while back on the Modiki: http://modiki.civfanatics.com/index.php/Civ4_SDK

Links in blue are documented.

 

[Nightinggale]

A bit late to the conversation

A week after last post isn't that late.

I documented a few of the C++ SDK files a while back on the Modiki: http://modiki.civfanatics.com/index.php/Civ4_SDK

The purpose of _precompile.cpp is not entirely correct. The conclusion that we shouldn't code in it is and that is all most people need to know. However for those who wants to know what goes on, here it is:
_precompile.cpp includes the header files. The libraries (mainly boost) is a spiderweb of headers including headers, which mean it takes quite a lot of time to open all included header files. _precompile.cpp does it once and places the result in CvGameCoreDLL.pch, which is then used by the other cpp files. This speeds up compilation on a scale, which is comparable with packing asset files and the core issue is the same.
The code to handle this is in the makefile. However the code is more or less unreadable unless you know what pch files are, which mean the real answer is actually in general online pch file documentation.


Speaking of makefiles, the one linked to on Modiki is Makefile 1.0. I wrote 2.0 last year and later updated it to 2.3. I highly recommend everybody to use the new one as it uses all CPU cores to compile instead of just one and it has 4 types of DLL files (not just release and compile). See signature for details.
I have been wanting to update Modiki for a while, but haven't found anybody with write access until now

 

[Afforess]

The purpose of _precompile.cpp is not entirely correct. The conclusion that we shouldn't code in it is and that is all most people need to know. However for those who wants to know what goes on, here it is:
_precompile.cpp includes the header files. The libraries (mainly boost) is a spiderweb of headers including headers, which mean it takes quite a lot of time to open all included header files. _precompile.cpp does it once and places the result in CvGameCoreDLL.pch, which is then used by the other cpp files. This speeds up compilation on a scale, which is comparable with packing asset files and the core issue is the same.
The code to handle this is in the makefile. However the code is more or less unreadable unless you know what pch files are, which mean the real answer is actually in general online pch file documentation.

Interesting, but the files I documented have my name in the history - that isn't one of them.

I have been wanting to update Modiki for a while, but haven't found anybody with write access until now

You don't have access?

 

[Vilx-]

Oh, wow!. I just received an email about updates in this thread. Which was sent on December the 8th. Better late then never, huh?

Anyways, thanks for the info!

 

[Fullerene]

If I read this right, you could in theory, create new exe/game engine based on CvGameCore.dll? For example to port Civ4 to 2D - graphics engine.

Of course you would lose the multiplayer mode, and support for many mods.

Would it be legal? You could make the new engine to require all the data files + maybe even the original exe to run.

Just wondering if it's technically and practically possible to do? (Read: No need to disassemble any code.) The effort needed naturally might not be worth the gain.

 

[Nightinggale]

If I read this right, you could in theory, create new exe/game engine based on CvGameCore.dll? For example to port Civ4 to 2D - graphics engine.

Yes, but I think you are underestimating what a massive task it is. A graphic engine is quite tricky to set up. Also the exe does lots of other hard to code right stuff, like pathfinding.

If I were to make a new exe, I would start by removing the hard limits, such as max players, 128 row of plots (when zoomed out) as well as compiling everything for 64 bit. I would also upgrade from 10 year old python and boost because newer versions are much better.

However I don't think it's realistic to think that will happen unless the exe source code is released, which is also very unlikely.


If you just want a 2D display, I wonder if the game can be forced to a top down view by just modding the camera controls. Sure it would still be 3D, but you would get the civ1 squares I think you are requesting. In fact I think there is a hotkey to switch to a view like that in vanilla, though I can't remember what it is.


All this is somewhat off topic for this thread though.

 

[Vilx-]

Heh, making a new Civ4 .exe is also something that crossed my mind. If anyone did that, it would be EPIC. But, yeah - extremely much work. Though, of course, much less than Firaxis had to do originally, because you've got it all spec'd out already and half of the game (the one in CvGameCoreDLL & Python) is already there. But still a MASSIVE undertaking.

But, as long as we're fantasizing...

I'm divided myself on the 64-bit thing. On one hand - it's a way forward, you get access to more RAM, you can do bigger maps and stuff. That's nice. On the other hand, all pointers and some other stuff take up twice as much memory, so CPU caches will be exhausted twice as fast - MAJOR slowdown. So it's not that clear cut.

What I'd like to see (apart from being able to use latest libraries, compilers & IDEs) would be a better multithreading support. I think some real performance can be gained in this area. If I understand correctly, the majority of time in turn calculations is spent in pathfinding, so parallelizing that would be great (on a side note - could a better pathfinding algorithm maybe be implemented in CvGameCoreDLL, bypassing the one in the .exe?)

And last but not least - it seems to me that the interface side between the .exe and CvGameCoreDLL is pretty hairy too. If I can trust Dependency Walker, from the 2432 exported functions only about a half is used by the .exe. That's still a lot, and the other half hints at sloppy coding to begin with. Well, maybe they're used somewhere else, or DW didn't detect their use correctly. I'll give it the benefit of doubt.

At any rate, I think that a simpler interface could be possible, which would be better suited for modding. This one seems to have grown rather organically, instead of having been planned. I've run into that problem at my job too.

The old mods could work through an adapter-DLL, which would be a bridge between the new interface and the old one.

OK, so that's my part of daydreaming.

 

[Vilx-]

Would it be legal?

Oh, by the way - yes, it would be totally legal. Unless you actually copied or reverse-engineered some parts of the original .exe, you're in the clear. In fact, this is what the ReactOS project is doing - they're creating Windows XP from scratch, based on publicly available documentation and experimentation results. Since they're not using any of the original code, the copyright is no problem.

 

[Nightinggale]

I'm divided myself on the 64-bit thing. On one hand - it's a way forward, you get access to more RAM, you can do bigger maps and stuff. That's nice. On the other hand, all pointers and some other stuff take up twice as much memory, so CPU caches will be exhausted twice as fast - MAJOR slowdown. So it's not that clear cut.

That's a good point. 64 bit applications tend to be faster due to handling 64 variables in fewer cycles. However we don't use those and the increased pointer size could be a slowdown. Not a big one, but it's there.

Memory latency seems to be a big issue in the civ4 engine and I have optimized to try to get the next variable into the CPU cache. However the biggest problem seems to be recalculating results, which never or rarely changes. The biggest single change I have made is caching the amount of yields a profession use in colonization. This is fixed for each player and can only be changed by getting a founding father, which affects it (read: never for most players). It was pure vanilla code being modded and it reduced the wait for next turn by 17.5%. It would likely be more if I had profiled vanilla, but I did profile RaR.

If you care for performance, look for often used functions, which return the same over and over. Use makefile 2 (my signature) to profile and track down performance hotspots.

What I'd like to see (apart from being able to use latest libraries, compilers & IDEs) would be a better multithreading support. I think some real performance can be gained in this area. If I understand correctly, the majority of time in turn calculations is spent in pathfinding, so parallelizing that would be great (on a side note - could a better pathfinding algorithm maybe be implemented in CvGameCoreDLL, bypassing the one in the .exe?)

The makefile takes care of compilation meaning you can more or less use whatever IDE you want. Libraries are a pain though.
I tried adding multithreading and after much hassle I managed to compile the thread dll for boost 1.32 (it can't be downloaded anymore). After introducing some threading I managed to save 0.4 seconds (on a quad core with hyperthreading) in a game where I had to wait a minute for next turn. Not quite the result I was hoping for. Adding threading to something written for singlethreading execution is actually quite hard.

The pathfinding algorithm is in the exe, but AFAIK it is only called from the DLL meaning you can hook into all those places and write your own function. It's no easy task though, but I would be really interested if somebody manages to write something, which is faster than the current one.

And last but not least - it seems to me that the interface side between the .exe and CvGameCoreDLL is pretty hairy too. If I can trust Dependency Walker, from the 2432 exported functions only about a half is used by the .exe. That's still a lot, and the other half hints at sloppy coding to begin with. Well, maybe they're used somewhere else, or DW didn't detect their use correctly. I'll give it the benefit of doubt.

I have been thinking of writing a script, which goes though the header files and for all DllExport, it looks in the exe for the function name. If it's not there, then the DllExport keyword is removed. We can't change arguments in functions called by the exe, which mean getting rid of as many DllExports as possible increases modder freedom. Such a script would be quite useful because some modders added DllExport "because vanilla use it" without knowing what it actually does. This mean some mods have quite a lot of extra DllExports.

I have looked a little bit already and there are unneeded DllExports in vanilla, which discarded my idea of simply comparing the headers to the vanilla headers. However I'm not too happy with looking into the exe for a number of reasons. It would however be to just figure out which DLL functions it can call, which are written in plain text. I haven't really figured out if that counts as reverse engineering, which would be the key to tell if I will do this.

 

[Vilx-]

Well, as I said, Dependency Walker can provide a list of all exports and a list of exports that it thinks are actually called by the .exe. And you can copy-paste them out of there, so then it's a trivial matter of getting a list of all "most likely unused" exports. After that though I'm not sure what to do. I don't relish the idea of removing over 1000 DllExport's by hand, but otherwise you'd need to parse the C++ code somehow... Perhaps a simple but imperfect parser can be made, and then the remaining entries handled by hand?