[TUTORIAL] Compiling the DLL on Linux

[bluepotato]

Getting the DLL to compile using wine isn't nearly as trivial as it is on Windows. But in the process of getting rid of my previous hacky Makefile setup I mostly figured this out, so I decided to write a guide for it.

Prerequisites
- A PC with a Linux distribution. I've done this on several distributions and haven't found one that this didn't work on.
- Civ 4 installed with wine. This is trivial if you have the GOG version, just winetrick msxml3 and d3dx9. See the winehq page for further information.
- Basic Linux knowledge. You should at least know stuff like how paths work and how to use wine in a terminal emulator.

Step 0: Getting the source code
You should probably have it by now, but if you don't, and want to start from scratch, clone Leoreth's repo at https://github.com/dguenms/beyond-the-sword-sdk.

Step 1: Getting wine 1.7
Why do we need such an ancient version of wine?, you might ask - and rightly so, wine 1.7 was released 5 years ago. Well as it turns out, most newer versions don't work with the VC++ 2003 compiler and will cause an internal compiler error. 1.8 or 1.9 might work as well, I just haven't tried. 2.0 definitely didn't work for me.

Luckily, PlayOnLinux provides an easy way to get wine 1.7. After installing it, go to Tools->Manage Wine Versions. Then select wine 1.7.55 (on the x86 tab), and install it. You won't need PlayOnLinux after this.
You should now be able to locate your wine executable in ~/.PlayOnLinux/wine/linux-x86/1.7.55/bin. I like to move this to a ~/.wine_versions folder so that its path is ~/.wine_versions/linux-x86/1.7.55/bin/wine.

Now create a new wineprefix using

$ export WINEPREFIX=~/compile_linux; ~/.wine_versions/linux-x86/1.7.55/bin/wine wineboot

After this, you should have a 32-bit wineprefix in the ~/compile_linux folder.

Step 2: Getting the Visual C++ 2003 Toolkit and the Platform SDK
For the VC++ 2003 Toolkit I like to use Leoreth's installer. Install this in your regular wineprefix (presumably in .wine, using your system-wide wine command), then move the .wine/drive_c/Program Files/Microsoft Visual C++ Toolkit 2003 folder to ~/compile_linux/drive_c/Program Files.

Delete the WindowsSDK folder, it won't work. Instead, download this installer from Microsoft. Mount and install it in your regular wineprefix (next next finish, though you may want to select the Core SDK only in the custom install option). Now you should have a Program Files/Microsoft Platform SDK folder; move this to ~/compile_linux/drive_c/Program Files as well.

Step 3: Getting fastdep
You will need a fastdep binary compatible with the build script. For this you'll have to download my patched version from here.
Place the resulting folder in your CvGameCoreDLL/bin folder, and optionally compile fastdep using make if you can't/don't want to use the binary I compiled.

Step 4: Compiling the DLL
This should be the easiest step if you've done everything correctly. I've made a shell script which functions like the Makefile (probably not perfect but it works; suggestions are welcome); just save https://raw.githubusercontent.com/bptato/RFC-Greek-World/master/CvGameCoreDLL/compile.sh in your DLL folder.

You will also have to download compile_settings.sh and save it in the same folder: https://raw.githubusercontent.com/bptato/RFC-Greek-World/master/CvGameCoreDLL/compile_settings.sh.

Then make sure the variables in compile_settings.sh match your setup. Specifically, PYTHON and BOOST should point to an existing folder; I placed these in my DLL folder, since this version of wine can behave weird with absolute paths outside of the wineprefix. If you really want to use absolute paths (to e.g. put your Python/Boost folders in your civ folder as recommended in Nightinggale's makefile thread), use something like "Z:$HOME/your/path". Also make sure the FASTDEP variable points to your fastdep binary.

Finally, make compile.sh executable:

$ chmod +x compile.sh

and launch it like this:

$ ./compile.sh [release/debug/final_release]

You can also use [clean] to remove the Release/, Debug/, Final_release/, or if no target was specified, all of these folders.
And that's it!

Further notes
The final_release target (idea from advciv's makefile) performs whole program optimization. This means that linking takes ages, but you supposedly get a somewhat faster DLL.

Also, by default, the script will spawn a child process for checking and compiling every single cpp file to make use of all available processing power. While this speeds up compilation on my (hexa-core) system, it may actually be slower than running a single process on systems with a low amount of cores. Change PARALLEL=true to PARALLEL=false in compile_settings.sh to disable this.

For editing the sources, any text editor works. Those that I've found so far with somewhat usable code completion support are Geany, and vim/neovim with the OmniCppComplete plugin. CodeBlocks's completion also works nicely, but it's a huge pain to set up (an ancient windows version is required, which of course can't use compile.sh). And I haven't the slightest clue how to get the clangd language server to cooperate, unfortunately.

Debugging can be done with winedbg, see this page for further information. Most of the time you'll want to use the info proc (get civ 4's process id), attach (attach to a running instance of civ 4), cont (after attaching) and bt (get backtrace) commands.

 

[bluepotato]

I'm pretty sure that I'm still the only person who is modding the civ4 DLL on Linux, but just in case I'm wrong, a slight update:
I've compiled the fastdep sources for Linux (with some small modifications), so that I could get rid of some hacks I had to use to fit its output into a variable (also, it's faster when running natively). So you'll need this fastdep version for the script to work, which you can get without downloading the entire mod like this:
svn checkout https://github.com/bptato/RFC-Greek-World/trunk/CvGameCoreDLL/bin/fastdep-0.16

 

[Nightinggale]

I'm pretty sure that I'm still the only person who is modding the civ4 DLL on Linux, but just in case I'm wrong

I know you aren't completely alone and it's possible that there might be more in the future. Just keep up the good work and post whatever you figure out in regards to working on linux.

 

[KeeperOT7Keys]

thanks, your scripts were very helpful

 

[Ramkhamhaeng]

Hi, I was able to compile the DLL due your instructions and compile script. Thanks.
For other readers I want comment a few things because the setup is a little bit easier as described.


• For me it wasn't required to take the ancient wine version. I've used wine-8.0 without problems during the installation/building.
• I've installed the SDK, etc. directly in this new created Environment/Wineprefix.

• The default install path for Leoreth's installer is ../drive_c/Program Files (x86)/Civ4SDK/Microsoft Visual C++ Toolkit 2003/
Do not wonder if it's not found in 'Program Files'

• If you get following error…

CvTextScreens.cpp(5) : fatal error C1083: Cannot open include file: 'CvTextMgr.h': No such file 
or directory

…simply remove 'CvTextScreens.cpp'. The default Windows-Makefile blacklists this non-required file during the build, too.

• If you adapt the Makefile in your project you has to transfer the changes into compile.sh, too.

 

[Ramkhamhaeng]

I've created a bash script to automate the setup process a little bit more:

pbmod/setup_dll_build_with_wine.sh at master · civ4-mp/pbmod

Mods for Civilization 4 Pitboss. Contribute to civ4-mp/pbmod development by creating an account on GitHub.

github.com

It's part of my mod, but the script should work as standalone, too. Probabley a few paths has to be altered. (e.g. to your CvGameCoreDLL/compile.sh file.)

Variant A) The following would download the Microsoft SDK and Toolkit and starting the update. You just need to mount the image file and clicking through the installers manually.

./setup_dll_build_with_wine.sh install

Variant B) This variant trying to avoid the manual steps of variant A.

./setup_dll_build_with_wine.sh unattended

Tested with wine 8.0 on Manjaro Linux. Hope the script didn't break sooe because of the used download links.

 

[bluepotato]

Thanks for the updates. Glad to see my post helped

• The default install path for Leoreth's installer is ../drive_c/Program Files (x86)/Civ4SDK/Microsoft Visual C++ Toolkit 2003/

Yeah, that is expected if you install in a 64-bit wineprefix. 32-bit only has Program Files, and I couldn't get wine64 of wine 1.7 working.

…simply remove 'CvTextScreens.cpp'. The default Windows-Makefile blacklists this non-required file during the build, too.

Ah right, I forgot about that, thanks for mentioning it.

• For me it wasn't required to take the ancient wine version. I've used wine-8.0 without problems during the installation/building.

Interesting, it seems to be working for me too. Though when I do

./compile.sh debug
./compile.sh release

with wine 8.2, this results in an error during linking, and only works if I insert a ./compile.sh clean between the two. Using the above two commands, wine 1.7 links without troubles. No clue why this happens, so I think I'll stick with 1.7 for now.

If you adapt the Makefile in your project you has to transfer the changes into compile.sh, too.

Sure, depending on what you change... I think the settings corresponding to the Makefile's settings are all in compile_settings.sh, so changing that should be enough in most cases.

 

[Zusk]

I have to say that you were an absolute lifesaver, thank you! Your instructions worked perfectly!

 

[mycophobia]

Dunno if I'm ever going to do anything with this but this worked like a charm with Debian 12's default wine. Thanks

 

[Charriu]

Recently I started looking into civ4 modding on a linux machine. This setup looks great, but what I'm curious about is, which IDE you are using?

 

[Nightinggale]

Recently I started looking into civ4 modding on a linux machine. This setup looks great, but what I'm curious about is, which IDE you are using?

Since the actual compilation is handled by the makefile, any IDE should work. I personally use MS Visual Studio on windows, but if you feel like setting up a project file from scratch, go with whatever you want. It would be beneficially to use one, which can be set to compile the makefile as well as start the exe with the command line argument to start the mod. It's really helpful that I can just press F5 in MSVC and it will compile and start with the debugger attached.

For editing python, read the WTP python debugging guide (it really is for the civ4 engine, but posted on the WTP mod's wiki). No idea if the actual debugger works in linux, but that page also tells about setting up the JetBrains python IDE in a way, which is compatible with the civ4 setup. There is a linux release of that one.

 

[Imperator Knoedel]

Instead, download this installer from Microsoft.

We're sorry, this download is no longer available.​

:x

Thanks Microsoft, very cool!

 

[bluepotato]

We're sorry, this download is no longer available.​

:x

Thanks Microsoft, very cool!

I've actually spent some time trying to get back into modding lately, and made a greatly updated version of this guide that works without that link (I figured out how to use Leo's version).
The bad news is that it's not fully done yet. The good news is that you can beta test it now:

Spoiler WIP new guide :

Spoiler Background :

"Recently" I've been playing around with the DLL again, using the setup I described in the previous version of this guide. That made me realize that it still had two rather annoying issues:
* compile.sh spends an eternity resolving dependencies every time I start it.
* You can't reliably abort compilation in parallel builds; usually it would just continue compiling even after typing Ctrl+C.

Equipped with slightly more knowledge about Makefiles than previously, I decided to rewrite my sh-based build system simulator as a real Makefile, and also update the guide in the process. Some parts remain the same, but overall it should be much easier to do now.

Prerequisites
- A PC with a Linux distribution (any distro should work). Theoretically it should also work on any other *NIX that can do wine, but note that you will need GNU make.
- Civ 4 installed with wine. This is trivial if you have the GOG version, just winetrick msxml3 and d3dx9. See the winehq page for further information.
- Basic Linux knowledge. You should at least know stuff like how paths work and how to use wine in a terminal emulator.

Step 0: Getting the source code
You should probably have it by now, but if you don't, and want to start from scratch, clone Leoreth's repo at https://github.com/dguenms/beyond-the-sword-sdk.

Step 1: Getting wine
This used to be a tortorous procedure of extracting an ancient wine version where the compiler did not die a horrible death upon invocation. I am happy to report that the Wine devs have fixed this bug, and the latest Wine version works perfectly.
In case you are reading this in the future where Wine broke something again, please try Wine 10.3. If that doesn't work, try Wine 1.7. If that doesn't work, find some task more worthy of your time than fighting with ancient compilers

Step 2: Getting the Visual C++ 2003 Toolkit and the Platform SDK
For the VC++ 2003 Toolkit I like to use Leoreth's installer. Install this in your regular wineprefix, so that you have a folder ~/.wine/drive_c/Program Files (x86)/Civ4SDK/Microsoft Visual C++ Toolkit 2003.

You can delete WindowsSDK from the Civ4SDK folder as it won't work.
Instead, download this installer from Microsoft. Mount the image and install it (next next finish, though you may want to select the Core SDK only in the custom install option). Now you should have a Program Files/Microsoft Platform SDK folder.

Step 3: Getting fastdep
Download this zip file, and extract it to your CvGameCoreDll/bin directory. If you want, you can optionally recompile the binary it comes with.

Step 4: Compiling the DLL
Now download the following zip file:
https://gist.github.com/bptato/0b29.../ed7e255acf881bdfd5af9b023c0fc274e7d44fea.zip
and extract its contents in your project's folder.
(The makefile is called GNUmakefile, so it can coexist with the Windows Makefile.)
Then, run chmod +x gensourcelist to make gensourcelist executable.

Now make sure the variables in the .settings file match your setup. Specifically, PYTHON and BOOST should point to an existing folder; I prefer to place these in my DLL folder, but if you want to use absolute paths (to e.g. put your Python/Boost folders in your civ folder as recommended in Nightinggale's makefile thread), use something like "Z:$HOME/your/path". Also make sure the FASTDEP variable points to your fastdep binary (which it should if you followed the above instructions).

Finally, to compile the DLL, just run make:

$ make

This will create a release DLL.
You can also do release, debug, final_release builds as follows:

$ make TARGET=release # same as without a target specified
$ make TARGET=debug
$ make TARGET=final_release

Note the casing: only debug works, Debug, DEBUG, dEbUg, etc. does
not.

Spoiler Target explanations :

debug includes debugging symbols, it is ideal for debugging with e.g. winedbg.
release does not include debugging symbols, and applies optimizations to the output DLL.
final_release is mostly the same as release, but it will apply Link-Time Optimization to your DLL. This means that you may get a somewhat faster DLL, but also that linking will take ages.

You can also use make clean to remove the specified target's generated object files.

To speed up compilation on a multi-core system, first check the number of cores using nproc. Then use e.g. make -j6, replacing 6 with the actual number of cores your system has.

To silence echoing of Makefile commands, use make -s.
(This will still show which files are being compiled, just not the commands used.)

You can combine the above two as to e.g. build a debug DLL like this:

$ make -sj6 TARGET=debug

That's it!

Further notes
If you add a new .cpp source file to your project, you must either remove the .sources file, or run ./gensourcelist.

For editing the sources, any text editor works. Those that I've found so far with somewhat usable code completion support are Geany, and vim/neovim with the OmniCppComplete plugin.
CodeBlocks's completion also works nicely, but it's a huge pain to set up (an ancient windows version is required, which of course can't use compile.sh). And I haven't the slightest clue how to get the clangd language server to cooperate, unfortunately.

Debugging can be done with winedbg, see this page for further information. Most of the time you'll want to use the info proc (get civ 4's process id), attach (attach to a running instance of civ 4), cont (after attaching) and bt (get backtrace) commands. You can also set breakpoints using e.g. b CvPlayer::init.

Feedback welcome.

 

[Imperator Knoedel]

Step 3: Getting fastdep
TODO recompile and upload it somewhere etc.
(For now, just use the one from the OP.)

svn: E170013: Unable to connect to a repository at URL 'https://github.com/bptato/RFC-Greek-World/trunk/CvGameCoreDLL/bin/fastdep-0.16'
svn: E160013: '/bptato/RFC-Greek-World/trunk/CvGameCoreDLL/bin/fastdep-0.16' path not found

This one you mean?

 

[bluepotato]

I've actually spent some time trying to get back into modding lately, and made a greatly updated version of this guide that works without that link (I figured out how to use Leo's version).
The bad news is that it's not fully done yet. The good news is that you can beta test it now:
[...]

OK looking through this again, I put too much faith into whatever I wrote a month ago being correct, sorry
(fastdep itself is easy enough to fix, but there are other major errors/omissions in the new guide. In particular it seems we still need the installer from MS, no idea why I thought otherwise.)

Plan B until I fix it:
* Instead of SVN checkout, git clone --depth 1 https://github.com/bptato/RFC-Greek-World and copy CvGameCoreDll/bin directly
* For the platform SDK, get http://web.archive.org/web/20220804...com/en-us/download/confirmation.aspx?id=15656, then

$ sudo mkdir /mnt && sudo mount /path/to/5.2.3790.1830.15.PlatformSDK_Svr2003SP1_rtm.img /mnt && cd /mnt
$ wine setup.exe

* For everything else, follow OP

 

[Nightinggale]

Step 3: Getting fastdep
You will need a fastdep binary compatible with the build script.

Just for the record, I got fed up with fastdep issues even on windows so I started writing a perl script to replace it. As I recall, it is actually working, but I have been sidetracked and haven't finished it enough to declare it stable yet. It does what fastdep.exe is doing, but it also supports compiling nested source file directories and it can handle the precompiled header file (.pch file) on a per file basis, so it supports not including CvGameCoreDLL.h in all cpp files. Being perl, it should be possible to run it natively in linux without considering wine. It also removes the need for runtime MSVC 2010 even in windows. Also unlike fastdep.exe, it doesn't have a 1% risk of randomly crashing for no reason. Yes fastdep.exe crashes aren't caused by wine, or at least not all of the crashes.

Do note that some of the changes requires makefile changes as well, so it's not a drop in replacement.

Source code https://github.com/We-the-People-civ4col-mod/Mod/blob/xml_reader/Project Files/bin/precompile.pl

 

[Imperator Knoedel]

Thanks for all the help everyone, I got it working! Nothing can stop me now, muahahahahahaaa!

Incidentally, this very thread is what tipped me over the line to give Ubuntu another shot. I was in the middle of downloading Windows10 to slap on my new computer, mentally listing all the pros and cons of either OS, and the deciding factor ended up being Civ4 modding. So I did some quick googling and a cursory read of this thread before deciding to abort the Windows download and go for Linux instead.
You've provided a valuable service to the Motherland, comrade.

 

[bluepotato]

Just for the record, I got fed up with fastdep issues even on windows so I started writing a perl script to replace it. As I recall, it is actually working, but I have been sidetracked and haven't finished it enough to declare it stable yet. It does what fastdep.exe is doing, but it also supports compiling nested source file directories and it can handle the precompiled header file (.pch file) on a per file basis, so it supports not including CvGameCoreDLL.h in all cpp files. Being perl, it should be possible to run it natively in linux without considering wine. It also removes the need for runtime MSVC 2010 even in windows. Also unlike fastdep.exe, it doesn't have a 1% risk of randomly crashing for no reason. Yes fastdep.exe crashes aren't caused by wine, or at least not all of the crashes.

Do note that some of the changes requires makefile changes as well, so it's not a drop in replacement.

Source code https://github.com/We-the-People-civ4col-mod/Mod/blob/xml_reader/Project Files/bin/precompile.pl

FWIW, the fastdep advertised in the OP is compiled for Linux (mentioned in the first reply).
I had to comment out some code so it runs at all, but that code was already ifdef'd out on WIN32, so it's probably unrelated to the crash you mention (which I haven't seen on Linux, but maybe I'm just lucky )

Thanks for all the help everyone, I got it working! Nothing can stop me now, muahahahahahaaa!

Incidentally, this very thread is what tipped me over the line to give Ubuntu another shot. I was in the middle of downloading Windows10 to slap on my new computer, mentally listing all the pros and cons of either OS, and the deciding factor ended up being Civ4 modding. So I did some quick googling and a cursory read of this thread before deciding to abort the Windows download and go for Linux instead.
You've provided a valuable service to the Motherland, comrade.

Welcome aboard


EDIT: I've now updated the links in the original guide and fixed up the "WIP" new version.
I'll probably make a new thread for the latter soon (simply because the new GNUmakefile is faster than compile.sh), so feedback is still appreciated.
(To note, I've changed the default toolkit path to the one from Leo's installer to simplify things a bit. Otherwise it is largely compatible with the old one.)

 

[Charriu]

I took the time to setup everything with the GNUmakefile solution. Unfortunately I get an error in the # Link objects. step of the makefile:

LINK : fatal error LNK1104: cannot open file 'Project/CvGameCoreDLL.def'
LINK : fatal error LNK1141: failure during build of exports file
make: *** [GNUmakefile:104: release/CvGameCoreDLL.dll] Error 117

My question what is this CvGameCoreDLL.def? I didn't find any file of that kind in my setup?

Thanks a lot for the great explanations so far.

 

[Nightinggale]

My question what is this CvGameCoreDLL.def? I didn't find any file of that kind in my setup?

It renames functions exposed to the exe, so if you have say DllExport UnitTypes CvUnit::getUnitType() and you don't want the exe to read that one, you can use the .def file to redirect it to say CvUnit::EXE_getUnitType(). We had problems with this in WTP too, through for other reasons. Turns out it's more compatible when using pragma comments instead of a .def file, like:

#pragma comment(linker, "/EXPORT:?getNIF@CvArtInfoAsset@@QBEPBDXZ=?getNIF@EXE_CvArtInfoAsset@@QBEPBDXZ")

No idea why you run into issues through. I would assume it should work just fine for you if you use nmake.exe and the 2003 compiler. The fact that you have issues could indicate that you are in fact using some linux tools rather than the intended windows tools through wine, but that's just a guess. I haven't tried compiling on linux myself and as such haven't encountered the issue you have run into, or at least not for the reason you have encountered.

It's also worth adding which mod you are trying to compile. I don't think vanilla comes with a .def file, but it is added in some mods.