[SDK] Using newer Compilers and C++ Runtime to compile the dll!

[alberts8]

Recently i continued my experiments about using newer C++ toolsets for the dll. Surprisingly using the latest versions of the msvc compiler or clang to compile the dll works after fixing up C++ standard compatibility problems in the code. This also runs pretty stable at runtime unless there are bugs in the code that the older compiler didn't notice.

I took it further and looked into also updating the c++ runtime because this would enable using everything C++23 has to offer. This is more complicated but can be done the only remaining issues i have are caused by the python binding. Firaxis modified the boost python source code so switching the boost python dll isn't possible at the moment.

To make the runtime change work it's necessary to use a modified msvcr71.dll and to use a modified version of the latest microsoft c++ stl. This requires anyone who wants to play a mod using this setup to modify their civilization Installation by copying and replacing dlls.

Is that something that's a showstopper or would the benefits of using a up to date c++ toolset justify having such a requirement?

 

[Nightinggale]

Mixing Multiple Visual Studio Versions in a Program is Evil.

The exe is compiled with MSVC 2003 (VC 7.1) and as such the dll should be too. The only way to avoid this would be to using a modern compiler in separate code and then communicate with between c++23 and c++03 using values only, no pointers or references.

In We The People we are experimenting with using the zig compiler as it has C++ compatibility and it can be set to use VC7.1 memory management. In theory this will work, but since nobody has ever done this before, it is considered highly experimental and time will tell if it actually works. Status right now is that it can actually compile and start the game, but the goal is to run without stability issues, not just to be able to start the game. Feel free to join our discord if you are interested in learning more about this and possibly working together towards a common goal.

To make the runtime change work it's necessary to use a modified msvcr71.dll and to use a modified version of the latest microsoft c++ stl. This requires anyone who wants to play a mod using this setup to modify their civilization Installation by copying and replacing dlls.

Modding vanilla will kill all other mods.

 

[alberts8]

The exe is compiled with MSVC 2003 (VC 7.1) and as such the dll should be too. The only way to avoid this would be to using a modern compiler in separate code and then communicate with between c++23 and c++03 using values only, no pointers or references.

Sadly Firaxis used some c++ classes in the exe<-> dll interface. I use the std::map and std::list templates from vc7.1 and a modified version of the latest microsoft stl to ensure binary compatibility.

In We The People we are experimenting with using the zig compiler as it has C++ compatibility and it can be set to use VC7.1 memory management.

I commented on the pr on github a while ago.

About the memory issue that is where the modified msvcr71.dll comes into play. Newer versions of microsofts runtime use the process heap instead of having a dedicated heap for the runtime. After modification of the msvcr71 source code the exe, the dll and python use the same heap to avoid memory related issues. Using a different xml parser is necessary however because the msxml3 used by the exe can cause memory problems.

Modding vanilla will kill all other mods

Hopefully not because only the memory management part is changing. I got the vanilla game, realism invictus and caveman2cosmos running successfully with this setup so it looks like the msvcr71 change is stable.

 

[alberts8]

Thinking about it this setup is probably not usable for most people who just want to play even if it's stable. Modifying files in the civilization installation directly is off the table.

However being able to use newer c++ functionality is a welcome breath of fresh air so i continue playing around and maybe i find a safer solution that is actually usable.

 

[f1rpo]

If a modern compiler were to motivate modders to new major development efforts, then I don't think it would be unreasonable to ask players to install some experimental libraries. Personally, I'd feel, on the contrary, rather deflated on account of my past efforts to work around the deficiencies of the old compiler. Still wishing you success of course. (I won't be putting my shoulder to the wheel in any case.) I'm amazed that you've managed to overcome most, if not all, the technical difficulties.

 

[alberts8]

If a modern compiler were to motivate modders to new major development efforts, then I don't think it would be unreasonable to ask players to install some experimental libraries.

Ideally the switch to a newer compiler should have been done a decade ago. Back then we had alot more interest in Civ IV from players and modders.

Personally, I'd feel, on the contrary, rather deflated on account of my past efforts to work around the deficiencies of the old compiler.

Advanced mods like yours or caveman2cosmos or many other mods had modders trying to get around these limitations in various ways. This was necessary to enable further development at the time.

I initially wanted to switch caveman2cosmos to a newer c++ toolset but i abandoned this idea. The things people invented to work around the old compiler make using a newer compiler much harder because they introduced additional incompatibilities.

This what caused me to loose interest in further working on this. The original source code doesn't need much work but who wants to play that and the more advanced mods require alot of work to even get them to compile. But just getting them to compile is just the first step in the process and in itself doesn't have a huge advantage without also adopting the code itself to take advantage of newer c++ features.