Potential Python Optimizations for Civ4 on Intel and AMD CPUs

[PurpleMelbourne]

Hello fellow Civ4 people!

I started playing Civilization on Amiga 500, and haven't stopped... And as you know, that means I've spent way too much time waiting for the computer players to finish their turns! ;-)

So I've wondered about a lot of ideas for optimisations which have been beyond my abilities. But now with the help of AI, it's a lot more possible to learn the required steps to get through these things.

I've been exploring ways to optimize the Python code in Civilization 4, particularly to improve late-game performance when AI turns can take a long time. With the assistance of AI tools, I've come up with some ideas that I wanted to share and discuss with the community:

1. Profiling and identifying bottlenecks: Using Python profiling tools like cProfile or Intel VTune Amplifier to analyze the game's Python scripts and identify performance hotspots. This can help pinpoint specific functions or loops that are taking the most time.
2. Optimizing loops and data structures: Reviewing the code for opportunities to optimize loops, such as using list comprehensions instead of traditional loops, or replacing slow data structures with more efficient ones like dictionaries or sets.
3. JIT compilation with Numba: Experimenting with using Numba, a just-in-time (JIT) compiler for Python, to accelerate math-heavy portions of the code. Numba can compile Python code to native machine instructions, taking advantage of Intel's SIMD instructions for parallel processing.
4. Cython for performance-critical sections: Rewriting computationally intensive parts of the Python code in Cython, which compiles to C and can provide significant speedups. This could be especially useful for sections that interface with the game's C++ codebase.
5. Multithreading with joblib or multiprocessing: Investigating the use of Python's multithreading or multiprocessing libraries to parallelize certain tasks, like AI decision-making or game state updates. This could help leverage the multiple cores in modern Intel and AMD CPUs.
6. Intel Distribution for Python: Trying out the Intel Distribution for Python, which includes optimizations for Intel CPUs and can potentially boost performance out of the box. It includes optimized versions of NumPy, SciPy, and other key libraries.
7. AMD AOMP and ROCm for AMD CPUs: For those with AMD CPUs, exploring the use of AMD's AOMP (AMD Optimizing C/C++ Compiler) and ROCm (Radeon Open Compute) platform, which provide optimizations and libraries tailored for AMD hardware.

I believe that by collaborating and sharing our experiences, we can find ways to significantly enhance Civ4's late-game performance. I'm eager to hear your thoughts on these ideas and any other optimization strategies you've tried or considered. Together, we can create an even better modding experience for the community.

Looking forward to your input and insights!

 

[alberts8]

Civ4 uses Python 2.4.1 which is pretty old and that makes some of the things that you posted unusable.

Most of the game logic is written in C++ anyways. Unless you are using a python heavy mod the reason for late game slow downs is most likely somewhere in the C++ code.

Using a profiler like Intel VTune or the Visual Studio profiling tools can be very useful to optimize the C++ code.

 

[Leoreth]

I agree that it would be useful to identify bottlenecks, but like alberts8 points out I don't think any of your proposed solutions would work or would be less effective than moving the expensive operation to C++ and exposing that to Python again.

I'd be curious to see a review of bottlenecks, but my expectation would be that for the base game you probably won't find much of impact. It mainly uses Python to draw the UI (not always efficiently but it's constrained by player interaction so does not really matter) and some handling of rather rare occurrences like events etc. Even if those were horribly inefficient their rarity doesn't really make them good candidates for optimization. Probably where Python is the most problematic is when the DLL calls a Python function - but the solution there would be to disable the call and move the logic to C++ instead.

As for optimizing the game on the C++ end - there are a lot of mods that already made improvements on that end, especially Advanced Civ and We The People (on the Colonization end, but there is much overlap), so I recommend to start from there.

 

[PurpleMelbourne]

What C++ code is alterable? I'm guessing that the original source is not available. But are there other ways of hacking the existing executable files?

I play "A New Dawn" and if I remember correctly, there were Python optimisations which sped things up quite a lot some time ago. Or is that the point? Its already been made fast and efficient with no more gains left to be had?

I'm by no means any kind of competent programmer. But I'm hoping that with the help of AI that I might no longer be a totally useless spectator.

 

[Nightinggale]

What C++ code is alterable? I'm guessing that the original source is not available. But are there other ways of hacking the existing executable files?

We do have the source code for the dll file. It's called CvGameCoreDLL and is located in "Beyond the Sword" and Colonization (note: no they are obviously not identical). It contains most of the code and let the exe handles mostly engine related tasks. This means all of the AI code is in the dll, hence the freeze between turns is mainly dll code being executed.

Profiling and identifying bottlenecks: Using Python profiling tools like cProfile or Intel VTune Amplifier to analyze the game's Python scripts and identify performance hotspots. This can help pinpoint specific functions or loops that are taking the most time.

This is the core of how to improve performance on anything. It doesn't matter that you make something twice as fast if it takes up 0.0001% of the total time to begin with. It's much better to make something even just 10% faster if it makes up for 50% of the total time spend between turns. This may seem obvious when written like this, but I have seen programmers trying to optimize something based on a hunch rather than actually investigating where the bottlenecks are and then all false assumptions will result no no visible improvements regardless of how well the code is optimized.

As for optimizing the game on the C++ end - there are a lot of mods that already made improvements on that end, especially Advanced Civ and We The People (on the Colonization end, but there is much overlap), so I recommend to start from there.

I agree. People started modding the dll 18 years ago (I think, something like that). Even for skilled programmers, just the amount of time other people have spend improving the code so far is not something you can just catch up on within reasonable time.

As for overlap between BTS and Colonization, Firaxis wrote Colonization essentially as a giant mod for BTS. A whole lot of the code is identical even down to vanilla typos. Also a lot of optimization isn't done in game features as such, more like code specific tasks like data access and cache handling.

 

[PurpleMelbourne]

We do have the source code for the dll file. It's called CvGameCoreDLL and is located in "Beyond the Sword" and Colonization (note: no they are obviously not identical). It contains most of the code and let the exe handles mostly engine related tasks. This means all of the AI code is in the dll, hence the freeze between turns is mainly dll code being executed.

OH! Is it available on GitHub or somewhere? Or released on condition it does not get viewed in public?

Has it ever been optimised for AMD and Intel separately? I notice how Intel have managed to triple the speed of parts of Python with Intel only optimisations. And I think similar things are done with AMD only optimisations for some other programs.

Does the Python based code in the game affect end of turn processing much?

Thanks for the info.
I'm not an experienced programmer in any way. But I love playing Civilization, and perhaps a bit of effort from another person might get lucky with doing something useful

 

[Sephi]

The game comes from a time before github .The source is distributed with the game for the steam version. Just got to the game folder and look for a CvGameCoreDLL folder. BTS expansion has a separate source folder. If you have the cd version of vanilla you might need some patch to get the source. Many mods also distribute their source as part of the mod.

python code is less than 1% of the overall game code and mostly just UI code which tends to be fast. In allmost all cases it does not matter when it comes to performance.

 

[PurpleMelbourne]

The game comes from a time before github .The source is distributed with the game for the steam version. Just got to the game folder and look for a CvGameCoreDLL folder. BTS expansion has a separate source folder. If you have the cd version of vanilla you might need some patch to get the source. Many mods also distribute their source as part of the mod.

Ah that explains a lot. Over the years I've misplaced and found various copies of Civ4. After having to move, I think I've got about 4 or 5 of them!

So the Python code doesn't really matter for the end of turn stuff, just for modding the game features.
And I had no idea about the source code being bundled with the download versions of the game. This is great news.

What compilers are relevant to the performance oriented code?

This is all great news. Although I'm pretty late to the Mod party, so its old news to everyone else.

 

[Nightinggale]

What compilers are relevant to the performance oriented code?

[BTS] - The Easiest Way to Compile a New DLL

In my opinion, it has always been too complicated to set up your system to compile a new DLL for your BtS mod. There are of course great guides on that out there (I want to give a special shout out to asaf's guide that I used to follow), but they still require you to piece together everything...

forums.civfanatics.com

It should be noted that while this guide recommends MSVC 2010, any version of MSVC can be used. However unless you know what you are doing, it's likely best to do precisely what the guide tells you to do. The important part is to use the 2003 compiler as that is the one the exe was compiled with and it's incompatible with other versions of the MSVC compiler.

We The People has a completely different guide if you want to compile that one. I wrote a very custom makefile for my own project, through the guide does use my project independent makefile so either way you can't avoid me

 

[PurpleMelbourne]

We The People has a completely different guide if you want to compile that one. I wrote a very custom makefile for my own project, through the guide does use my project independent makefile so either way you can't avoid me

Isn't that more like YOU can never get away from US asking you questions and needing your help?

 

[alberts8]

On the topic of optimizing the performance of python code used by Civ IV.

It is possible to compile the python24.dll with a newer compiler like MSVC 2022 and after a few changes to the source code using Clang 18 is possible.

I attached a version optimized for the AdlerLake cpu architecture.

 

[PurpleMelbourne]

On the topic of optimizing the performance of python code used by Civ IV.

It is possible to compile the python24.dll with a newer compiler like MSVC 2022 and after a few changes to the source code using Clang 18 is possible.

I attached a version optimized for the AdlerLake cpu architecture.

Well that is really cool.
Back on the Amiga, software would often come with several optimised versions compiled for different CPU's. So we can do that not too. Nice.
Though I don't have a CPU as recent as that. I'm just about to upgrade to LGA2066 gen 7. But I am impressed anyway.