What are the major obstacles to rewrite Civ4 for x86-64 compilation?

[ManUnited4Ever]

I know only the bare basics of:

• Windows development (or desktop app development in general)
• C++ and any other language statically compiled to assembly

I've revisited this question sporadically the last couple of years, and if I remember it right, the situation is this:

• Firaxis has released all the source code for Civ4 which they exclusively own themselves
• Source for Gamebryo's engine for Civ4 is not released
• Civ4 uses a Python 2.4 interpreter compiled to x86

If someone here is patient enough to explain, and assuming that my conceptions are correct in the first place, I'm wondering (1) why it's impossible or very hard to replace the current Python interpreter with an x64 version or even a different Python version, and (2) how monumental a task it is to convert the x86 assembly of the Gamebryo-files to working x64 assembly. I think that part of the answer to question (2) is that developers make hardcoded assumptions about available memory space inside the code, but I'm wondering if there's more to it.

Spoiler :

I'm a CS student in my final year. I have done some enterprise Java development, and my feet are beyond wet in PHP and Python 3.x, and I have limited experience with a flurry of less popular languages as well.

I've wished my entire life that I could play Civ4 with all my RAM, and in my most forbidden, guarded dreams, all my cores as well. I fantasize about gigantic maps with swift end-of-turn loads. I'm sick of just growing older and never even attempting to try and fail at this task, and my childhood self will never forgive me until I do. I have a lot of free time next January, and I want to learn. Getting experience with C++ and Windows is never a very bad idea anyway, and it's best to learn through something that interests me.

 

[Nightinggale]

The biggest problem isn't even 32 vs 64 bit. It's the fact that with each generation of Microsoft Visual Studio, the compiler will use a different version of API for memory allocation. If you mix versions, you can quickly get weird bugs. This means we have to use the 2003 compiler. If you compile the dll with even just the 2005 compiler, then you are in for a mess if you allocate memory in the dll and free it in the exe or vice versa. Unsurprisingly modern compilers can optimize better than what they managed to do in 2003.

Source for Gamebryo's engine for Civ4 is not released

This is the main issue. If we could get our hands on this, then we could change the game to use a newer compiler. Sure it would take some work, but without the source code we can't do it without rewriting an engine from scratch.

Getting our own engine would be awesome though. Imagine all the stuff we could do if we could add our own python commands to the GUI stuff... oh and we could make the game truly cross platform. However for the time being I don't consider this to be realistic due to the workload and skills required. Also I haven't even considered the copyright issues it might include, though there would be a chance one could get around that if it requires an install of the original files to start.

Civ4 uses a Python 2.4 interpreter compiled to x86

Python 2.4 is open source. We can recompile using a different compiler if we have to. The main reason why nobody bothered is that it isn't that useful to recompile if we can't also recompile the game engine.

 

[ManUnited4Ever]

The biggest problem isn't even 32 vs 64 bit. It's the fact that with each generation of Microsoft Visual Studio, the compiler will use a different version of API for memory allocation. If you mix versions, you can quickly get weird bugs. This means we have to use the 2003 compiler. If you compile the dll with even just the 2005 compiler, then you are in for a mess if you allocate memory in the dll and free it in the exe or vice versa. Unsurprisingly modern compilers can optimize better than what they managed to do in 2003.


This is the main issue. If we could get our hands on this, then we could change the game to use a newer compiler. Sure it would take some work, but without the source code we can't do it without rewriting an engine from scratch.

Getting our own engine would be awesome though. Imagine all the stuff we could do if we could add our own python commands to the GUI stuff... oh and we could make the game truly cross platform. However for the time being I don't consider this to be realistic due to the workload and skills required. Also I haven't even considered the copyright issues it might include, though there would be a chance one could get around that if it requires an install of the original files to start.


Python 2.4 is open source. We can recompile using a different compiler if we have to. The main reason why nobody bothered is that it isn't that useful to recompile if we can't also recompile the game engine.

Damn. I appreciate the insurmountability of that task now.

Do you think it probable that Gamebryo considers their 2005 source code a secret business asset? Clever design patterns for current products, etc.

 

[Nightinggale]

Do you think it probable that Gamebryo considers their 2005 source code a secret business asset? Clever design patterns for current products, etc.

How should I know? I wouldn't put it past corporate thinking to think making wheels round to be a secret business asset if they view that to be profitable. I have no idea what Gamebryo is thinking internally.

However I'm not sure Gamebryo is the major concern here. Looking at the screenshots of games using Gamebryo, only the civ games use civ like maps and graphics. This gives me the impression that Gamebryo provides an API for drawing 3D. Switching to a different 3D engine should not be able to violate anything regarding Gamebryo. I mean if somebody decides to rewrite and release the game using Unity instead, then it will be clear that it's Unity and not Gamebryo, which deals with the 3D graphics and no company secrets can have been revealed. The Unity license would be good (portable and free until X copies sold, free downloads doesn't count as sales, hence free for freeware). It is however a poor choice because the API is written in C#, not C++. There are other engines and it shouldn't be impossible to find a good C++ based one.

The problem is what is inside the exe. The modded dll file doesn't talk directly to Gamebryo. Say you place a unit on a plot. The exe will ask the dll for which unit to draw on a plot and for the graphics of said unit. This is then told to the Gamebryo in some way. Most likely the Gamebryo engine doesn't know the map at all in the plot system, but rather it's a giant polygon (or rather two triangles), which has the map drawn on them. The input from the exe makes it appear like plots. We don't know how this works, but a whole lot of work goes on in the exe in code where we don't have the source code.

Also other stuff in the exe, which we don't have the source code for: network code, multiple screens like main menu and diplomacy conversation, locate files in vanilla and/or mod, xml passing (partially in dll, partially in exe), savegames (the dll part just read/write in a byte stream and has nothing to do with file location), the entire string/translation system, GameFont (icons used as characters in text). Firaxis didn't give us all the source code when they released the source code for the dll file. Sure it's awesome as it is and we can mod a whole lot of stuff, way more than in most (all?) other games of the era, but it's not limitless.

One thing, which annoys me is the spinbutton (the one with a number and the up/down buttons). It works nice and it can be added to popup windows. However if you design your own screen it will be attached to an x,y location on the screen, meaning you can't attach it to a location in a scrollable list. This is clearly possible due to being possible in the popup windows meaning it's a feature the exe has internally, but isn't providing in the API. This has nothing to do with Gamebryo and is 100% an exe limitation.

 

[ManUnited4Ever]

How should I know? I wouldn't put it past corporate thinking to think making wheels round to be a secret business asset if they view that to be profitable. I have no idea what Gamebryo is thinking internally.

I wouldn't presume you know, but given your obvious skill, I am audacious enough to presume you're an experienced developer I have no clue whether a 15+ year old code base is likely to contain intellectual property that is still relevant today, but thought maybe you had a more qualified guess.

 

[Dancing Hoskuld]

Do you think it probable that Gamebryo considers their 2005 source code a secret business asset? Clever design patterns for current products, etc.

Given that it is used in many games still being bought and played I doubt they have any interest in giving free access to it. If I remember correctly Skyrim even uses it.