Rebuilding parts of Civ4! Multithreading & 64bit memory access to increase Civ4's speed in large games?

[PPQ_Purple]

This is surprisingly easy and will sort of happen automatically.

Easy yes. But something to keep on our radar newer the less.

Hell, we could even just hardcode it as a DRM sort of thing. There is after all only so many versions of the CIV BTS 1.19 EXE files out there. As in, what CD, CIV4 Complete and Steam. And I assume they are identical. Compute the MD5 hash of those and store them in the code. Than on start check for a file called Civ4BeyondSword.exe in what ever location the user specifies and than MD5 it and compare. That's how I'd do it.

 

[Slaughter]

Considering all the other similar projects, I think the legality is quite chill, all things considered. Just gotta require the vanilla files and its all good AFAIK.

By the way, I've heard someone from the Caveman2Cosmos project has been doing some work in an opensource port based on Godot and using C#. I ought to check their discord one of these days, as I work with C# so I might be able to help.

A weird idea: Civilization IV was done using Gamebryo. Is there any similar Opensource port project for other NetImmerse/Gamebryo games? It's an engine which exists since 1997 (crazy how old it is), and has a lot of games made with it. If someone else has tried a similar opensource port project, they may know more about how Gamebryo works under the hood. Might be worth hitting up TES: Oblivion modder community, seeing as that game is contemporary with Civ IV.

Of course, these games are likely pretty different codewise from Civilization IV, but there might be some information there that is pertinent to us. No idea how useful this might be, but putting it out there.

There's a list of games in the Gamebryo page in wikipedia: https://en.wikipedia.org/wiki/Gamebryo

It should be a simple solution like this and not something about looking into windows registers as that is prone to fail in the long run. If there is a custom game engine like this, then it should be open source and run natively on whatever the future brings. Where is the world in 10 years? Will it make sense to recompile everything to ARM? I don't know, but picking a design, which isn't locked to a specific OS or CPU seems to be the right path.

Yes, those are good points. OpenSource and natively run is the way to go.
Open X-COM has a lot of ports as well, in 2016 I was already running it in Android.

 

[Nightinggale]

Civilization IV was done using Gamebryo. Is there any similar Opensource port project for other NetImmerse/Gamebryo games?

How would this benefit us? It could make sense if we had the exe source code and would need a library to link to. However without the source code we would need to write the code ourselves anyway. We might as well pick a modern engine to work with, preferably some open source one, which will likely be updated for years to come.

 

[Slaughter]

How would this benefit us? It could make sense if we had the exe source code and would need a library to link to. However without the source code we would need to write the code ourselves anyway. We might as well pick a modern engine to work with, preferably some open source one, which will likely be updated for years to come.

More in the sense of knowing how gamebryo works, I mean. Engines are intricate.
Yeah the idea is for our port to be a modern, opensource engine which can be upgraded for the future, unlike gamebryo.
I heard there's already an effort to do this with Godot from someone in the C2C team.

 

[PPQ_Purple]

How would this benefit us? It could make sense if we had the exe source code and would need a library to link to. However without the source code we would need to write the code ourselves anyway. We might as well pick a modern engine to work with, preferably some open source one, which will likely be updated for years to come.

What matters is that you support all the old art files. So basically the NIF and KFM file format.

 

[snowern]

A weird idea: Civilization IV was done using Gamebryo. Is there any similar Opensource port project for other NetImmerse/Gamebryo games?

Skyrim. The most popular game of all. Or rather, NifSkope gives you the file format in its XML description. Deserialising the file format would be important for implementing an engine that uses the original 3D models and animations. It should be relatively trivial. Very classicy OO scene management.

 

[Zusk]

Considering all the other similar projects, I think the legality is quite chill, all things considered. Just gotta require the vanilla files and its all good AFAIK.

By the way, I've heard someone from the Caveman2Cosmos project has been doing some work in an opensource port based on Godot and using C#. I ought to check their discord one of these days, as I work with C# so I might be able to help.

A weird idea: Civilization IV was done using Gamebryo. Is there any similar Opensource port project for other NetImmerse/Gamebryo games? It's an engine which exists since 1997 (crazy how old it is), and has a lot of games made with it. If someone else has tried a similar opensource port project, they may know more about how Gamebryo works under the hood. Might be worth hitting up TES: Oblivion modder community, seeing as that game is contemporary with Civ IV.

Of course, these games are likely pretty different codewise from Civilization IV, but there might be some information there that is pertinent to us. No idea how useful this might be, but putting it out there.

There's a list of games in the Gamebryo page in wikipedia: https://en.wikipedia.org/wiki/Gamebryo


Yes, those are good points. OpenSource and natively run is the way to go.
Open X-COM has a lot of ports as well, in 2016 I was already running it in Android.

OpenMW is very far along!

 

[Blake00]

Yeah seeing a lot of the present day Morrowind community moving over to OpenMW as most old mods work with it and it's actually more stable than the original game. Think it's just the fancy graphical script extender mods that don't crossover.

 

[Slaughter]

OpenMW is very far along!

MW uses the older interation, NetImmerse, right? So its older than Civilization IV but the data might still be useful if there's any need to understand the engine.

 

[Gameboy2396]

Just boosting this up. Are you guys legitimately actually making Civ 4 FROM SCRATCH? New engine, open source? To support this multithreading and stuff? I thought nobody was doing even a remotely similar kind of rework! You all have to spread the word about this project, even if it's not official or something. But still... I'm really, really invested.

 

[Nightinggale]

Just boosting this up. Are you guys legitimately actually making Civ 4 FROM SCRATCH? New engine, open source? To support this multithreading and stuff? I thought nobody was doing even a remotely similar kind of rework! You all have to spread the word about this project, even if it's not official or something. But still... I'm really, really invested.

Step one would be to have somebody actually working on it, ideally an entire team. Right now I don't think anybody has even started so.... I will likely support it if it gets started, but starting on my own doesn't seem like something I want to do by myself. One can still hope through.

 

[Gameboy2396]

If you've at least signed up, that's awesome. I don't know a thing you all are doing though, but still, I've always wanted a Civ 4 from the ground up. As long as you can get people together and find those willing to help, you know, have a place where you all can easily communicate so you can commit yourselves in real time. Discord chat room for instant messaging is a great place to go too, for instance.

But definitely, your main goal is going to be finding all the people, including those in this chat (just verify with them) that want to do the project and actually make it official. Spread the word, get fans involved (like me). I can tell @snowern won't say no to that.

Basically, a Civ 4 remaster with much more programming potential and memory, but also important HD graphic and audio updates and balance fixes. Being open source is the best part because then modding is even easier for all of us.

Sorry, I'm just... so thrilled to have stumbled upon this thing.

 

[Blake00]

Well that was my intent in creating this.. ask some dumb questions to some very smart people and get everyone talking cause that's where it all begins haha.

Part of what I do here and also in my own Master of Magic, Colonization and other groups I run is try to bring talented creative people together to form classic game mod/rebuild/reconstruction/clone/whatever you wanna call it programming 'think tanks'. There's some great Civ1, Civ2, Civ3 rebuild projects going on on this site recreating the original games with lots of 'dream additions' and 'limit removals' chucked in. Seems only right that those conversations start happening for Civ4. Yes there's no official project here but people brainstorming, breaking down technical barriers, and throwing quick proof of concept prototypes together is how the magic begins. So thank you to everyone who's contributed to the conversation so far, it's fun to watch.. let's see what happens in the years to come!

EDIT: In fact I might re-title the thread a bit to better suit the direction the convo has gone in over the year.
.

 

[snowern]

The DLL is 99% of the game. All somebody has to do is glue a UI to it and call functions in the right order. Hopefully I can do something close to that. x64, portability, galactically-sized maps, and multi-process map finding are by-products. And I have some brief notes on how to interface with the DLL if ever useful.

Multi-threading is not a sure thing though. That would require modifications to the DLL beyond just getting it to work. And there is also low-hanging fruit to improve performance on a single thread, at least a little bit.

If I can get this thing into some sort of semi-complete state, anybody out there with further quantities of free time can add ~~GRAPHICS~~. Bit of complexity there as you need to load and render NIFs, you have the L-System, the greatwall, bridges, unit movement animation, those nice cultural borders... and to support galactically-sized maps, you will need to dynamically generate plot meshes, or eat a few gigs of VRAM. Some complexity can be reduced though as I think the engine likes to *retain* more state than necessary, like the "founding border" around the settler, and plot visibility.

A graphical reimplementation will also add another complexity. My TUI is strictly event based, but Civ4 is real-time. It's update is supposed to be called every 250ms (and Civ4 does that on a second thread I think). So a proper implementation will have to handle real-time invalidation of data.

 

[Nightinggale]

classic game mod/rebuild/reconstruction/clone/whatever you wanna call it programming 'think tanks'.

I think it would fall under the category of "remake".

The DLL is 99% of the game. All somebody has to do is glue a UI to it and call functions in the right order. Hopefully I can do something close to that. x64, portability, galactically-sized maps, and multi-process map finding are by-products. And I have some brief notes on how to interface with the DLL if ever useful.

Just use something like Dependency walker to list what the exe can call in the DLL. Likewise the DLL can call the exe, but here we have both dependency walker and some header files to figure out what can be called.

For the most part, DLL calls exe for UI, audio, network and graphics. If we have our own working exe, then we could add more features like better music control.

Multi-threading is not a sure thing though. That would require modifications to the DLL beyond just getting it to work. And there is also low-hanging fruit to improve performance on a single thread, at least a little bit.

First goal of optimization seems simple enough: optimize for a CPU, which is less than 20 years ago using a compiler, which has optimization theory/code, which is less than 20 years old. I did some tests around 2005 (not civ4 related) where I lowered CPU load by 7% by just enabling all CPU registers and unlocking CPU specific instructions. No code change and no compile change other than specifying target CPU. Keep in mind this was the difference between two generations of CPUs, not two decades of progress and it was using the same compiler.

If I can get this thing into some sort of semi-complete state, anybody out there with further quantities of free time can add ~~GRAPHICS~~. Bit of complexity there as you need to load and render NIFs, you have the L-System, the greatwall, bridges, unit movement animation, those nice cultural borders... and to support galactically-sized maps, you will need to dynamically generate plot meshes, or eat a few gigs of VRAM. Some complexity can be reduced though as I think the engine likes to *retain* more state than necessary, like the "founding border" around the settler, and plot visibility.

A graphical reimplementation will also add another complexity. My TUI is strictly event based, but Civ4 is real-time. It's update is supposed to be called every 250ms (and Civ4 does that on a second thread I think). So a proper implementation will have to handle real-time invalidation of data.

Graphics being the biggest issue, at least in my point of view. I will be honest and admit that I have never coded anything regarding 3D graphics and it is indeed my weakest point regarding computer science. Everything else seems doable, possibly with a bit of studying to find engine specifics.

This means from my point of view, what is holding this back is somebody showing up saying "I know how to do the graphics". I might just start working if there is a valid plan for graphics including the needed manpower/skills.

One of the reasons I really want this to actually end up working out is not just due to modding limitation, but also hardware/software restrains. If we compile everything and can freely pick compiler and engine, we can go cross platform. Linux would be an obvious choice, but also mac and SteamOS. Different CPUs like ARM (including Snapdragon and Mac Metal). Since I last posted here (well before today's posts), somebody posted about getting WTP working on Steam Deck so there is apparently a demand for that. There are also multiple posts about getting it working on Steam Proton, which apparently isn't an out of the box experience. Getting it to just work without having to adjust settings would be nice. If we support just about all platforms, the interest in the modding community could increase in general as a whole, including possibly more modders and even new mods.

 

[Gameboy2396]

Sure, I’ve dabbled a good amount in XML and I’ve taken a look at python every now and then, but my coding and “think tank” skills are not really present. I do though love stuff in the more creative departments like text strings (huge fan of diplomacy statements), audio, context and events, you know, I relish realism and unpredictable amounts of content.

The thing I’ve noticed though is that Civ 4’s forums, while they may not be the most active compared to later instalments, they do have the most posts in its lifespan. And you know, I’ve always known people on Twitter (never calling it X) who stick with Civ 4 and said that this installment was pretty much the franchise’s best.

But personally I think it’ll be even easier to attract coders and “think tanks” to this project compared to Civ 1, 2 and 3’s rebuilds, honestly. A more active base of players is a great start point. We’ve still got Caveman to Cosmos still going strong, Rhye and Fall: Dawn of Civilization back in development, you know, the community IS still passionate.

By the way, I get so sick of the game crashing and ruining my save game progress because it “can’t locate any more RAM for video specifications”

Well that was my intent in creating this.. ask some dumb questions to some very smart people and get everyone talking cause that's where it all begins haha.

Part of what I do here and also in my own Master of Magic, Colonization and other groups I run is try to bring talented creative people together to form classic game mod/rebuild/reconstruction/clone/whatever you wanna call it programming 'think tanks'. There's some great Civ1, Civ2, Civ3 rebuild projects going on on this site recreating the original games with lots of 'dream additions' and 'limit removals' chucked in. Seems only right that those conversations start happening for Civ4. Yes there's no official project here but people brainstorming, breaking down technical barriers, and throwing quick proof of concept prototypes together is how the magic begins. So thank you to everyone who's contributed to the conversation so far, it's fun to watch.. let's see what happens in the years to come!

EDIT: In fact I might re-title the thread a bit to better suit the direction the convo has gone in over the year.
.

 

[snowern]

Just use something like Dependency walker to list what the exe can call in the DLL. Likewise the DLL can call the exe, but here we have both dependency walker and some header files to figure out what can be called.

For the most part, DLL calls exe for UI, audio, network and graphics. If we have our own working exe, then we could add more features like better music control.

Logging is what you need. You already have an idea of what the exe can call by looking at the DllExport annotations in the DLL's code. But you just can't get the flow of things by looking at a list of functions, and you also have virtual functions. It's a complex spaghetti of calling into the DLL and the DLL calling into the engine through interfaces and the engine calling back into the DLL. It's very much like they took a bowl of noodles and cut it down the middle. And you've got python on top.

But logging. After I've annotated the DLL with numerous logging calls, I can just activate logging by pressing a button on my gamepad to find out how something is done. Clean it up by removing duplicate lines.

 

[<Nexus>]

Oh, I love this sexy talk

One of the reasons I really want this to actually end up working out is not just due to modding limitation, but also hardware/software restrains. If we compile everything and can freely pick compiler and engine, we can go cross platform. Linux would be an obvious choice, but also mac and SteamOS.

Yeah, Civ4 is probably one of the greatest thing holding me back from switching (or at least trying) to Linux. (I had to abandon Win7 recently and I still hate Win10)

With no programing skills I can only offer to help by beta testing (which I know is still far away) and I'm sure there are many others who would.

 

[Nightinggale]

You already have an idea of what the exe can call by looking at the DllExport annotations in the DLL's code.

I used dependency walker to delete all the DllExports, which the dll provides, but the exe will never call. The vanilla BTS DLL provides 2268 functions while the exe "only" looks for 1207.

But you just can't get the flow of things by looking at a list of functions, and you also have virtual functions. It's a complex spaghetti of calling into the DLL and the DLL calling into the engine through interfaces and the engine calling back into the DLL. It's very much like they took a bowl of noodles and cut it down the middle. And you've got python on top.

True and in particular the virtual functions are undocumented and largely unknown. For the same reason it would be nice to get rid of all virtual functions in the interface. The thing about making our own exe is that the goal isn't actually to make a 100% clone, it is to make something, which works. This means the goal is to make an engine, which provides the functions the DLL calls while making the rest work based on the DllExports provided by the dll. We can even require code changes in the DLL if needed, but then converting mods to the new system will require more work.

Python doesn't seem like a major issue. The functions in question will overall not really be problematic to implement. It's just a bunch of work rather than something difficult to do. Sure there is the whole UI setup, but that is strictly speaking under the UI part rather than the exe-python interface itself.

We can't avoid having to update the DLL code through. Vanilla contains some code with undefined behavior, which the old compiler accepts, but modern compilers will reject due to the high risk of being bugs (which it is in some cases). Also my plan is to make it work with vanilla BTS and next step is to make it work with WTP. Since moving forward all active Colonization mods will likely be WTP or based on it, updating the mod to work with the same "exe" as BTS mods will make future maintenance easier and kill the "us and them" setup we have right now.

However it's simply not possible to get WTP to work with the BTS exe because that exe has no support for drag-n-drop widgets. That will have to be added together with the appropriate DLL function the exe calls when activated. This however wouldn't be bad to add to BTS mods. We could make it a standalone class so it's just a cpp and header added if a mod doesn't use it. Adding an optional python call to it to allow it to work with python only code wouldn't be bad either. We should probably also add some "get enum value" interface and that way avoid hardcoding enums into the exe entirely. Right now some of them have secret meanings, which are undocumented.

Despite being a bunch of work, it all seems highly doable if we can find somebody, who knows how to handle the graphics.

 

[Sephi]

for graphics you could ask tomekum. He did some impressive stuff for his remaster mod like writing a new FOW shader. But imho graphics is too much for one person alone.

Civilization IV Remaster

I just wanted to share with you guys the mod I am working on. It's called Civilization IV Remaster. It is mostly focused on improving the visuals of the game and keeping the gameplay vanilla. The mayor features are: - changed the fog of war...

forums.civfanatics.com

Or just use a modern engine like unity or unreal and write converters for nif files and python scripts. Daggerfall has a remake in unity but I don't know if the original uses the gamebyro engine.