Rebuilding parts of Civ4! Multithreading & 64bit memory access to increase Civ4's speed in large games?

[Slaughter]

How deep down the rabbit hole should I go for a proof of concept miniengine.

Open X-COM Deep.
An Open-Source Civ IV port would be insanely epic.

 

[snowern]

Loaded up an existing save. It's beautiful.

Started on doing a city screen. Then it would be adding a research bar, doing popups, combat, messages, fixing more unit selection bugs. But something like this is only a proof of concept. Useful for other devs to make something nicer. Could maybe build an AI learning environment out of it or replace the graphics with actual 3D. Oh, and multithreaded map finder. That too.

But I got side tracked. Now I'm doing MILP optimisation stuff for Civ4. You can fomulate city optimisation as an MILP problem, but something like CBC sure does struggle to solve for 200+ turns. Probably because of symmetry (city working plot decision). So I may have to look at Dantzig-Wolfe decomposition, in which something like this is trivial to formulate as. Just split it up into groups of turns and link them together in the linking constraints. Or, make a specially crafted branch-and-bound solver. It would be nice to do B&B without formulating as an MILP, but the search space is just so galactically huge and it's difficult to get a decent upper bound on the objective function (what's the least total output of your city that's guaranteed greater than the true optimum). But something like a hand-written city simulation with symmetry-breaking logic in combination with LP+MILP cuts for the upper bound may be feasible.

 

[1SDANi]

Loaded up an existing save. It's beautiful.
View attachment 694525Started on doing a city screen. Then it would be adding a research bar, doing popups, combat, messages, fixing more unit selection bugs. But something like this is only a proof of concept. Useful for other devs to make something nicer. Could maybe build an AI learning environment out of it or replace the graphics with actual 3D. Oh, and multithreaded map finder. That too.

But I got side tracked. Now I'm doing MILP optimisation stuff for Civ4. You can fomulate city optimisation as an MILP problem, but something like CBC sure does struggle to solve for 200+ turns. Probably because of symmetry (city working plot decision). So I may have to look at Dantzig-Wolfe decomposition, in which something like this is trivial to formulate as. Just split it up into groups of turns and link them together in the linking constraints. Or, make a specially crafted branch-and-bound solver. It would be nice to do B&B without formulating as an MILP, but the search space is just so galactically huge and it's difficult to get a decent upper bound on the objective function (what's the least total output of your city that's guaranteed greater than the true optimum). But something like a hand-written city simulation with symmetry-breaking logic in combination with LP+MILP cuts for the upper bound may be feasible.

Do you have/plan to have a repo for this? This is absolutely insane!

 

[Mr Smith]

An Open-Source Civ IV port would be insanely epic.

Afaik C2C's Flabbert already works on a Godot rewrite.

 

[Slaughter]

Afaik C2C's Flabbert already works on a Godot rewrite.

Flabbert?
Wait, C2C made their own opensource Civilization 4 engine port?

 

[Mr Smith]

Flabbert?

Flabbert.

Wait, C2C made their own opensource Civilization 4 engine port?

Not yet, I didn't say that. Just wanted to point out that another initiative was present.

And if anyone's interested, there is a Civ5 rewrite on Java called Unciv - people already made Civ6 mods for it, might be feasible to recreate Civ4 as well. It's below C2C devs' performance standarts, though.

 

[Nightinggale]

Not yet, I didn't say that. Just wanted to point out that another initiative was present.

I can't find anything about it. Is it progressing in a way where it might actually end up usable at some point or is it some discord statement that it would be nice if he could do it? It would be a game changer for modding if it would actually work. It would also allow native support for mac and linux rather than the wine/proton layer used today, which is sub standard.

And if anyone's interested, there is a Civ5 rewrite on Java called Unciv - people already made Civ6 mods for it, might be feasible to recreate Civ4 as well. It's below C2C devs' performance standarts, though.

Since the civ4 mods are written in C++, it would make the most sense to use an engine, which supports C++, both for technical reasons and the fact that we can assume modders on average knows C++ better than java.

 

[Mr Smith]

I can't find anything about it.

GitHub - flabbert/Stones2Stars.Backend: c2c clone backend

c2c clone backend. Contribute to flabbert/Stones2Stars.Backend development by creating an account on GitHub.

github.com

Is it progressing in a way where it might actually end up usable at some point or is it some discord statement that it would be nice if he could do it?

I still hasn't caught up on Dev-talk channel, but iirc he actually started to work on it, also there was a lot of preparatory discussions on technical design involving other devs.

Since the civ4 mods are written in C++, it would make the most sense to use an engine, which supports C++

He said it's in C#, should be close enough.

Just talk to him directly, I'm merely an informed passerby.

 

[Slaughter]

He said it's in C#, should be close enough.

Wait, in C#?! I know C#, I work with it!

FASCINATING

 

[PPQ_Purple]

He said it's in C#, should be close enough.

No, not really. As someone whose day job is working with both a comparison I'd give is trying to read German vs English. They sound and look similar enough, especially in text. But good luck to you if you only know one.

Not that it matters anyway since the DLL edits done for mods are by their nature specific to the CIV4 code. They can not be ported in any meaningful way to another engine. The best you could do is take them as an inspiration for brand new code.

This all being said, I would say that C# is a lot easier to work with for the casual programmer than C++ because it comes with a lot of ease of use features like built in parsers for XML and not having to worry about memory. So it is one of the easier languages to pick up.

 

[Mr Smith]

The best you could do is take them as an inspiration for brand new code.

Yeah, that's what I meant.

 

[Nightinggale]

Not that it matters anyway since the DLL edits done for mods are by their nature specific to the CIV4 code. They can not be ported in any meaningful way to another engine. The best you could do is take them as an inspiration for brand new code.

This is where I completely disagree. Since a new engine is something, which has to be designed from scratch, it can be designed to act like the exe and have the interface, which connects with existing DLL files. It is entirely possible that the DLL files would then have to be recompiled using a modern compiler, but that would be a bonus. This is also why a new engine should be in C++ as creating a bridge between C# and C++ is possible, but it adds complexity.

This all being said, I would say that C# is a lot easier to work with for the casual programmer than C++ because it comes with a lot of ease of use features like built in parsers for XML and not having to worry about memory. So it is one of the easier languages to pick up.

Then again the class vs struct, which dictate if they are passed by reference or value makes C# super confusing when passing classes around. C++ is much more readable as & and * are used in the calls to indicate this while in C# you have to look up to see if it is declared as a class or struct to know if the code will add a & without you typing it.

 

[PPQ_Purple]

This is where I completely disagree. Since a new engine is something, which has to be designed from scratch, it can be designed to act like the exe and have the interface, which connects with existing DLL files. It is entirely possible that the DLL files would then have to be recompiled using a modern compiler, but that would be a bonus. This is also why a new engine should be in C++ as creating a bridge between C# and C++ is possible, but it adds complexity.

That could work, yes. I will concede that I had not considered this approach. And the more I think about it the more I like the idea.

I was working off the assumption that this would be a complete remake of the whole thing DLL and all. So my comment should be taken in that context. But I like your idea more.

Then again the class vs struct, which dictate if they are passed by reference or value makes C# super confusing when passing classes around. C++ is much more readable as & and * are used in the calls to indicate this while in C# you have to look up to see if it is declared as a class or struct to know if the code will add a & without you typing it.

To be fair, I do not think I have ever seen actual non example code of anyone using structs in C#. Well, excluding the ones defined in the base library by Microsoft.
An that sort of problem is certainly far less taxing than working in pre smart pointer C++ from the early 2000's where you have to manually remember to release memory. Especially for beginners. You have no idea how many times I've seen peoples eyes glaze over when I try to explain the concept of a pointer to them.

This said, I wonder what the Fireaxis legal team has to say about all of this.

 

[Nightinggale]

This said, I wonder what the Firaxis legal team has to say about all of this.

The most important rule of such an engine replacement is that it should rely on the vanilla files and those files should not be included in any downloads. This way there is no way it will hurt sales. It's also not about stealing exe internals as it is matching the DLL interface, something modding already allows us to mod if we like. The super secret parts of the exe we aren't allowed to steal would have to be rewritten anyway to make use of modern libraries rather than some 20 year old closed source ones.

If we do create our own exe, then we can do a bunch of stuff, which isn't currently possible, such as native support on mac and linux even with mods, full unicode support. Modern monitor support (like 4k), 64 bit to avoid out of memory, proper unicode support and a bunch of other such features.

We can also bridge the gap between BTS and colonization by making one shared exe file. This would likely be doable by porting WTP to the more BTS like interface, but it does require adding support for drag and drop widgets. Obviously that would require the exe to work with assets from vanilla of BTS, colo or both.

The biggest problem: who should make such a project and how long will it take?

 

[PPQ_Purple]

I am still dubious about anything like that being done without explicit Fireaxis permission as they both own the rights to Civilization and could have a legitimate case of treating this as a derivative work.

As to who can do it the answer is no single person. This sort of thing is too much for a single person to do unless its a full time job. It would require a team of developers.

 

[<Nexus>]

This said, I wonder what the Fireaxis legal team has to say about all of this.

I believe that VCMI did for Heroes 3 exactly what you are talking about. They have ported the game to Android phones, added a lot of things previously impossible.

 

[Nightinggale]

I believe that VCMI did for Heroes 3 exactly what you are talking about. They have ported the game to Android phones, added a lot of things previously impossible.

It has been done to a bunch of games. Just see Openxcom, OpenTTD and a bunch of other games. In fact a quick search found something like that mentioned on this very forum. Sometimes the copyright holder gives permission, but more often than not they don't and as long as it doesn't include assets from the original game, it's quite limited what they can do about it. Also it would be a PR nightmare to try to claim ownership of an open source project they aren't involved in. Besides companies usually think of it in terms of money and there is no profit or loss prevention in going after such projects. It is however a loss to even get their legal department to figure out if their rights allows them to go after such projects. There was somebody, who asked a company to buy the rights or license to do a remake of a DOS game and it was rejected because it would be too expensive for legal to figure out if they even had the rights to sell. They didn't even know if they owned the rights anymore so the game had effectively become abandonware.

Obviously an exe remake should have the disclaimer Firaxis copyright to ensure that there is no argument that an exe clone/remake will be a slippery slope towards eroding the copyright claims. Also the goal isn't to make a clone, but to allow using the original assets and gameplay optimized for a modern world, like modern compilers, hardware, OSes etc as well as unlocking more modding options meaning the goal isn't a clone.

[CTP] - Call To Power 2 SDL2 Port for Modern Windows, Linux & MacOS

In 2023 Spyroviper went public with his very impressive special release of Call To Power 2 that uses the Apolyton Edition source code to create a special SDL2 port for Modern Windows, Linux & MacOS that unlocks the frame rate, is fully 64bit compatible, and fixes many of the compatibility bugs...

forums.civfanatics.com

 

[PPQ_Purple]

I am familiar with all of those projects. But my understanding was always that they just exist in a sort of legal blind spot rather than being strictly legal.

This said, if we want it to be 100% safe it can't just rely on having the DLL because its code is open and thus anyone can download it and compile it. You need to have reliance on something that is not open like that. So either the original assets like OPENXCOM or the exe it self.

Also note that if you really want to get the maximum benefit of things like 64 bit and multithreading you would also heave to heavily edit the DLL because that is where most of the actual code is.

 

[<Nexus>]

But my understanding was always that they just exist in a sort of legal blind spot rather than being strictly legal.

Perhaps they are legal because no one can/want to prove that they are illegal

 

[Nightinggale]

I am familiar with all of those projects. But my understanding was always that they just exist in a sort of legal blind spot rather than being strictly legal.

It's not that easy to state as to my knowledge no case has gone to court without it being obviously illegal, like including all assets. There has been a court case about copying DVDs on to a HD and it ruled that as long as you own the DVD, making a backup for personal use is legal. If it's copy protected, then you are allowed to break it if that is required to use it on your own device. In other words Hollywood lost completely against the first guy, who wrote a DVD player for linux (he was 14 I think).

This doesn't stop companies for claiming all sorts of unenforceable garbage. Take for instance warranty void stickers. Court will dismiss them, but that will not stop companies from trying to add them. EULAs contains a lot of garbage too, which nobody reads. For instance the game has to be played from CD and playing exclusively from the HD is illegal. That was common back in the day when games came on CDs. However companies themselves don't read the EULAs and hence this text stays there even when the games are on steam.

In short: just because a developer/publisher says something about their game and what can and can't do doesn't mean that is correct.
Also when talking about a game this old, all a company will think about is how to avoid being liable for anything in their answers. If somebody asks Firaxis directly, then there is a good chance they will refuse to answer because nobody knows right away and they don't want to spend time investigating. Sometimes companies replies that they won't answer and sometimes they just ghost such questions (Openxcom states they never got a reply).

This said, if we want it to be 100% safe it can't just rely on having the DLL because its code is open and thus anyone can download it and compile it. You need to have reliance on something that is not open like that. So either the original assets like OPENXCOM or the exe it self.

This is surprisingly easy and will sort of happen automatically. Just keep the idea of reading from the mod and if the file is missing, read from vanilla. Add some text that to make a mod, which uses this engine/exe, said mod is not allowed to contain so many files that vanilla is no longer needed. Maybe add a bit more to it, but it isn't an impossible task to accomplish.

It should be a simple solution like this and not something about looking into windows registers as that is prone to fail in the long run. If there is a custom game engine like this, then it should be open source and run natively on whatever the future brings. Where is the world in 10 years? Will it make sense to recompile everything to ARM? I don't know, but picking a design, which isn't locked to a specific OS or CPU seems to be the right path.