Mini-engine progress

[snowern]

Ah, somebody else getting the DLL compiled on x64. And on Linux. Haven't done that yet.

Python 3? Any problems with that? The reason I didn't do that was because it might silently change the behaviour of code or have other errors. I would specifically worry about division in map gen code.

I do have some script overrides though. Obviously for the new screens, but some to override BUG too. And a small edit in Civ4lerts to stop an engine assert.

The extra effort that would be really great, is Graphics. That's the big thing that will be lacking. Something that can be started on largely independently. And would be much more valuable if super huge maps were to be viable.

 

[<Nexus>]

The extra effort that would be really great, is Graphics. That's the big thing that will be lacking. Something that can be started on largely independently. And would be much more valuable if super huge maps were to be viable.

Just wondering...
FreeCiv was 2D but has 3D version also. Perhaps some cooperation with them? Maybe even using thei engine as a start?

 

[keldath]

@snowern i'm is delighted with your project!
Last year i'm began to make a similar project, but "from the other end": I rebuild CvGaemeCoreDLL(Realism Invictus mod) using boost 1.74 & python 3.9 for linux arh x64. In future planing create cross-platform build (linux/windows).
Today:
* the project use cmake for build.
* It can be imported to python3.
* I work on Python Assets

Github link: https://github.com/oldbay/civ4_ri_rebuild
CvGaemeCoreDLL: https://github.com/oldbay/civ4_ri_rebuild/tree/master/GameCore/CvGameCoreDLL

I propose to join Yours efforts.
My experience:
* Python (First programming language)
* SQL (Advanced level)
* С++ (Second programming language, base level)

wow you managed to python 3.9 it??

im also can do for this project.
i can do python, SQL, C++ to some degree

 

[snowern]

FreeCiv was 2D but has 3D version also. Perhaps some cooperation with them? Maybe even using thei engine as a start?

Doubt there would be much overlap. Unless they just happen be loading NIFs and interfacing with a similar DLL.

It shouldn't be difficult just to get something going. The most difficult part would probably be interfacing with the DLL correctly. And you ideally keep the UI going while the game is updating, which means multithreading, assuming that's how it works.

Things like just drawing NIFs, animation, the UI, those are the easy parts as you have the most control. Would be a lot of code, but most of it is easy code.

 

[Nightinggale]

I have been busy modding, so I haven't paid enough attention to this as I feel like I should have. I will however say that I'm still very interested and do see myself contributing at some point. I think it's well known on the forum by now that I have years of experience with C++, through I can pretty much work with any programming language (that's not the same as I like all of them).

It shouldn't be difficult just to get something going. The most difficult part would probably be interfacing with the DLL correctly. And you ideally keep the UI going while the game is updating, which means multithreading, assuming that's how it works.

Getting the DLL interface going is just a bunch of work, but it's not technically difficult. We can use dependency walker to get a list of what the exe can call in the DLL and work from there. We have some header files in the DLL, which tells us what we can call in the exe. We have the list of python exposed functions in the exe, through strictly speaking I don't think we have a clear list of what the exe can call in python.

Having said that, I don't think the goal is to make a 100% clone of the exe. Instead we can adjust what it calls/offers to match our needs better. We should likely make new files for the GUI windows, which are inside the exe. They should be in the DLL (or python) so they don't have to be identical in all mods. The game should use utf-8 so we can support all languages. We should redo network to assume modern internet connections (more bandwidth, NAT, so lots of open ports is an issue etc). The list goes on and on regarding what we can/should do to get the most out of this.

We should make one exe, which supports both BTS and Colonization mods. We actually have two ways we can accomplish this and the actual solution will likely be a mix of both. The thing is, the only Colonization mod we should really care about at this point is We The People as it's so advanced compared to other mods that multiple modders have told me that if they start up again, they will restart with WTP as base. This means making a universal exe, which supports both can partly be accomplished by modifying WTP to be more BTS like. Colonization offers drag-and-drop widgets, but it would make sense to offer them to BTS mods too. If it's offered and not used, then nothing will change for BTS mods so it's just adding GUI features.

Things like just drawing NIFs, animation, the UI, those are the easy parts as you have the most control. Would be a lot of code, but most of it is easy code.

That's actually the part where I would have the hardest time. I don't even know where to begin. I have never done anything with 3D graphics and have zero knowledge about that.

 

[<Nexus>]

So the project's stumbling stone is that Civ4 modding requires C++ coding, python coding and 3D modelling but there was no need or way to mod anything in the game (or in any Civ1-6 games) that requires graphical engineering. In our current community there are no suitable people for one important task, right?

 

[Nightinggale]

So the project's stumbling stone is that Civ4 modding requires C++ coding, python coding and 3D modelling but there was no need or way to mod anything in the game (or in any Civ1-6 games) that requires graphical engineering. In our current community there are no suitable people for one important task, right?

That's my main concern and I don't know how to fix this issue. There was a recommendation to use the Godot game engine as it's released under an MIT license, hence 100% free (stays free). It's supposed to be simple to use (relatively speaking anyway), yet it's fast and written in C++ so there wouldn't be a language barrier to get it working (unity is C#, hence much more complex to use with the existing DLL code). It supports 2D and 3D and works on multiple different OSes as well as on both 32 and 64 bit CPUs.

My main issue with it is that I have never tried using it, so my statements are based on what I have read. There was also the issue last year with infighting and a fork named Redot started. Looking at it now, it seems that this fork may have fizzled out and people have gathered in Godot again. Not 100% sure and it is something we would need to make sure of should we use it.

 

[snowern]

Well, nevermind which bit is more difficult. The biggest hurdle is starting. Or maybe not really. Fork the TUI and replace everything with static graphics. Stupid simple basic Civ4 graphics. It just works. A leisurely afternoon for anybody who wants to learn NIF and OpenGL.

I can feel it... I'm going to have to do it one day...

We should redo network to assume modern internet connections (more bandwidth, NAT, so lots of open ports is an issue etc)

Hurdle #2. I'm completely ignoring that, have no idea how it works. If you do implement that, then you have things like net messages invalidating UI.

Having said that, I don't think the goal is to make a 100% clone of the exe. Instead we can adjust what it calls/offers to match our needs better

If you understand the DLL well enough to modify it, then you already know how to use it! One would hope so.

 

[Nightinggale]

Hurdle #2. I'm completely ignoring that, have no idea how it works. If you do implement that, then you have things like net messages invalidating UI.

I'm not worried about network issues. I dealt with network sync issues even prior to Civ4 being released. When I started modding with the civ4 engine, I managed to find an OOS bug after just 2 days just by reading the code. I don't think anybody else noticed it ingame, through it would be hard to pick up on due to triggering OOS while doing CvCity::doTurn().

Summary: I don't fear dealing with network code.

 

[snowern]

There was a recommendation to use the Godot game engine

I've never actually used one of these game IDEs. Godot/Unity/Unreal. So, no idea how useful they would be. But, drawing stuff is only one little thing. You still have to do everything else yourself. And, if you use an engine, you'd need to translate all the NIF stuff to it, and special effects. It might add to the difficulty.

 

[Blake00]

I've never actually used one of these game IDEs. Godot/Unity/Unreal. So, no idea how useful they would be. But, drawing stuff is only one little thing. You still have to do everything else yourself. And, if you use an engine, you'd need to translate all the NIF stuff to it, and special effects. It might add to the difficulty.

Hmm post Old World, @Dale 's been making a new Col like game in Godot in his spare time. Considering that plus his past experience with heavily modding Civ4 and Civ4Col it would be interesting to hear his thoughts/experience on taking on such a challenge, along with the rest of this stuff being discussed.

 

[oldbay]

Python 3? Any problems with that? The reason I didn't do that was because it might silently change the behaviour of code or have other errors. I would specifically worry about division in map gen code.

The logic of my actions is as follows - here one thing pools to the other:
* To make CvGaemeCoreDLL(.so|.dll) x64 ,In actual environment, you need new boost & python 3
* New builded CvGaemeCoreDLL x64 importing to python 3

Porting Python 3 Assets relatively simple, porting CvGaemeCoreDLL is much more complicated (for my).
I have an experience of porting legacy programming code in python (2.7->3.6, old django -> aiohttp, ... etc),
change the behaviour of code and code errors - successfully corrected debaging.

The extra effort that would be really great, is Graphics. That's the big thing that will be lacking. Something that can be started on largely independently. And would be much more valuable if super huge maps were to be viable.

Unfortunately, I have almost no such experience (
There was a shallow study game engine Panda3d (https://www.panda3d.org/ , https://github.com/panda3d/panda3d)
Opensource cross-platform engine on C++ & Python:
* (+) Native Python scripting
* (+) Relatively simple API
* (+) Relatively advanced 3D
* (---) Is no support NIF (since this is Gamebryo format only)

For NIF support i'm found:
* Editor (https://github.com/niftools/nifskope)
* C++ lib (https://github.com/niftools/niflib)
* Python lib (https://github.com/niftools/pyffi)
* Pprogram code OpenMW - They also imitate Gamebryo work (https://gitlab.com/OpenMW/openmw)

wow you managed to python 3.9 it??

im also can do for this project.
i can do python, SQL, C++ to some degree

I can invite you to my project to github. Send a link to your github user!)
First, I'll take a short pause for:
* create github organization to project
* small formating project strucrure
* create project post in this forum (I write very badly and slowly in English - for my this 1-2 hours)) )
* writing minimal documentation (The same problem)

I did not expect, that my little pet project - will cause interest at other people. Therefore, the project now needs to be "combed" )

 

[Nightinggale]

I've never actually used one of these game IDEs. Godot/Unity/Unreal. So, no idea how useful they would be. But, drawing stuff is only one little thing. You still have to do everything else yourself. And, if you use an engine, you'd need to translate all the NIF stuff to it, and special effects. It might add to the difficulty.

The idea is that it is a shortcut to a number of features like hardware acceleration while at the same time provide support for multiple platforms. If we can get it to work, odds are that it will be less buggy than whatever we can make ourselves. I also don't have high hopes we can compete against an engine when it comes to performance.

As for converting the assets, I would say we would probably need to do that anyway, either as a one time thing or at startup. There are open source games of other gamebyro based games so perhaps we can borrow converter code from one of those.

I looked up Godot games and found this trailer. Clearly the engine is able to do what we need. There are other sources, which shows clickable 3D objects, drag and drop widgets and whatever else we can imagine. I saw a let's play of this game and it has mouseover help text on buttons. The question is how much we need to code from scratch or if we can find a shortcut due to knowing civ4 uses Scaleform's GFC library.

EDIT: I just had an idea. Imagine if we can figure out how to replace GameFont with say png images and then let the game insert those as needed in a way, which scales with font size. In fact redoing the UI will likely allow us in general to make everything scale to modern resolutions. That's however an issue for later. We need the basics first.

 

[snowern]

Scaleform

It's Scaleform is it. I had no idea.

UI could be annoying. There's not much out there for making proper UIs without a full blown engine. You have CEGUI. But the last release was in 2016. But there's been commits since. But if you make your own UI, you can reuse Civ4's assets (and scale them up). But that's a lot of code. Can't just print text like you can with a TUI.

 

[Nightinggale]

But if you make your own UI, you can reuse Civ4's assets (and scale them up). But that's a lot of code.

On the other hand, if we write something from scratch, then we will have the code and can adjust it to just about whatever we want like adding new concepts like new spin buttons or whatever. We would also be able to make our own API so perhaps something like my GUI API rewrite idea. If we make something brand new, we should however keep in mind that we should have some bridge system to stay compatible with existing code in mods. We may also need to be a bit creative with the minimap.

I will however say that 2D graphics "just" seems like a bunch of work. It's doable in the sense that we have a long list of functions to implement one by one. I'm more concerned with 3D graphics as I don't even know where to start with those.

 

[Dale]

Hi all. I know I'm on the outside, and linked in by @Blake00 for some comments.

Firstly, it's been near on 15 years since I modded Civ4 or Civ4Col. So a lot of what I knew, has been replaced by what I now know, or lost to memory chips blinking out due to the passage of time and alcohol.

Blake is right that I've been working in Godot for the last year and a half, and yes it is a great engine. It's not as mature as Unity or Unreal, but it is comparable in power. Godot is written in C++ itself, and fully supports C++ code. However, it also supports natively C# and it's python-like script GDScript. GDScript is extremely powerful, and you can do everything you can do in C++/C# in it. I've also been working on prototypes of a couple of games, one is similar to the combat chess from Conquest of the New World, and the other is using Imperialism II as a base (I'm going to make Imp2 in Godot first, then iterate on it to my satisfaction). If you're worried about whether these three game engines will limit or slow you down, don't. I suspect you'll find it easier, cleaner, and definitely less bug-prone, if you use a game engine. That's also my professional opinion (we use Unity for Old World, and a couple of us are messing around with Godot successfully).

Anyways, onto what you're trying to achieve. I note I've really only read the posts on this page 7, plus what Blake mentioned to me. I gather the point of this project is to upgrade the Civ4/Col engine for modern times? Sorry, I don't know how to do that.

My advice would be to try to get everything running on Godot. Godot comes with baked in systems such as networking, UI, game loops, multi-platform support, hardware agnostic, etc. Using an engine such as Godot allows you to get into actual game coding very quickly, and using the existing baked in systems is pretty easy to add your own bolt-ons to make these systems do exactly what you want. I saw you mention using image fonts with re-scaling. Don't even bother, use Godot's UI and you can use fonts, automatic scaling, even translation services. Image fonts just aren't an issue because you would use Godot's UI instead.

The complex bit I believe, would be implementing an asset reader to ensure the use of original asset files (and keep the legal ownership bit alive). Or just make from scratch. Blender files plug natively into Godot, so makes the whole 3d pipeline very simple.

That would be my advice anyways. Happy to answer any questions you might have.

Edit to add: to ensure legal ownership, you probably don't even need to use the original asset files. You could convert them to what you want for Godot, and include in the first-run something where the player has to prove ownership. Maybe have it poll their steam library for game? Or ask to be pointed at the Civ4 game folder and do some checks to ensure a few files exist there? There's ways to do it, which also allow use of your own assets. Ownership checks shouldn't lock you into using their assets in game.

 

[snowern]

I love fast compilation. Is godot fast to compile and link with? Even better if it can be a DLL. What would not be great if it's like LLVM and all you get is one huge static lib on windows. That's why the clang-cl compiler is a 9GB install.

UI style doesn't matter I guess. Could just reuse button icons here and there on top of whatever godot gives you.

Can godot have a window above the map view, and also allow you to click buttons in the background? And also have modal windows? These windows will also need to be stacked as that's how victory works.

*Oh, and dynamic UI layout.

What if I wanted a custom particle system. Might have one of those in the NIFs.

Another thing to be aware of is that super huge maps will need terrain streaming. That would be Fun to implement.

And what about multithreaded resource loading. That would be a plus. Though, with GL, everything will have to go through the main thread.

 

[Dale]

Think of Godot, as a replacement for Gamebryo. But updated for modern usage, practices, and support.

You can do everything you said in Godot, because Godot is like Gamebryo in that it is the layer between the hardware and your code. You can still have a Visual Studio solution with integrated projects. Compiling is the same as using VS to compile other code. If you use Godot's GDScript, it's all done in the Godot editor. There is zero compile time for GDScript as it's runtime interpretation. You can even have all your GDScripts runtime loaded (great for mods). DLL modding would be the same as normal, just runtime load the assemblies through your own code.

One of the projects I made whilst learning Godot was a Minecraft clone. Fully implemented multi-threading, CPU-GPU optimisations, etc etc. Ran great! Could've done more optimisations, but the project ran its purpose and I stopped working on it. But like everything, if you are interacting with anything system layer (screen, audio, etc) you need to ensure safe threading. You can't get around it, because the OS will lock out your app if it tries to manipulate the same thing from different threads. Modern engines do protect from that too, to an extent.

 

[Blake00]

It's funny.. not being a programmer I hadn't heard of Godot until Dale started talking about his projects on the Colonization fans discord a year or so ago but then after he mentioned it I started seeing more and more Godot fan game and fan remake projects pop up around the fan communities so it's clearly becoming a popular choice. I don't think I've seen anyone do a 'Civ like' in it yet but am keeping an eye out for one as I'm sure there's already something out there not discovered yet. Some of the guys in the Master of Magic Discord have been talking about switching from Unity to Godot.. and over in the Starflight communities (a 80s and 90s space exploration and trading game with planet surface landings kinda like Eve but 2D top down) not one but 2 projects are currently in the making using Godot. First project isn't reading original game files as I think he's just extracting all the data out and putting it into his own files but he's imported Starflight 1's entire galaxy map, solar systems, and I think he's currently working on importing the planet atmosphere and surface maps with terrain type & height map data in too. I also had a look for Godot strategy game making videos and sure enough there's all sorts of interesting ones showing various tricks for generating maps in 2D or 3D although of course they're making original games instead of trying to replicate an existing one, but I'll slap them all in a spoiler tag below if they're of any use and inspiration..

Spoiler Click to expand :

Starflight stuff..
https://www.youtube.com/watch?v=HFfa1mZhufI

https://www.youtube.com/watch?v=f8XUVQbQ1dQ

 

[Nightinggale]

You can do everything you said in Godot, because Godot is like Gamebryo in that it is the layer between the hardware and your code.

It's the logical next step from what Microsoft did with Windows 95. Rather than letting games speak to the hardware directly and have support for each type of sound and graphic card, games could tell Windows what to do and then Windows would deal with drivers and making it work even on cards not released when the game was released. The role of a game engine is similar, except it is between the game and the OS. That way one game can run on multiple different systems without having to consider how those systems are different. There are ways to do this without a game engine, but it's more work and usually less optimized.

I tried working on something without an engine, which was supposed to work on multiple systems. This created an extra workload because constantly one OS did something differently than some other OS. Even just getting free HD space to see if there was space for writing a file required like 5 different versions, in some cases even more than one way of doing it for the same OS depending on which version of the OS. Game engines are designed to let programmers make one call and how this is forwarded to the OS is a problem for the game engine, not the programmer. I have a feeling that proposing not using a game engine is mainly (only?) done by people, who haven't tried working cross platform without one.

I see linux gets mentioned more and more in recent years, partly, but not only because of SteamDeck. Civ4 is also available for mac, so it would make sense to use an engine, which supports windows, mac and linux/steamOS as a bare minimum.

But like everything, if you are interacting with anything system layer (screen, audio, etc) you need to ensure safe threading. You can't get around it, because the OS will lock out your app if it tries to manipulate the same thing from different threads. Modern engines do protect from that too, to an extent.

Even without this being mentioned, I would assume this to be the case unless specifically stated otherwise. You can't assume code to be thread safe without verifying that it's thread safe. It would probably make sense to create a class, which is a bridge between our code and Godot. That way we can have class internals to handle calls being thread safe and we can start with something super simple and then optimize it once everything is working without having to look into where what is called in our code. Sort of like an engine layer between our code and Godot.

I saw you mention using image fonts with re-scaling. Don't even bother, use Godot's UI and you can use fonts, automatic scaling, even translation services. Image fonts just aren't an issue because you would use Godot's UI instead.

I was thinking of something like this forum post called RichTextLabel add_image() adding image to end. I didn't actually mean we should manually rescale images, through I suppose it could be read that way. My thinking is that it's opposed to the civ4 engine where GameFont is a fixed pixel size. I suppose that in order to keep the current code, a GameFont xml would be needed so we can still add GameFonts using an int (aka xml index), which will go fetch the image required.

This is however a detail for later. First step really is to get the game to display the 3D map and not consider 2D UI until after the 3D part works. Again 2D UI seems like a bunch of tasks while the 3D requires somebody to figure out how to handle 3D. I really do view 3D as the show stopper while everything is will be something we figure out once we get to it.