Mini-engine progress

[snowern]

but rather that Linux tend to come with a package manager

Ideally. Ubuntu doesn't have SFML 3 yet. Or updated pugixml. Or python 2. You have to build these things from source, because I like the bleeding edge.

everything has to be compiled with that version. Linux doesn't suffer from this

It is said that the Linux C++ std lib is supposed to be ABI compatible. But we always need to build from source or get specific binaries from the distribution's package manager. It's not like on Windows where you can just download latest SFML from the lib's website. Some Linux binaries of various things exist out there, but I think they statically link everything.

 

[Thrasybulos]

Something I've just noticed in the original game: see here.

Do you have the same behaviour in your game engine or is it fixed? (Looks like some arrays are not cleared when exiting to the main menu).

 

[snowern]

Something I've just noticed in the original game: see here.

Do you have the same behaviour in your game engine or is it fixed? (Looks like some arrays are not cleared when exiting to the main menu).

Yes, it seems these history variables are never reset in CvPlayer. That's the bug-prone style of code we have here. When resetting a game, maybe they should have just done the equivalent of player = {} to totally reset all member variables instead of manually doing it by hand. Or not use globals, but that would be way more difficult.

    CvTurnScoreMap m_mapScoreHistory;
    CvTurnScoreMap m_mapEconomyHistory;
    CvTurnScoreMap m_mapIndustryHistory;
    CvTurnScoreMap m_mapAgricultureHistory;
    CvTurnScoreMap m_mapPowerHistory;
    CvTurnScoreMap m_mapCultureHistory;
    CvTurnScoreMap m_mapEspionageHistory;

 

[<Nexus>]

Hi!
Are there any recent improvements about the engine and/or the graphical adaptation? Even some tiny things?

 

[snowern]

This was actually a side project before moving onto a different project, so this engine/proof-of-concept is unlikely to have any significant updates. But maybe Chris might get map rendering in 3D going one day. Or somebody else. And the other idea was a DLL rewrite for maximum parallelisation.

 

[<Nexus>]

Oh... Okay... I thought this was a start to create a modern exe.

 

[Nightinggale]

Oh... Okay... I thought this was a start to create a modern exe.

We all did, but I had my concerns regarding the lack of focus on reading vanilla media files.

 

[snowern]

Yes, I read sound files for the sound. But something like reading NIFs and textures won't happen. That's for a graphical reimplementation.

 

[<Nexus>]

Yeah, I was following as I could but I still don't understand whether it's impossible to create an app that can read Civ4 graphicsp or possible, just know one knows how to or know one volunteers to do it?
So basically: how much hope is there?

 

[snowern]

That's what Chris might be doing. Or, well, anybody can try. The file formats are known, Cv4MiniEngine shows how to invoke the DLL. All somebody would need is time. And to figure out the graphical parts Cv4MiniEngine doesn't cover. How the terrain is pieced together, how the LSystem works. One could use the debug screen in Civ4 to guess how the graphics work.

 

[black_imperator]

Yeah, I was following as I could but I still don't understand whether it's impossible to create an app that can read Civ4 graphicsp or possible, just know one knows how to or know one volunteers to do it?
So basically: how much hope is there?

my understanding is that it's likely possible, but we're lacking people with time/motivation/skills to do it.

 

[Gameboy2396]

But something like reading NIFs and textures won't happen. That's for a graphical reimplementation.

I can tell having to write a NIF parser from scratch would be hell for any of us, it'd certainly slow development massively and drive us away from this thing. It seems like no one is really pitching in to help with the.nif and.kfm loading right now. It's more like, though, we all want this to eventually become a modern executable, right? Because right now the mini engine just runs in a command prompt window.

I wanted to at least help by getting some expert opinions. I used a deep reasoning AI, Google's Gemini 3.0 Pro (that released this week!) with state of the art reasoning to do some extensive research on exactly what is suitable for us to get the graphics running without rewriting the whole logic core. I got this roadmap generated by them. I have really good faith in what they are saying and I know they're setting much, much higher benchmarks, but of course, double check what they're saying just to make sure. But I think they know what they're doing after they read through this whole thread and looked at the mini engine's GitHub repository. This should get us to work with an actual executable, a modern one, instead of using command prompt.

1. The "Build vs. Buy" Decision
Instead of porting to a massive engine like Unity or Godot (which struggle with legacy formats), the research suggests a "Bring Your Own Engine" approach using lightweight C++ libraries. This preserves the Mini Engine's identity as a highly optimized native host.

2. The Recommended Tech Stack

• Asset Loading: Niflib
  Why: We shouldn't use Assimp. Assimp "flattens" scene graphs, destroying the specific Gamebryo node hierarchy (NiNode vs NiAVObject) that the Civ4 DLL relies on for attachments and effects.
  Benefit: Niflib natively supports NIF version 20.0.0.0 (Civ4's version) and, crucially, can parse .kfm files, which are necessary to decode the unit animation state machines.
  
• Rendering Backend: bgfx
  Why: We don't need a full game engine; we just need a way to submit draw calls. bgfx is cross-platform (DX11/12, Vulkan, Metal, Linux) and is designed exactly for this kind of custom engine work.
  
• Windowing & UI: SDL2 + Dear ImGui
  Why: SDL2 handles the window creation. Dear ImGui should replace the current custom TUI. It allows us to render debug info and the UI layer directly over the 3D viewport.

3. Critical Technical Blockers & Solutions

• The XML Encoding Bug: Vanilla files declare "ISO-8859-1" but use "Windows-1252". Modern parsers (pugixml) choke on this. We need a custom loader step that forces a Windows-1252 -> UTF-8 conversion in memory before parsing.
• Terrain Splatting: Civ4 uses alpha-mask splatting (defined in Civ4ArtDefines_Terrain.xml). We can implement this in bgfx using Texture Arrays and a custom shader that samples the 8 neighbors to determine alpha blends in a single pass.
• Animation: The.kfm file is just a state machine lookup. We need a KFMManager class that maps the DLL's NotifyEntity events to the.kf files referenced in the KFM, then interpolates the bone transforms using Niflib.

Proposed Roadmap

1. Init SDL2 window + bgfx.
2. Port the current TUI to Dear ImGui (get a visual console running).
3. Implement Niflib to load a static mesh (e.g., Warrior.nif) and convert NiTriStrips to bgfx vertex buffers.
4. Implement the KFMManager to handle animation states.

Meanwhile they also produced this full technical report via this Google Doc that you can read here for more information. https://docs.google.com/document/d/1IMIDCEsYdHFO5xEY0wUoB6A_aN3dxsepdH9QHoReyg0/edit?usp=sharing

 

[snowern]

This should get us to work with an actual executable, a modern one, instead of using command prompt.

Hey now, it is an actual executable! And it's modern, using fancy AVX-512! It's just not graphical.

I can tell having to write a NIF parser from scratch would be hell for any of us,

This part should be easy. It's not like Havok HKX files where you have this pointer remapping BS going on, NIFs are just classic scenes serialised in a classic hierarchical way. Once you have your framework setup, the rest of it is just reading stuff as the nifskope XML tells you. And you don't have to implement all NIF classes, just the ones that you need right now.

Or use niflib as Chris might be doing. It loads the NIF, but you also need to use the NIF. Unless niflib comes bundled with all the NIF rendering and animation logic too, you'll need to create a parallel implementation.

Or, you might get real hecking lucky and do what OpenMW did: Convert the NIF nodes to OpenSceneGraph nodes. I found that out recently and was surprised it's possible.

Instead of porting to a massive engine like Unity or Godot (which struggle with legacy formats), the research suggests a "Bring Your Own Engine" approach using lightweight C++ libraries.

I don't know how this AI managed to read my mind, but yes, that's my preferred idea. Just bang out some prototype with a raw graphics API. There's no need for heavy things at this stage. Just get those meshes on screen.

The XML Encoding Bug

Not a problem at all, until you get to the stage where you want to load all mods.

Terrain Splatting: Civ4 uses alpha-mask splatting

Does it? Sounds like it could. And don't forget the terrain meshes. They need to somehow exist and be rendered.

Animation: The.kfm file is just a state machine lookup.

Huh, are they. Guess they look like it. They have a list of animations in them and transitions.

• Port the current TUI to Dear ImGui (get a visual console running).

It's one of the ideas, "just" take Cv4MiniEngine and emulate the console in a 3D graphics API. But it's a bit of a silly idea because you don't want TUIisms in your graphical implementation. You want to make it realtime like Civ4 should be. It's how the DLL should work.

And Civ4 UI code expects a retained-mode UI, so if you use an IMGUI, you'll end up making it not an IMGUI.

Anyway, get a map loaded. Just fire up that DLL in your own application. Than start handling the graphics. Draw plots, draw units. Don't even need a UI, as can be seen when Civ4 bugs out on me and has no UI. Or figure out terrain drawing first and then fire up the DLL. Either order will do.

Meanwhile they also produced this full technical report via this Google Doc that you can read here for more information

By Tiamat, that thing is massive.

 

[retired]

I can tell having to write a NIF parser from scratch would be hell for any of us, it'd certainly slow development massively and drive us away from this thing.

Anyone who ever modded Civ IV should be familiar with https://github.com/niftools/nifskope. It already does the parsing and rendering and it's open source. There's also a Blender plug-in which makes it possible to convert the nif files into lots of other formats.

Because right now the mini engine just runs in a command prompt window.

It's actually fun to play that way. One thing i would want to add is the ability of displaying the UI icons from the gamefont.tga.

Just fire up that DLL in your own application. Than start handling the graphics. Draw plots, draw units.

I remember playing around with that over a decade ago and i remember another similar project around that time. But you actually made it the farthest with your very playable approach

 

[retired]

Does it? Sounds like it could. And don't forget the terrain meshes. They need to somehow exist and be rendered.

The terrain mesh itself is generated using heightmaps at runtime. I don't remember the details but in the terrain art XML it's possible to adjust the blending of terrain between plots. Not sure if that only applies to the textures or the mesh as well.

 

[Zusk]

Hello!
Saw people posting here lamenting the lack of a rendering solution for the mini engine, so I figured I would take my crack at it!
A friend on Discord - kaczkar123 - helped me get over some of the hurdles involved in doing this, but I was able to write up a OpenGL 3.3 renderer for Civilization 4 era .nif files!

A big hope is to eventually make a rendered frontend for the mini-engine that is able to read models and animations.

 

[Gameboy2396]

Hello!
Saw people posting here lamenting the lack of a rendering solution for the mini engine, so I figured I would take my crack at it!
A friend on Discord - kaczkar123 - helped me get over some of the hurdles involved in doing this, but I was able to write up a OpenGL 3.3 renderer for Civilization 4 era .nif files!

A big hope is to eventually make a rendered frontend for the mini-engine that is able to read models and animations.
View attachment 750355

This is brilliant! Please keep us posted!

 

[snowern]

Now it's a race!

NIFs are an important start. Have to be able to be able to get them going. But at some point, you need a map to put them on. After you get a version of the DLL running.

A big hope is to eventually make a rendered frontend for the mini-engine

It would be more like a rewrite using ideas from the mini-engine. Which means somehow piecing together a UI, and running the DLL in realtime. You might need two threads if you want to maintain responsiveness during AI updates.

 

[Zusk]

Now it's a race!

NIFs are an important start. Have to be able to be able to get them going. But at some point, you need a map to put them on. After you get a version of the DLL running.


It would be more like a rewrite using ideas from the mini-engine. Which means somehow piecing together a UI, and running the DLL in realtime. You might need two threads if you want to maintain responsiveness during AI updates.

Multithreading the application will be necessary, yeah! Since Civ 4 is so CPU reliant, you want to minimize how much you are drawing on that!
Not really aiming for this frontend to be like, bitwise-accurate with Civ 4 or anything, so just drawing a heightmap plane to place the scenegraph objects onto to simulate a map should be relatively straightfoward after I implement animation processing.

(This is really out of my coding wheelhouse admittedly, but its a fun challenge for me! )

 

[snowern]

Not really aiming for this frontend to be like, bitwise-accurate with Civ 4 or anything

It probably will be to start with. The original screen scripts construct the UI. Unlike with a TUI, you can implement that UI as the scripts expect. Or are you going to write a whole new UI. A whole new main interface script.

But, you don't really need a UI just to get a map going. I think you can just not call into the screen scripts.

so just drawing a heightmap plane to place the scenegraph objects onto to simulate a map should be relatively straightfoward after I implement animation processing.

Yes, it might be something like that. Well, it can be a real map. You're not putting off DLL integration are you? You can't do that forever!

Animations would be interesting because there's some weird spline data in the NIFs I don't know how to decode. So that's what you'll have fun with.