Mini-engine progress

[Gameboy2396]

Yes, the console IO could be replaced with SFML, and then you'd use a Unicode private block to reference symbols in existing textures. This also means that text width is no longer string size if symbols are double-width.

And you could go further and just have resource bubbles in the world view.

But full 3D would be ideal. Maybe this could go down a really odd path where somebody implements 3D rendering inside the world view only, keeping the rest of the TUI.

Well, @snowern, we should definitely get a plan in place, like, a roadmap of what's happening and when. Like, I'm super excited for this Civ 4 engine, it's really promising from all we've seen for starters. Because personally, Civ 4's probably one of the more iconic Civ games and we'll easily get tons of people coming in and wanting to pitch in. Problem is, we just don't know how to organize it, at the moment.

The places that need to be in 3D are (gotta be authentic):

• Main menu background
• The map:
  • Terrain, features, and the look of the map
  • Cities and buildings
  • Units
  • Resources
• Leaderheads (NOTE: If you think about it, compared to Civ 5 and 6 (and 7, when it comes), making leaderheads for Civ 4 will be a breeze because we don't have to deal with voice and complex body frames, well, unless we decide to go further HD and make more detailed leaderheads)

...That's it. Well, there's nothing else in my head that needs to be 3D, but compared to how much needed to be done in later games, Civ 4 can be pretty darn easy for us considering where we are, technology-wise, today. We just need to find some people.

What about a new font? Not letters but Civ4 symbols? That would be much prettier but also less modder friendly.

Oh, a new font. That's certainly taking a TUI to the extreme. If I were to let symbol substitutions be configurable, then a font replacement could be used along side that configuration, as a mod.

A bit out of scope for now though. You'd need a whole font too, so you'll have to source various character blocks from elsewhere. And build the font and install it.

Would probably be easier to just use a better existing font that has more character blocks. And Linux consoles can do better out of the box. You can just straight-up print pigs in Ubuntu:

So, in terms of fonts, we'll need to find one out there that supports a wider character set, because if you think about it, the game only supports numbers (1 to 9, 0), letters (a-z), and some diacritical letters (all diacritics in French, German, Italian, Spanish, Portuguese and Scandinavian languages. Maybe a few others. But languages like Polish, Hungarian, Vietnamese, Czech, Turkish, those with the Latin alphabet, their specialized letters are NOT compatible with the Civ 4 font in the official build. If we can find a font with a bigger database of letter glyphs across many more languages and make it more accessible to everyone. But of course, we also need to support the Cyrillic, Greek, Arabic and various Asian letter sets too. That's gonna be another rule of thumb for us.

Because honestly, it's better to use a real font that exists than rather try to make our own with custom symbols, because it's harder to work with.

 

[snowern]

The places that need to be in 3D are (gotta be authentic):

Yes, the map. Start simple. Just static 3D. Load the fpks, NIFs and textures, and just draw them. Leaderheads seem simple too. A NIF, a KF, and a KFM. The file formats are all known, afaik, except FPK. But that's not a blocker.

Then you'd get the map animated, implement the LSystem, make the terrain look nice. That motion effect when a unit moves quickly. And because it's realtime, you might also have combat animations.

We just need to find some people.

Anybody out there can get started. GL/D3D, gleam file format info from NifSkope. Get a basic scene manager and resource manager going (or use a graphics engine). Draw those NIFs. Test some animations. Civ4 even has all these debug tools. Test unit animations. LSystem debugging. There's these tree exclusion spheres. You can even save a plot to NIF I think. Open it up in NifSkope. Doesn't seem to save bare terrain though, unfortunately.

Civ 4 can be pretty darn easy

It's like an engine reimplementation starter kit. Somebody just needs to have the time to do it.

So, in terms of fonts, we'll need to find one out there that supports a wider character set

As far as the TUI is concerned, on Windows, it's just going to output UCS2 (2-byte UTF-16), and the rest is up to the console. Whatever font the user configures. Consolas seems to support a bunch of languages. What's lacking is my XML loading. I still have garbage in some city names.

That's the advantage of a TUI. You can leave it up to the console to draw text. Once you leave the console and enter general graphics, it's a can of worms. Most people won't implement full Unicode support. There's just no lib or tutorial for it. Also, if a font renderer was really good, it'd have fallback fonts, so the main font won't have to support every character.

But that's not really a problem. An initial implementation could just use Freetype and any system font that looks nice.

 

[<Nexus>]

Shouldn't we try to get a little more publicity? I mean a news article on the front page of CFC or something like that.

 

[Gameboy2396]

Shouldn't we try to get a little more publicity? I mean a news article on the front page of CFC or something like that.

Right on the money. That should definitely be a huge boost if we can see to gain a news article on CivFanatics. Something people see first if they go to the main menu of the site. I'm just not sure how it works or how you suggest an article to add.

 

[1SDANi]

I'd say wait until the first FOSS release. No reason to increase pressure on snowern when no one else is contributing, not to mention no reason to risk 2K deciding it "competes" with the upcoming Civ 7 release and sending a cease and desist or such.

 

[snowern]

Two advisor screens left. Victory Conditions and Info Screen.

UI layout just about hanging in with all this complexity. Performance is noticeably not-instant in the debug build. Takes a fraction of a second to update the city list in this UI. Probably the python bindings with a bit of textcode conversion overhead.

Better yield display:

 

[snowern]

All misc advisors functional. Victory also "works", but the screens are far from complete.

But FAStar. The FAStar implementation is one of the things reversed from the EXE because no other way could I be sure about exact mechanics.
And that is the case. It appears there's a bug in it. It reuses stale heuristic values (when state reuse is allowed). There is nothing in the code that updates heuristic costs on reused nodes. Surely, this can't be right. A* needs an admissible heuristic.

With pathCost changed to return PATH_MOVEMENT_WEIGHT, you get typical uniform cost A*, so you can see its behaviour more clearly (the vanilla costs are "terrible" in that they result in visiting all the plots around the start plot as if there was no heuristic).
If you path down, then up, you can see that FAStar expanded around the old goal node and finished with a sub-optimal path. The last dump is from generatePath with bReuse = false.

This means an exact description of the Civ4 FAStar algorithm is needed to preserve exact AI behaviour. Can't just take the neighbour visit order and use the algorithm off wikipedia.

I do wonder whether this is why Civ4 sometimes chooses a different path for your unit to move along. But that could also be up to ties.

Alternatively, I could delete the implementation and someone else can add it in. Something similar enough, but no longer preserves exact behaviour (no longer buggy). But I can provide descriptions of inner workings and what the callback constants mean. And it might be possible to get close to exact behaviour by doing test cases that hash the entire FAStar state, which is accessible from the DLL.

 

[Tabris]

I know OpenMW was brought up on the other post but I want to point out that it has premature support for loading Civ4 asset. I believe Capostrophic is the one that has been working on support for assets from other Gamebryo based games.

Spoiler Screenshot :

Spoiler Screenshot :

On their Discord there is also a video from a Fork in which someone loads Gandhi's model too. I'm no programmer but I assume at least some stuff from their project could be used for 3d graphics?

 

[snowern]

There could be some reusable code, but as we have NifSkope, and because Civ4 doesn't seem to do anything fancy, writing a NIF loader wouldn't be all that difficult. But some things could be messy. Maybe particles or animation controllers.

 

[<Nexus>]

Cute spaceship

 

[f1rpo]

I had assumed that the K-Mod implementation of A* is equivalent to the one in the EXE or very nearly so, however, while this old commit indeed proposes that "the results of the new pathfinder now match the old results exactly," – the little bit of test code (linked to above) suggests to me that merely the same destinations are reachable by both pathfinders; doesn't seem that equal path lengths and nodes have been ensured. So I guess K-Mod is no real help. :/

 

[snowern]

void KmodPathFinder::RecalculateHeuristics()

Ah, yes. That's what's missing from Civ4.

Getting the bug to create suboptimal paths with vanilla costs is difficult it seems. But I have found one by random search at iteration 1497.

Take the heavily railroaded save below (let's hope it works with vanilla BTS...).

Mouse over the plot "A" and press G on it. Then cancel by right click. Don't path to other plots by accident. Then, mouse over the plot "B" and press G on it. Count the plots on the path. Should be 12 excluding start and end.

Then, select the settler and reselect the warrior, to reset the path finder (seems to be what it does). Then path to "B" again. Now, the path has 11 plots.

An odd little peculiarity. I guess that, the faster you can move, the more likely that you can get suboptimal paths. Haven't yet found a case on the unroaded save.

 

[f1rpo]

I guess that example is pretty conclusive. (Not that I had really doubted your earlier analysis.) The correct path is a tiny bit better, by a tenth of a movement point. Also highlights a loophole in the DLL – the pathfinder waypoints shown on the map give away routes in the fog of war. (I've spent some time weeding out such info leaks. There's also one that can give away islands that are still entirely unrevealed.)

Having taken another look at the K-Mod code, there seem to be quite a number of implementation decisions that could at least affect tie-breaking behavior, so an exact reimplementation of what the EXE does seems rather hopeless without a disproportionate reverse-engineering effort. Oh, well.

 

[snowern]

so an exact reimplementation of what the EXE does seems rather hopeless without a disproportionate reverse-engineering effort. Oh, well.

Yes, an exact replication of the EXE's algorithm would be needed for exactness, and it's what I have (assuming it's correct). But, if I want to be careful, I'm going to have to delete such code in a release. The release would then be non-functional until somebody fills in the blanks, maybe by clean-room reverse engineering style. In case 2K would ever care about such small amounts of code in an old game.

 

[<Nexus>]

A noob's curiosity:
Why is it necessary to re implement something that is buggy?

 

[mycophobia]

Accuracy, probably. In other words, given the same map with the same settings and the same seed one should get the exact same results between the original game and the remade engine.

 

[snowern]

And examples of suboptimal paths are test cases one can test their implementation against. Bug-for-bug replication. But that's not as important as being careful. I could just rip out the implementation and rewrite it myself, just make something that works, but I worry that somebody might think I was a little too inspired by the original implementation. You can't clean-room by yourself. So I'm giving a heads up of what will be needed.

 

[1SDANi]

It'd also be simple to use #ifdef and a compiler flag to allow users to choose whether they want to compile an exe with or without the bugs

 

[onkelm]

Any news/progress on this hot topic ?

 

[snowern]

Had a brainwave. I can just put reverse engineered/decompiled code in a different repository. Problem solved. So release code should have its pathfinding.

Decided to also begin on audio support. Here's an image of it:

Audio is IMA ADPCM. Decoded then played using SFML. SFML for a console game.

Oh, and here, a replay screen of a loaded replay:

So that's DXT1 encode and decode too.

And I've deciphered the FPK archive format, as I didn't see it anywhere. It's just a simple list of file records with ROT-1 "encryption", with file addresses and sizes. No audio in them from a cursory glance, so I probably won't need them.

Getting close to a playable state. And then you'd just need a main menu to start/load without fiddling with main.cpp.