Mini-engine progress

[snowern]

Continuing on from https://forums.civfanatics.com/thre...ed-in-large-games.688441/page-3#post-16622683

Now that's a pretty picture. It's actually a decent replication.

The TUI is now my own lib instead of FTXUI. So I can have proper tooltips, scrollbars, and modifier key support.

And, this UI is written as a CvMainInterface.py file. So, yes, the TUI lib is exposed to python and the original UI will be overridden as normal python code.

I would like to give the world view a polish too. It was written using 16 colours, and I've found out that the Win32 console supports 256 colours. I might also wonder about making the plots bigger with a smaller font size, or maybe have an adjustable plot size. Maybe have the city billboards always shown. It's difficult to stuff things into a 9x5 box.

I'll also need to get popups working again too. Can't found a city currently.

As a Fun test, how big can I make the maps? Let's just tweak Pangaea a little...

size = grid_sizes[eWorldSize]
scale = 10
return (size[0] * scale, size[1] * scale)

Start a WORLDSIZE_HUGE game and two minutes later, you get this:

Huh. We're all on top of each other. How did that happen. Would be terrible if you had a two-minute debug-edit cycle to fix a bug...

Something wrong with map gen maybe?

Looks fine. Nice island region. About the size of a huge map all on its own.

It's starting locations. And it's CvMapGeneratorUtil.py that decides starting locations. There's not much going on in it. And iRange = player.startingPlotRange() stands out clearly. Range you say?

And it's a calculated range. We have this line:

iRange *= (GC.getMapINLINE().getLandPlots() / (GC.getWorldInfo(GC.getMapINLINE().getWorldSize()).getTargetNumCities() * GC.getGameINLINE().countCivPlayersAlive()));

At 665k plots, we don't have integer overflow yet. But the resulting number is still huge. This term is quadratic and grows faster than the width of the map (maintaining aspect ratio). So to support galactically-sized maps (using the same settings), a mechanic-changing modification is needed for at least these map sizes. Say, capping getLandPlots here to the number of plots in a normal huge map.

Turn times are about a second or two.

Memory usage? Just under a gig when loading the game. That's pretty tight. The A* node arrays alone are 633MB, and CvPlot[] is 233MB. CvPlot's nested allocations are probably about 100MB.

This map is saved successfully, but loading in Civ4 crashes at 2GB memory usage. As you would expect. Even with a 4GB patch, assuming it works. Does Civ4 create a complete graphical mesh for the whole map?

 

[snowern]

Does the AI ever ask for peace in BTS?

https://forums.civfanatics.com/threads/is-there-anyone-who-played-before-bts-released.651788/

It seems there is a bug in the exe. The CvDiplomacyScreen is reset, and then if the war-time deal is non-empty, it is checked before populating the trade lists. BTS attempts to find a peace item in the trade lists, but because the trade lists are empty, it doesn't and aborts diplo.

 

[f1rpo]

Thank you for keeping us updated. The UI has its charm aesthetically.

The CvDiplomacyScreen is reset, and then if the war-time deal is non-empty, it is checked before populating the trade lists. BTS attempts to find a peace item in the trade lists, but because the trade lists are empty, it doesn't and aborts diplo.

That sounds plausible, I guess. And you've inferred this ... just by taking a close look at the sequence of DLL calls in the EXE?

The TUI is now my own lib instead of FTXUI. So I can have proper tooltips, scrollbars, and modifier key support.

And, this UI is written as a CvMainInterface.py file. So, yes, the TUI lib is exposed to python and the original UI will be overridden as normal python code.

Just to make sure I follow you: So the text-based UI is something you wrote largely from scratch (or maybe as a FTXUI fork) in C++ to replace Firaxis's UI functions in the EXE that ultimately make use of Gamebryo? And your CvMainInterface.py is, I assume, structured quite differently and, thus far, much smaller than the humongous BtS version of that script? This would seem like a good opportunity to separate the city screen from the (proper) main screen. Of course you didn't say that there aren't already component modules that CvMainInterface.py delegates its work to ... Well, probably an unnecessary worry that you would recreate Firaxis's unwieldy software design without a good reason.

Just to put in perspective the map size in your experiment: The dimensions seem to increase by about 25% from each world size type to the next, a little less on average. E.g. Huge to Large has a 32/26 width ratio and 20/16 height in Civ4WorldInfo.xml (each to be multiplied by a "cell" size of 4 to arrive at the size in terms of plots). So your tenfold increase should correspond to (log 10/ log 1.25) hypothetical world size increases, i.e. ca. 10.3, closer to 11 if we consider that 25% overstates the increment a bit. Let's see, Rise of Mankind has 2 extra sizes (that do follow the 25% pattern), named "Giant" and "Gigantic". Well, Pangaea with its smaller custom dimensions is really only a Large map (26x16 cells) at WORLDSIZE_HUGE. So I guess we're 8 sizes above Gigantic. And, of course, in terms of the plot count it's simply 100 Large maps taken together – or 65 Huge maps. Encouraging that the map generation succeeded with just one easily amended error. Actually, you even managed to pass the turn, even into the late game? I would've expected lots of overflow issues in AI code and elsewhere. Possible that scaling up the various WorldInfo data in accordance with the dimensions would cause more trouble. Oh, it's a 64-bit binary; that should help a great deal. Compiled with Clang or ...? At any rate, it seems that at least maps larger than anyone could ever conquer will work without major adjustments to the DLL.

 

[snowern]

In CvPlayerAI::AI_doPeace, where the AI offers TRADE_SURRENDER, diplo is started. It logs CvDiploParameters, and a canTradeItem for the capitulation. It then loops through all the trade item types, looking for one with isEndWar.

50b73f Getter CvGlobals::getDiplomacyScreen
553fe3 Getter CvDiploParameters::getData
553fec Getter CvDiploParameters::getAIContact
553ff5 Getter CvDiploParameters::getRenegotiate
553ffe Getter CvDiploParameters::getTheirOfferList
554007 Getter CvDiploParameters::getOurOfferList
554015 Getter CvDiploParameters::getDiploComment
55401e Getter CvDiploParameters::getWhoTalkingTo
...
551df7 ComplexGetter CvPlayer::canTradeItem
...
553981 Getter CvTeam::isAtWar
...
585901 Getter CvTeam::isAtWar
58591a Getter CvGlobals::getNumTradeableItems
585927 Getter CvDeal::isEndWar
585950 Getter CvGlobals::getNumTradeableItems
585927 Getter CvDeal::isEndWar
585950 Getter CvGlobals::getNumTradeableItems
...
585950 Getter CvGlobals::getNumTradeableItems
50b755 Mutating CvDiploParameters::setPendingDelete

Maybe it could be checking if the offer contains an isEndWar trade. But then the loop would stop early and do something else.

Or it could be trying to build an end war trade, or check if it's possible. But it can't do that because it hasn't yet called CvPlayer::buildTradeTable/updateTradeList,which is called for something like AI-initiated open borders.

This check is skipped for empty offers. If you remove TRADE_SURRENDER, you get an AI-initiated cease fire. Does not display the trade table though, which is odd.

Whatever this check is, it's the final check here. Nothing else happens. At least, having these bugs confirmed by others is useful.


CvMainInterface.py is currently at about 1400 lines. Compared to 3000-5500 lines. The new code tends to be more compact due to trivial simplifications here and there. And there's a few more things needed. The top right screen buttons, the scoreboard, research bar. Moving the city screen out could be done, but it's manageable anyway, as my editor has code folding.

Getting the UI in python was useful enough, as I can reload python without restarting the game. I didn't want to write the UI in C++ and recompile for every little adjustment.

Yes, the map was only 1040x640. And I only tested the first turns.

The thing that's currently blocking progress (on a normal map) is handling first contact diplo. Which should be easy enough. CvDiploParameters gives you all the info and you just have to check it against the trade lists and put it in a UI. The diplomacy script tells you what diplo options to display.

The second blocker is combat. Unimplemented functions.

If you want True Huge x10^2, I've just tested Archipelago and Pangaea, 1280x800. That's a little over a million plots. 1.4GB memory usage. I wonder if I'll ever finish one of these maps. Forget managing thousands of cities, abuse vassals. And I'm sure late game turn times would be horrendous. Get your X3D chips out. There's quadratic time complexity all over the place. Maintenance calculation. Trade routes. Religion spread.

For early turns, it seems the biggest bottleneck is void CvPlayer::doWarnings(),where it loops over each plot for each player (including AI) to find enemy units... "and you just sit there and you wonder"

Using VS. Codebase is very Windows centric right now (the DLL, my TUI, debug output), but porting to Linux should be relatively trivial compared to everything else. There's a whole nice things wish list. Maybe I might add audio, depending on complexity. A middle ground between silent TUI and full 3D graphics.

 

[Tecumseh1]

This is crazy hard work, kudos! I was wondering if you think it's possible to get some help from firaxis on this? Maybe a partnership. I would love a remake/rebundle of civ4 with better performance and maybe even other small improvements, better graphics etc.

 

[snowern]

The Black Mesa treatment. You only need a whole team to create something really good. Well, I'm sure they know about this forum. Maybe they might even still visit the Civ4 corner.

Behold, the diplo screen.

(yes, looks like we have a text replacement mix up)

 

[f1rpo]

Ah, right, you need to redo the Diplo screen (and something to replace the Custom Game and other game setup screens, for that matter) and that's why you've had to scrutinize the call sequences involving that screen particularly. Still wrapping my head around the task you're dealing with. And of course you couldn't have played far without the Diplo screen. The pathfinder too. Modders have implemented pretty efficient replacements for that in the DLL; though, at least in the case of K-Mod, the waypoints shown by the UI are still handled by the pathfinder in the EXE. Plus plot group updates and some other minor uses of A*.

Thanks for the log output. In my experiments, the problem didn't seem to occur in AI_doPeace/ beginDiplomacy. The offer did end up in the recipient's m_listDiplomacy and was only discarded as the Diplo screen was supposed to display it, i.e. at the start of the human turn – and feigning peace at that time was an effective workaround. At least for a non-capitulation peace offer, I'm not getting setPendingDelete during the AI turn, i.e. when the offer is initiated. The setPendingDelete call does happen right before displaying the offer. Seems to happen for most/ all diplo popups at that point. Not sure if it's necessarily indicative of a problem when setPendingDelete gets called on an AI turn; doesn't seem normal at least. Possible that there's one problem in the EXE that can happen on AI turns and another on human turns.

Avoidable loops over all plots: I'm sure I've added a couple of those myself while modding. How to get all tiles, units or cities that– ... duh, just check the whole map, it'll be no slower than going through all units of all players and probably faster than checking a large radius around a center plot. Unless the map is quasi infinite, that is.

The text interface really is impressively readable (in addition to not looking unappealing), come to think of it.

This is crazy hard work, kudos!

Yes, some amazing technical feats in modding in the last couple of years. Some features of the "Civ 4 Remaster" spring to mind, efforts to get a 64-bit DLL to work with the 32-bit EXE, the savegame editor in the works. All despite few major additions to any widely used mods (Dawn of Civilization being the notable counterexample I can think of).

 

[Gameboy2396]

This is crazy hard work, kudos! I was wondering if you think it's possible to get some help from firaxis on this? Maybe a partnership. I would love a remake/rebundle of civ4 with better performance and maybe even other small improvements, better graphics etc.

That's the plan. Minus the partnership.

We're actually discussing a remake of the Civ 4 engine from scratch, and we've got some crazy good minds in these forums here. Almost twenty years old, the game, and since a few days ago this stuff has exploded with popularity. We know there's a demand for an updated game engine for an awesome Civ game, you know. I'm not able to help with the process because I know nothing about the coding stuff they're all doing, though. But it's great to get involved in a way and spread the word.

 

[onkelm]

Awesome !

 

[Nightinggale]

I was wondering if you think it's possible to get some help from firaxis on this? Maybe a partnership. I would love a remake/rebundle of civ4 with better performance and maybe even other small improvements, better graphics etc.

Odds are that Firaxis wouldn't be interested. They most likely prefer the current setup where they sell, stores have the hassle of the actual sales and as such has become a completely passive income. For them to get involved, they would need to profit from it. Having to have paid staff to look into this as well as adding liability means their profits from joining will always be lower than them staying out. Since any exe remake/remaster would still require the game itself from Firaxis, such a project could increase sales regardless of Firaxis joining or not. I see no reason why they would join.

Also from our point of view, unless Firaxis decides to make the exe source code open source (unlikely), I'm not really sure we would gain anything from working together. Not working together means less beaucracy, which is likely what we want.

A bundle sale would naturally make distribution easier, but let's not be concerned with distribution until we have something to distribute.

 

[onkelm]

Do you have a repo for this?

 

[snowern]

Do you have a repo for this?

Not until I can hopefully finish a game.

Until then, we have a rather long scoreboard:

 

[snowern]

Odd trade bug where if you demand a tech enough, it stays in the offers, and you can still counter propose, even if they'll refuse to trade techs. More obvious when it's recoloured red. This is a Civ 4 bug, assuming my Civ 4 isn't broken.

 

[AcaMetis]

Side note, I love Napoleon's annoyed(?) diplo face being

>:(

 

[snowern]

I would maybe use the ASCII face characters, but they are always happy...

 

[snowern]

Roads (as pretty as I can make them), game data gauges, the coloured plot for events, processing the textcode for turn messages, a minimal tech splash window, diplomacy done until I start actually playing a game with it.
Really got to do something about that long BUG GP bar. Need to invent some coloured ASCII symbols or something.

So, what's the major stuff left. City view worked plot circles, all those screens like the city table screen and the research screen, unit move by dragging, minimap, worldview polishing, yield display in the worldview, Victory (if I ever get around to that), main menu, game setup UI. Playing a game will be sure to reveal some needed interface functions.

 

[<Nexus>]

I starting to like this minimalistic style
I was wondering  if it continues like this and we will really have a FreeCiv4, than maybe we could also keep (and improve) it. Maybe it would work better on smartphones... I mean... A mobile port is also considered, right?

 

[Gameboy2396]

Just a question too while I’m here. Loving every pic you send. Is this using a new engine yet? I don’t understand technicalities as much as you do. If it is a different engine, is the game, like, faster yet?

 

[snowern]

I starting to like this minimalistic style

I hope you like TUIs! Because there's gonna be a lot of it.

A mobile port is also considered, right?

A mobile port? I have absolutely zero idea about that. If phones don't have a console, you'd probably have to, like, emulate one with SDL or SFML or something. And the UI is definitely not designed for it. I suppose that would be a fork by whoever knows what to do. A framerate of approximately zero should be fantastic for battery life though.

If it is a different engine, is the game, like, faster yet?

Yes, a whole new engine. It has to be. x64 engine with x64-compiled DLL and x64 python. I think it is faster. Haven't done much testing. It can launch and load a Huge pangaea save in 1.5-2s (depending on XML caching) vs 15s mostly because there's no art assets (*uncertain about Civ4's time, might have used the DLL with logging compiled in but not active). Turn times should be faster, but that's up to how good the compiler is at optimising, as 93-95% of time is spent in CvGame::update for a slow turn.

 

[Gameboy2396]

What I was trying to ask was like, what you’re doing now with the open source thing you’ve been showing to us. I’m kinda trying to wrap my head around it all. What I am asking is if that’s been made with another engine yet or not because I’m confused (and impressed at the same time) with how you’re working on it.

By the way, at the end of the day, I still really want to enjoy a new Civ 4 from the ground up and still enjoy the 3D environment. Problem is, we haven’t assembled a developing team yet.