How To: Compile WtP 4.1DLL from source ! 08.05.2024

[Nightinggale]

I am currently watching one 40 minute Netflix episode between each turn.

Can I have that savegame?
Sounds like a good candidate to measure where the CPU time ends up. Think something like when Let's game it out managed to get Satisfactory down to like 1 fps. The developers received the savegame, found the bottleneck and then made his savegame playable. He then proceeded to do something else crazy and got less than 1 fps again, but that's another story.

As for performance issues, we are aware of a few and we do have plans to fix them, like actual implementation concepts, but it will take time.

Only got 8 core Ryzen

That's not very specific. That can be anything from Ryzen 1700 to 7800X3D, the latter being 130% faster and that's not even taking into account that the game seems to love the 3D cache. Besides the number of cores is currently not that important as most bottlenecks seems to be single threaded anyway, hence 4 cores will be faster than 16 cores if each core is faster in the 4 core CPU. This might change in the future through as we aim to use more cores. However vanilla will only use a single core, so if we use more, it will have to be something we write ourselves from scratch to do so. That's not trivial.

 

[Cyberian]

yes its only 3700x
i will probably upgrade this year again these 3d cache models sound great

@Nightinggale heres my savegame 1766 on big Americas map https://drive.google.com/file/d/17bW6vY6wUYTKoN66HTVlx03LK94BP_cX/view?usp=sharing

 

[devolution]

@Cyberian

The poor performance of trade routes is something that has escaped me, since I rely almost exclusively on AI autoplay for testing performance. Non-AI players generally assign trade routes, which are not currently used by the AI (instead, they use the 'automate fully' alternative), and hence I haven't been aware of this particular issue.

Since I suspect that redundant pathfinding is the reason for the lackluster performance, I've added a caching layer that I will test shortly. Hopefully, this should improve the performance quite a bit until we can properly rework the trade route system in its entirety.

 

[Barra]

it seems you guys already found a solution for TR painfully extending turn times.Either Branch AI_Traderoute_rework OR branch traderoute_revision and/or branch task_parallelism (keep in mind i dont really know what im doing i just merged all the stuff ) , seems to have solved the long turn times for me.
I just merged all into my barradev branch (WTP dev branch + a bunch of edits/branches I found while browsing long forgotten realms ). Im now at Year 1715, huge map with extended landmass, 8 colonies, a whole bunch of automated carts and ships. Turn times are at 5-7 seconds.
Im not sure yet but I would say its indeed a impactful shortening of round times.
Im with you devolution, i have the gut feeling, one of the pathfinding changes in them optimized it to some extend.

Its to mention that because all of it is just randomly merged, there are ctd bugs BUT they are verrrryyy rare, once in a session of many hours, but the time spared is worth it.
Already compiled Vers


Edit: Ok lol your save game IS massive, but still it tooks round about 4-6 mins while fight anim is on

 

[devolution]

@Barra , it would be great if you could test this branch https://github.com/We-the-People-civ4col-mod/Mod/tree/cache_traderoute_paths

 

[Barra]

@devolution Dev Branch + cache traderoute paths works quite well, but I'm not sure if it really makes a difference. A few times there is a CTD at the end of the round without the logs making (me) smart.

If you end all imports/exports in this (pre)round, you can reliably enter the next round, if you keep the trade routes, I have the feeling that it is necessary to go back a few autosaves. My gut feeling is that a cached route may not “notice” that the route is no longer available, wants to “enforce” it but it doesn't fit the “rules of the game”. But it could just as well be something completely different in the cache context. The fact is that I only experience these crashes with the cache branch.

I'll check today to see if I can find the save or get a new one. However, it is difficult to find a save with a reproducible error, as it happened that the error was repeated with the same save, same save somtimes not, but then the issue could reoccur on the following round.

The bottom line is that I don't find any perceived performance problem in the raderoutes since the dev.branch post 4.1

At least not in terms of round times, rather it would be nice if the popups for traderoutes would open more performant, unfortunately the Domesticadvisor is also very laggy from the mid. game to late game before it opens.

I was/am trapped by ColoWTP for the last 2 weeks like every year. Therefore, all the greatest respect to all involved, especially because the colo release of Civ 4 was probably intended more as a loveless homage to the original by the developers.(I can remember there was not even an EU Screen)

Edit:
several games, Setting huge world, speed epic and marathon, once until independance. tested atleast 25 hrs.

 

[devolution]

@Barra
Thanks for testing!

Apparently there was a recent CTD bug that was reported by @Mr. ZorG in the internal discord. This has been fixed in the recent develop and I've just rebased cache_traderoute_paths on it. Hopefully the CTD should be fixed since I can't really see how the cache would be the culprit since it is re-generated on every call to AI_tradeRoutes and hence there is no bad state than can creep in.

Another tip: Play with the assert build, it will make it far esier to detect bad stuff happening

 

[Cyberian]

@Barra my save a few turns later is now in an endless loop. I tried it twice while going to sleep and it did not finish by next morning. This is now with your above "Barradev" branch

if it helps this is the save https://drive.google.com/file/d/1BKz5ATj8XhACzCVYF8YG5QCtP8yGEEhr/view?usp=sharing

edit: also loaded savegame a turn earlier and still game freezes forever midturn on July 1769 when traderoutes happen

 

[devolution]

@Cyberian
I tried to load your save with both the latest cache_traderoute_paths (dece294) and develop but it fails due to this this error:

This is puzzling since DevTest (could not find the elusive BarrasDev ) did not add any serialized members (not that it should matter). However, the issue resembles https://github.com/We-the-People-civ4col-mod/Mod/issues/1005 (which should be fixed by now but the save could not be salvaged though so a new game would have to be started)

 

[Barra]

@Barra my save a few turns later is now in an endless loop. I tried it twice while going to sleep and it did not finish by next morning. This is now with your above "Barradev" branch

if it helps this is the save https://drive.google.com/file/d/1BKz5ATj8XhACzCVYF8YG5QCtP8yGEEhr/view?usp=sharing

edit: also loaded savegame a turn earlier and still game freezes forever midturn on July 1769 when traderoutes happen

Will check this today in the evening, I probably have to compile it again, to be hones I had the freeze issue on your first save too, in the moment when the own turn should start and im asked to choose a trained profession. Because the compiled uploaded version here is not updating itself I forgot to reupload a working compile, if it is urgent it probably does help to cancel all TRs and then hit next round.
Anyway an updated recompile is needed because of the ctd bug the pros found

 

[Nightinggale]

@Cyberian
I tried to load your save with both the latest cache_traderoute_paths (dece294) and develop but it fails due to this this error:
View attachment 691587


This is puzzling since DevTest (could not find the elusive BarrasDev ) did not add any serialized members (not that it should matter). However, the issue resembles https://github.com/We-the-People-civ4col-mod/Mod/issues/1005 (which should be fixed by now but the save could not be salvaged though so a new game would have to be started)

This is really weird as the game crashes inside the exe file. There is nothing from the dll in the call stack. I have no idea what happened or what fails. I don't know how to gather more information either so the savegame is just completely broken for some reason we will likely never be able to figure out.

 

[Cyberian]

Thanks for everything when a new build hits that I can download I try loading 10 turns earlier or better just start a new game.

 

[Barra]

Thanks for everything when a new build hits that I can download I try loading 10 turns earlier or better just start a new game.

Ok just turned all Traderoutes of citys off, this does lead to the turn beeing able to finish and starting the following turn.
So it indeed is somehow related to traderoutes or pathfinding (i would say) its also remarkably more performant, the turn also took just a minute or so.
BUT sure, the save is absolutly overkill. I dont know whats causing the issue besides TradeRoutes in this amount in general.

Maybe its to many ships tryimg to enter a city, but this is just fantasy, i could imagin such things but thats no hardfact (for sure lol...)
or
Maybe there is a hardlimit how big a file could be and therefor howmany TR could be active?! (because max ramsize seems limited to ~2GB)
So basically an ram issue (not yours in specific)


I dont know, anyway 10 Turns would cost you by the previous performance 24 realtime hours i think. In most cases (own colo gaming experience) it is enough to go back 1-3 autosaves, BUT this does not say that the issue will occur a few turns earlier or later...
Besides that I think resetting the TR would probably going much faster.
Tried to overcome the freeze, while letting TR as is with the current dev version by today and with TR Cache, both did not help.
You could also manually cancel all ship traderoutes on unitbasis, this would probably also help to get over this turn...but dont know when it will appear again.
I have found no clue in the log (because it freezes midturn and not at end of turn/while loading next turn, to be exac:t it freeze when the in-turn automovement should be proceeded)

I would try to setup all the routes again, if you like to proceed on this massive save (in general way to big for such an oldtimer 1 CPU game...atleast its now almost without opponent KI ...just my 2 cents)

In general I had the last 2 Weeks several games, 2 times I had repeating issues (crashtodesktop) while loading into next turn. Both times it was related to specific opponent colonys...somehow. Once i couldnt destroy/capture last colony and one time i dont know, AI log just stopped each time by an opponet nation and huron AI were plannings a war against each other. For this lategame i just used the worldbuilder and force shrinked the AI of the nation to a far away colony, problem also solved...somehow

Link to reset the modsname of the save(one click solution):

Civilization 4: Colonization savegame mod renaming tool

Edit: Link to compiled very current DevBuild as of today (overall working well):

we-the-people-4.1.7z

drive.google.com

 

[Cyberian]

@Barra
playing with the build you posted BIG THX for the effort. Tried enabling trade routes again and a single ship then takes 56 seconds when I click the Auto trade button plus the new turn takes ages. BUT disabling all traderoutes and disabling all Automate functions on all ships and waggons makes next turn extremely fast and removes all issues. So I am actually finishing the game.
I guess the trouble is just the combination of the extremely high amount of cities and units plus map size and maybe the long shape of the Americas map does not help either when pathing between west coast and east coast.

I am extremely happy that I can actually finish the game and we didnt have Automate functions in the olden days either and I just buy Custom houses everywhere plus big stacks of Carriages for the inland cities should work.