Lategame Slowdown

Hey everyone, seems this forum is pretty dead these days, but no worries.

I just started playing FFH (I know, I know), and so far I'm loving it. Really well-done mod, and so far every new unit and civ I encounter has been an exciting experience.

The one thing really holding me back from enjoying it, though, is that late-game I run into annoyingly long times between turns. Now, part of this is the fact that my games are probably going way longer than intended because I want to play around a bit and try and understand the game more (aka completing the tech tree...), but looking around it seems to be a fairly common problem and I was wondering if anything's been done to address it?

As far as I can tell, the main issue seems to be that the AI likes to build a massive amount of units. The problem with this, as far as I can tell, is that those units tend to be pretty weak, meaning the computer ends up devoting a lot of processing time moving around a whole bunch of units that the AI doesn't really profit all that much from having.


Has anyone come up with a workaround for this? A modmod that makes the AI build and prune its army more intelligently? Or even one that just increases unit maintenance to a punishing level once a certain amount have been built? Because my computer is decently fast (8 GB of RAM, plus I'm running the game off an SSD, though I don't know if that would make a difference in this case) and it shouldn't be having these sorts of problems on standard map sizes.


I'd like to try Large or even Huge, because the game seems like it would be fantastic on a grander scale, but given how this is my experience on Standard, I'm a little apprehensive to do so...

You should probably look into getting one of the modmods like More Naval AI. There has been quite a bit of AI improvement (amongst other things) to the game. I've heard good things about Erebus In The Balance and ExtraModMod as well and I'm sure the other modmods are also good as well I just haven't looked into them much myself.

I can assure you that the problem that MilesBeyond has with long turn times is not fixed in ExtraModMod, as it is one of the things that bugs me the most about the game and that I have never gotten around to fix. One of these days I should profile EMM again and I'll try to improve turn speed...

They should have made tigers a 1 turn duration summon, but i think the real problem is how bad is the AI at waging war. Arendel Phaedra has this tendency to build stacks of hundreds of Priests of Leaves, and when she enters war the game goes boom with all the tigers moving randomly back and forth. Yeah, i have seen it with birds. The AIs "battle plans". My guess is, when they reach an absurd number of units, they just dont know what to do with them, this is especially true in the case of summons like Tigers, or Treants, or whatever (i once witnessed how Charadon used "wild hunt" for his war against Auric: his stack of like 40 wolves stood in a hill distant from the battlefield, they didnt move at all in the course of the war. Not that the Doviello human battalion was being much more useful, all they did was move back and forth between 2 Illian cities, to finally be slaughtered in a suicidal attack when the Illian army came by (beastmen vs champions).

Long turn times is improved in most"modern" mods, but not by much (and AI improvements can increase the number of decisions, anyway). Its still a noticeable issue, for sure.

But I'm not sure it is possible to make it better. I know Tholal looked at speeding up the game/making the code more efficient a while back, but found that there weren't any obvious areas for improvement. I'm nowhere near a good enough coder to be able to judge that, but I'd be surprised if large improvements could be made to turn speed without compromising other game functions.

Has it been tested to see what turn times scale with?

Is it more number of AIs or number of units? Or size of the map? Is the pathfinding/plan generation taking up more of the time?

Or could it be the maths with traderoutes, where more cities = more nodes in the graph, and the complexity of the graph scales exponentially with each additional city? I remember when blockading would make the processor work hard and slow the game down.

The_Reckoning said:

Has it been tested to see what turn times scale with?

Click to expand...

To my knowledge, no thorough performance/profiling tests have been made on vanilla Fall from Heaven 2 / More Naval AI. Others FFH2 based modmods may have done it and added improvements, but I'm not aware of any such attempts (if anyone knows of any, I'd be quite interested on learning about them). I made a naïve and misguided attempt some time ago (see http://forums.civfanatics.com/showthread.php?t=487573), but besides helping me in learning a lot it wasn't very useful. I'll explain why I abandoned it later in this post.

The_Reckoning said:

Is it more number of AIs or number of units? Or size of the map? Is the pathfinding/plan generation taking up more of the time?

Or could it be the maths with traderoutes, where more cities = more nodes in the graph, and the complexity of the graph scales exponentially with each additional city? I remember when blockading would make the processor work hard and slow the game down.

Click to expand...

It would definitely be worth the effort of testing all of those factors separatedly, along with some others. Other Civilization IV mods have tackled some of those issues; both Caveman 2 Cosmos and K-Mod include a new pathfinder and AI performance improvements, C2C has new code for trade networks, and so on.

But, in my opinion, starting by trying to tackle separate issues without doing a global analysis first would be a mistake. Once that a global analysis is done, all found issues could be addressed ordering them by the amount of problems they cause. Of course, once that one of the problems is identified as one of the possible problems you mention, the whole issue should be tested as a whole.

I'm going to get more technical now, as I'd like to share my impressions and what I'm planning to do in the future in order to address this problem. If you are not interested in technical details, the TL;DR of the following wall of text is that in some months I may be doing some stuff that would improve this situation a bit.

----

As I mentioned earlier, some years ago I made an attempt at increasing game performance, which failed because of the following two reasons.

• My analysis was wrong. My problem was how much time ending turns needed in the late game, but instead of analyzing just that, I analyzed X turns while running the game automatically. When I fixed stuff I saw great improvements on the numbers, but the game felt as sluggish as ever. Once I realized this, I did not want to start all over again because...
• ... while trying to fix performance problems I wasted most of the time chasing OOS errors. Whenever I ran a multiplayer game with some friends to test my performance improvements, we would run into OOS errors. I could not be sure if the error was caused by my performance changes, or if it was an error already present in the game. Since these test games took hours and always ended in frustration, I couldn't really justify wasting so much time in this problem.

The first issue has an obvious solution; I should just create a monstruously huge game and debug only end turns (like Caveman 2 Cosmos did with their scary 6 minute end turn savegame, see http://c2cdev.blogspot.com.br/2013/06/multi-threading-support-infrastructure.html).

The second issue is what has stopped me from trying again, as it did not have such an easy solution. Because of this, I redirected my efforts to a different problem. In these years I've spent a lot of my time chasing and solving OOS errors in More Naval AI, in order to both improve multiplayer games, and to make sure that any OOS errors found after making a performance change can mostly be attributed to that change. The game is now mostly OOS free; the only OOS error I'm aware of happens when someone creates a puppet state.

I recently coded a special game mode in my mod (available only by compiling with a specific flag) which does two things. The first one is making the AI take over human players automatically in multiplayer games. It only stops when the game detects an OOS. Therefore, I can just open up two instances of the game, make them fight each other, continue with my life and return some hours later to see if they managed to break the game or not (my english knowledge is not good enough to express the joy I felt when I no longer had to waste my evenings pressing Enter).

The second thing it does is implementing an extremely strict synch check, that runs all default calculations (and some new ones) every turn instead of in certain turn slices. This allows to make sure that the game will stop as soon as the problem which caused the OOS can be detected. For those who are curious about the implementation details, you can check most of the changes here: https://bitbucket.org/Terkhen/extramodmod/commits/c143a06da322e627a0834432c09443b113f3e6c3. With regard to fixing game performance, this special mode will allow to automate playtesting performance improvements to make sure that they don't break multiplayer, reducing the effort required to address this problem considerably.

As the sources of my frustrations when trying to fix slowdowns have now been dealt with, I want to give another try to improve them once I finish with the last big features I have planned for my mod. I don't plan to dig too deep (it would take too much time), but I would fix the most obvious performance problems and set everything up in case that others want to improve the situation further.

What I'm planning in this regard can be checked in the following issue: https://bitbucket.org/Terkhen/extramodmod/issue/269/profiling-and-performance-improvements

It will still take me some time to address these problems, but if anyone wants to suggest any improvements in what I plan or point any errors and mistakes, I would be thankful. Also, if someone wants to give this a try before I'm able to do so, I would be glad to stop whatever I'm doing in order to port and request the inclusion of the required tools in More Naval AI (automated OOS testing and C2C's profiler) in order to help that person start.

I read about what other mods have done with regard to performance, and created a compilation of other possible improvements that should be helpful in case that anyone is willing to address them. I don't plan to do these as they'd take too much time. You can find them here: https://bitbucket.org/Terkhen/extramodmod/issue/268/possible-performance-improvements

- Very interesting Terkhen. Thank you for your perseverant work.

- What I can say about the lategame slowdown, it's the only way I have to stop it for a small moment (several turns), it's to restart the game. The problem on me come from the lack of RAM memory. I often cannot reload save game when the game is in mid-late game due to a bad RAM.

It's very annoying to have to do this when I play to ExtraModmod in multiplayer. There is clearly a memory leack.

I don't know the ins and outs of how the algorithms and the whole methodology involved in this.

But the game broke me with these stacks.

I was playing Magister's Modmod. It was kind of late and what I thought was a well fortified city was being attacked by a stack ...of something I don't remember.

You know how when they have too many units you can't really count them right? Anyway I had like 4 longbowmen, some other miscellaneous units, Bambur, something else in the city.

I hit next turn and went to get a drink after the fight started. Decided to make a sandwich.

I come back five minutes later and the fight is still going on, one of those combat animations every 10 seconds or so. I sit and watch this crap for another couple of minutes. I can't even imagine how big this stack was and how many low end units got crunched through. In the end I lost the city.

What gets me is that Civ 5 really doesn't seem to slow down in the end game. Some of the FFH mods are unplayable at the very end due to the slowdowns. Or you have to disable some features of the mod to speed things up, like Magister and his crazy code for Ashen Veil and demonic units.

Anyway long story short, I finally decided to try Civ 5, which seems to work a whole lot better when you get used to the different mechanics.

I am never going to play this mod or Civ 4 again seriously.

Whoever designs these mods... well you need to do something with this thing. Number of cities, cost of units, nothing even slows the AI civs down honestly.

It's just stupid and no fun. I've just plain had it with this kind of thing.

If I understood you right and you're complaining about the length of the combat animations you should maybe try the "Quick Attack" and "Quick Defense" options (via CTRL-O).

It's the sheer number of units that are involved. Anyway I've had enough of the mechanics involved with this version of the game.

It just shouldn't be possible to have that many units. With Magister's mod I've gone into worldbuilder and seen civs with stacks of 92 soldiers of kilomorph on round 190 something. This was pretty much the bulk of their units, in one stack, but still...

Maybe someone else gets the strategy or whatever behind it, but it is just stupid and tedious in my book.

Then there is the absolute endgame where you might have to wade through around 20 cities to destroy a civ. Not my cup of tea. Do one, do two fine. Do ten and we have a boredom factor going on.

Jojo_Fr said:

- What I can say about the lategame slowdown, it's the only way I have to stop it for a small moment (several turns), it's to restart the game. The problem on me come from the lack of RAM memory. I often cannot reload save game when the game is in mid-late game due to a bad RAM.

It's very annoying to have to do this when I play to ExtraModmod in multiplayer. There is clearly a memory leack.

Click to expand...

ExtraModMod (and nearly any other program in Windows, really) usually crashes right when a memory leak happens, and if it does not crash immediately it'll crash some seconds or minutes later. They are very noticeable; in fact the insidious bug present in 0.5.0-beta1 was a memory leak, and fixing it made me release 0.5.0-beta2 early. To my knowledge 0.5.0-beta1 is the only EMM release with a known memory leak, so make sure that you are using either 0.4.2 or 0.5.0-beta2.

Your problem sounds as if the game wasn't freeing memory properly (maybe in english this kind of error is called memory leak too? if so, excuse me ). I have never noticed a problem similar to that one; when we are playing multiplayer at about turn 240 and we need to restart because of an OOS error, the game continues being as slow as it was before the restart. If you can reproduce this issue with 0.5.0-beta2, please provide me with a multiplayer savegame from 12 or 16 turns before it happens and I'll give it a look.

Ok, thank you. I'll do it if I meet again this problem.

sunbeam said:

Whoever designs these mods... well you need to do something with this thing. Number of cities, cost of units, nothing even slows the AI civs down honestly.

Click to expand...

Yeah I share your frustration with this. I really think the answer is maintanence costs for cities, troops, and civics does not contrast with the amount of commerce you can develop. Maintanence costs in general need a serious jack because by mid game your making monster amounts of cash.

I know the AI loves dotting the landscape with cottages, and I think costs don't really take into account how much an economy can grow. The inflation needs looking at too, maybe even key to aid in fixing the problem.

But they must know this already, perhaps it was their intent to have superstacks of 95 units running around? Seems silly as I think its the source of the issue.

Perhaps a semi-realistic game mechanic could discourage massive SoD's - e.g. stacks of living units >20 run the risk of plague/diseased (promotion) occurring. Would require reprogramming of AI unfortunately.

Bad Player said:

Perhaps a semi-realistic game mechanic could discourage massive SoD's - e.g. stacks of living units >20 run the risk of plague/diseased (promotion) occurring. Would require reprogramming of AI unfortunately.

Click to expand...

The SoD was a long running issue in Civ 4 (and before?, I really only played FFH with Civ 4). I guess that was one of the things they really tried to fix with Civ 5.

But your plague thing. That is a really cool idea, and it gibes well with actual history. To quote Corb Lund from his "Horse Soldier, Reprise" song:

"TB and dysentery, which in the end proved to have killed more men than the vilest enemy."

If it can help about this suggestion of plague into large stacks, I advise people to look after the Ancient Europe Mod, which include this kind of feature : Link.

And this mod is, to me, in the top five of the best Civ IV mods, after FFH2 Extra Mod Mod and Master of Mana Xtended.

Wouldn't the plague thing be a bit pointless, since if the stack has a priest with cure disease then the problem is solved?

Plus that's not a solution to the problem. 90 diseased units are still 90 units which eat up processing time and will have to generally be defeated through 90 separate combats. It's actually really simple to deal with 90+ basic units by T190, if you know what tools to use. Just hitting a stack with the appropriate collateral spells will be as or more effective than having them all be diseased. But again if the issue is that it just takes to long to kill 90 units in terms of time investment, then there unfortunately aren't many solutions. Best I can suggest is crippling those stacks with some combination of blinding light, debuffs and collateral, and then just stack-attacking your units into them once you're getting 90% odds or so.


And for the love of The One, turn off combat animations. They've slowed Civ games to a crawl since Civ 3.

Just related to the SoD discussion - maybe use the auto-damage all units in stacks >20 that Rhys and Fall uses for foreign units in Russian territory?