Improving speed to the extent possible?

[TheMeInTeam]

Alright, so this has been nagging me since before I switched over to Linux (now running Mint 14, previously windows xp and vista). No matter what I do, the game gets pretty slow over time, and running things like HoF mod slow it down slightly more.

I always thought it was a processor bottleneck leading to 5+ second times early-ish and 30+ second turn times late game. However, recently I alt-tabbed to check the cores. I have a 2 core processor where each runs at 2900. If I were processor bottlenecked, I'd have expected, when alt-tabbing in the middle of a long turn, to see at least one of the two core (since civ IV won't use 2 cores at once) actually hit 100% usage. It's not happening; i'm seeing 5-10+ second turn times where the more-used processor "spikes" to 75% at most.

Considering I have 4 GB of RAM, the other thought then goes to the graphics card. I have a Radeon (I forget specifically which now and am not where I can check, I think in the 7000's --> recall a system check saying 780 or something) that has 1 GB of RAM itself, and am running single-unit graphics, low settings across the board, no animations. There's no reason on Linux or Windows that I should be GPU limited by such an old game, yet it happened on both OS.

What IS causing this slowdown? Is there some graphics card bottleneck idiocy going on? Is my system monitor being inaccurate in saying my usage isn't hitting 100%? Why do I have to wait several seconds for my green end-turn button to turn red, always? Is there some setting I don't know about? It's sad when games like Sc2 are running much better than this one on both OS.

 

[kiwitt]

Late game there is generally a lot more scripts running and these are single threaded.

 

[Mec AntiKythera]

Does anyone have experience running Civ4 on a RAM-disk? Someone mentioned doing that a long time ago and I have always wanted to try it to see if it was any faster but I have not attempted it yet. I don't see a logical reason for that to improve speed, I doubt the HDD is slowing things down once the game is loaded, but I would love to hear from people who have tried it.

 

[mintegar]

Could it be that not seeing full cpu usage in system-monitor is because CPU is waiting for Memory which isn't in Cache? Ram is very slow, relatively speaking.

Don't know enough about linux internals myself to say for sure, so take this "cpu being blocked by memory access is counted as idle" as speculation.

Some system-monitor programs make a distinction between the User and the System part of cpu usage, and show the system part in different color, or not at all unless turned on in the options. So this is another thing to look out for, i think the system-part is the time which is spent in linux syscalls.

@30sec delay:
It seems much on a modern system. Which size of map do you experience this on? If you post an example save i could test how long it takes on my system. I would have to install that HoF mod first though.

 

[Quintillus]

What's the average CPU usage across both cores when the game is between turns? My guess is what's happening is that the OS is load balancing between the two processors, and that's why neither one is at 100%. However, whichever one has the load at any given time would be running at 100%, and the average should come out to approximately 50% of your total CPU power being used.

I can't say for sure if Linux does this, but Windows definitely does. So when I'd pull up Task Manager when Civ (either IV or III) was between turns on my Core 2 Duo laptop, the graphs would show the cores somewhere between 25% and 75%, but if you went to the Processes tab and sorted by CPU usage, you'd see Civ at 49% or 50%. If your computer has turbo boost, the average might be a bit higher - perhaps as much as 60%. The load balancing happens quickly enough that for the intervals between the measures shown by monitoring utilities, both cores will be partially used, and neither will be entirely used.

On Windows, you could test this theory by right-clicking on the process in the Processes tab, selecting "Set Affinity", and de-selecting all but one core. That core should then have 100% usage between turns. The time between turns should be similar in that case; Civ4 and Civ3 are single-threaded, and the background programs will be load-balanced to the other core. The core Civ is on would run a bit hotter, though. There may be a way to do this on Linux, too, but I don't know what it would be.

Civ (at least 4 and 3; haven't played 5 much) is quite processor-intensive, and for big maps those in-between-turn times don't sound that bad to me. Especially at 2.9 GHz. Might be a bit slow for a standard-sized map, but the slowdown as the game goes on is unavoidable.

Graphics cards don't affect in-between-turn times significantly. Any Radeon 7000 should be absolutely fine, including integrated ones. And even if you had a lowly Intel card from 2007 or so, the CPU would have a lot more effect on turn times.

mintegar could be on to something with the cache, too. I don't know how Civ4 works with the cache as well as Civ3 (and even in Civ3, I haven't had the hardware to test independently of other variables myself). But several years ago, there were reports that the Cedar Mill Pentium 4's worked noticeably better with Civ3 than the Prescott ones - and the biggest difference was the doubling of the caches in Cedar Mill. And that is a differentiation between processors today - lower-end ones often have less cache. So switching to an otherwise-equal CPU with more cache could help. But I'd be surprised if the 30-second turn times fell below 25 (although that would be interesting to learn about).

 

[kiwitt]

I have looked and analysed the CPU graphs and Civ IV does use 4 cores, but when it run python scripts it only uses one. This is the main bottle-neck. I have moved to SSD and that makes loading times very fast as most XML is loaded into the game cache.

 

[TheMeInTeam]

It seems much on a modern system. Which size of map do you experience this on? If you post an example save i could test how long it takes on my system. I would have to install that HoF mod first though.

Huge maps, admittedly.

On Windows, you could test this theory by right-clicking on the process in the Processes tab, selecting "Set Affinity", and de-selecting all but one core. That core should then have 100% usage between turns. The time between turns should be similar in that case; Civ4 and Civ3 are single-threaded, and the background programs will be load-balanced to the other core. The core Civ is on would run a bit hotter, though. There may be a way to do this on Linux, too, but I don't know what it would be.

There is, but it doesn't appear to have any utility based on what you've said.

I have looked and analysed the CPU graphs and Civ IV does use 4 cores, but when it run python scripts it only uses one. This is the main bottle-neck. I have moved to SSD and that makes loading times very fast as most XML is loaded into the game cache.

Interesting. I will definitely consider getting one the next time I buy a machine.

 

[Mec AntiKythera]

SSD's will soon be standard I should think. At least for middle to high-end machines.

I really must try the RAM-disk variation until I get a new desktop, SSD on the cheap (and small).

 

[TheMeInTeam]

SSD's will soon be standard I should think. At least for middle to high-end machines.

I really must try the RAM-disk variation until I get a new desktop, SSD on the cheap (and small).

I find this conceptually fascinating. How would I set it up? Create a partition in my RAM and then use WINE to install civ IV into that partition?

Hmm, but, I'd lose that when shutting down the machine and limit RAM used for other things. It might be better to copy a standard installation of the civ IV wine prefix into that partition instead of installing it proper (likely doable on Linux).

I might need to get more RAM just to accomodate that though, assuming it would even work the way I'm envisioning it.

 

[Mec AntiKythera]

Well, I haven't done it yet myself, and certainly not from Linux, but I can share my limited knowledge on the subject.

You use the RAM-disk software to partition a part of your RAM to be used as HDD space. You can choose to have it remember the setting and recreate it each time you boot if you so desire. This means you don't lose it when shutting down, however you do limit RAM available for other purposes. From what I understand it is often used for photo and video editing, databases, and gaming. Anything that uses large amounts of data and would benefit from quick access to that data.

I would agree that if you found this to be beneficial adding more RAM would be desirable. But that is never a bad thing.

Here is a good introduction article: PC World on RAM disks

I don't have any idea how much CIV writes to the HDD but load times should improve significantly. The article states that saved RAM disks slow down boot and shut down times by quite a bit, however. I guess that is to be expected.


There's a better article, with Windows instructions, here: laptopmag

It gives the following tip:

Fortunately, if you use your RAM disk to hold applications, you don’t need to resave the image file on a regular basis. When you install or update an application, you can manually save the image file by hitting the Save Image Now button under the Load and Save tab in the RamDisk configuration utility.

 

[TheMeInTeam]

I would almost certainly have to double up my RAM, though I'm considering it. Maybe I'll just hold out and grab a SSD along with a new machine in the not-too-distant future.

 

[strstrep]

In Linux, adding more RAM alone rather than using a filesystem in RAM (tmpfs) should help a lot if the problem is that the game is I/O-bound. The disk cache is pretty good, and will keep a lot of those Python scripts in RAM if they keep getting accessed by the game and there isn't something else that's taking up memory and bumping them out.