Memory leaks and Inter-turn lag

[DrakenKin]

This issue is very annoying. I have 2BG of total memory, and ~1.6Gb free when opening the game. (Windows XP 32.)

Currently playing on huge earth-like map (in the 1900s), and yes after a couple of hours the system runs out of memory, the turns slow down to almost a full minute (i am guessing it runs out of ram and start using the hard drives), and if I don't restart the game it will eventually crash.

I guess this will be my first and last huge/large map... Shame.

 

[Fistalis]

This issue is very annoying. I have 2BG of total memory, and ~1.6Gb free when opening the game. (Windows XP 32.)

Currently playing on huge earth-like map (in the 1900s), and yes after a couple of hours the system runs out of memory, the turns slow down to almost a full minute (i am guessing it runs out of ram and start using the hard drives), and if I don't restart the game it will eventually crash.

I guess this will be my first and last huge/large map... Shame.

Ya it seems as if Huge/large maps are an afterthought in civ IV and V. As if they literally designed it for standard then threw those in to add more options. Atleast thats the feeling I got when they never addressed the issue in IV.. and its cropped up again in V.

 

[EmpireOfCats]

Civ IV to this day always, every single time, crashes on every computer I have ever used with the huge map in late game play because of memory problems. They are not going to fix it, of course.

Maybe we will some day get games with a modern programming language like Go .

 

[Blacklanner]

I've heard about plenty of memory allocation crashes with civ 4 but i never saw one. On my old system I only had 1 gig of ram, but i didn't play huge maps with it because the processor was too slow to handle it and i didn't enjoy the huge turn lag. I've been runing an i7 920 with 6 gigs of ram (winxp64) for over a year now. Its much nicer on huge maps and i did play some with it. No memory issues ever.

Civ 5 has ridiculous but stable memory usage in my experience so far, and doesn't scale well to multiple cores. How much memory it uses varies with map size from about 1.5g for standard to around 2 for huge. I have no idea wtf its doing with all that memory, it seems like in general there should be less objects on the map than in civ iv, and i don't think the actual map dimensions are larger although i haven't tried to math that out. Even playing the 42 civ mod or however crazy many it was with civ IV, memory usage was about half of V. I've had the game running for several hours now with no sign of a memory leak. haven't been playing the whole time, I just leave it minimized a lot rather than bothering to close it if i'm going to leave my computer on.

As far as multheading goes, civ V mostly depends on one processor. I'm fair certain it will run faster on a good overclocked dual core than a stock quad. (such as my 920). I suspect I might even get better performance by disabling hyper-threading and setting civ to only try to use one core in the ini because of the automatic overclocking feature of newer intel processors when only one/two cores are in use. So far the only game I've seen that efficiently uses multiple cores is dragon age.

 

[DrakenKin]

Maybe we will some day get games with a modern programming language like Go .

It's not the language, it is the programmers and the quality standards of the company. I play Starcraft 2 and other games, and so many of them are prettier than Civ5 and run much smoother. Starcraft in particular scales so damn well on old computers I can play it on my 3 year old laptop (one of the first/slowest duals + a basic video card), which is just plain amazing.

 

[Tylerryan79]

I have tried. Have you? You don't need to go into the BIOS. Windows supports it through the task manager. Result: 100% of usage on a single core. And a single hyper-thread of that as well and small amounts elsewhere. It takes the same amount of time.

As for the even usage of your cores, the answer is simple: Windows spreads a single thread across all cores, using each one in turn (I think this is done for heat reasons).

On my computer I get 15-17% CPU usage. Given that a hyper-thread is not as good as a core and that there is some other work that needs to be done, this shows that the game basically doesn't multi-thread at all.

I did do this. I have an i7 860 overclocked @ 3.80ghz, I disabled HT and 3 cores, leaving only one. I also played using all cores with HT and then just setting the affinity to one core. The only thing I noticed was that the starting video was very choppy with one core, or the affinity set to one core. I didnt really play the game long enough to see how else it was effected, and only loaded the same save for comparison. With all cores/HT everything was smooth.

Cpu usage with one core was 79%
Cpu usage with all cores/HT was 8%
cpu usage with all cores enabled but affinity set to one was 14%

The thing I noticed is that it all pretty much scaled evenly, but like I said the starting movie with the old man was very choppy on the one core. The game may be one thread, but like you said it has the ability to spread that across multiple cores, hence making it mulit threading. No other single thread program I own is able to spread it like that. The game is able to break thing up into smaller jobs, and it works well at that. You may not be satisfied, however I am.

 

[Infiltrator]

Definitely memory leak. It is especially noticable when leaders try to talk to you after extended periods of time.

 

[Zelig]

Has anyone else noticed that memory usage grows over time? I started a new game and the software was using 2.6 GB of memory. I re-started civ and did the same thing. Only 850 MB, one third as much.

This isn't necessarily indicative of a memory leak, the game could simply be caching more data in memory from a longer play session - data that isn't cached until you use it for the first time.

As for the even usage of your cores, the answer is simple: Windows spreads a single thread across all cores, using each one in turn (I think this is done for heat reasons).

Windows doesn't do this.

 

[Abegweit]

This isn't necessarily indicative of a memory leak, the game could simply be caching more data in memory from a longer play session - data that isn't cached until you use it for the first time.

Nonsense. What data do you think it should cache when I start a new game? It should reset back to the state it had at the start of the previous one.

Windows doesn't do this.

This is yet more proof that you have no idea what you are talking about. It most certainly does.

 

[Zelig]

Nonsense. What data do you think it should cache when I start a new game? It should reset back to the state it had at the start of the previous one.

What do I think should be cached? Pretty much everything, I've got lots of memory, if caching 6GB of data is going to improve performance, great. Obviously, there should be some sort of priority system to limit what gets cached when less free memory is available.

What do I think might be getting cached? Leaderheads, terrain graphics, unit graphics, sounds. It makes sense not to load Monty's graphics into memory until you meet him for the first time. It also makes sense to keep the graphics in memory at the start of a new game, you might meet Monty again.

FWIW, memory usage of some browsers when they're opened up and directed to Google:
IE: 11mb
Firefox: 26mb
Chrome: 10mb

Memory usage after some tabs (CFC, Apolyton, 2kgames, Firaxis, Steam) are opened and then closed again:
IE: 21mb
Firefox: 64mb
Chrome: 23mb

Oh no! They're all leaking!

This is yet more proof that you have no idea what you are talking about. It most certainly does.

Nope. By definition, a thread can only run on a single processor at a time. If a single-threaded process is showing equal usage on a number of cores, it's just getting moved around from core to core quickly, but still only executing on a single one at a time.

 

[Lancor]

Can't say much about the memory issue, never monitored it.

But about the turn times... they are okay for me at last. I'm barely above recommended system stats and while a turn in the first few dozen turns of the game takes ~1 second, it's only about 15-20 seconds in late game. Not really an issue.

 

[Abegweit]

What do I think should be cached? Pretty much everything, I've got lots of memory, if caching 6GB of data is going to improve performance, great. Obviously, there should be some sort of priority system to limit what gets cached when less free memory is available.

What do I think might be getting cached? Leaderheads, terrain graphics, unit graphics, sounds. It makes sense not to load Monty's graphics into memory until you meet him for the first time. It also makes sense to keep the graphics in memory at the start of a new game, you might meet Monty again.

FWIW, memory usage of some browsers when they're opened up and directed to Google:
IE: 11mb
Firefox: 26mb
Chrome: 10mb

Memory usage after some tabs (CFC, Apolyton, 2kgames, Firaxis, Steam) are opened and then closed again:
IE: 21mb
Firefox: 64mb
Chrome: 23mb

Oh no! They're all leaking!

There is a slight difference between 11 Meg and 2 Gig. Good Lord.

BTW, that difference, small as it is, is easily understandable. It comes from caching open pages. You still haven't explained what info you think that ciV should save from one game to the next. Hint: the answer is nothing.

Nope. By definition, a thread can only run on a single processor at a time. If a single-threaded process is showing equal usage on a number of cores, it's just getting moved around from core to core quickly, but still only executing on a single one at a time.

Did you read what you started arguing with? This is exactly what I said it was doing. You actually quoted me.

Windows spreads a single thread across all cores, using each one in turn.

Right. That's what it does. You actually can see it by looking and the Performance Monitor. Assuming that you have a Quad-core system, the load on each core will be around 25%. Maybe less. And guesses as to why that might be?

 

[Zelig]

There is a slight difference between 11 Meg and 2 Gig. Good Lord.

BTW, that difference, small as it is, is easily understandable. It comes from caching open pages. You still haven't explained what info you think that ciV should save from one game to the next. Hint: the answer is nothing.

26mb -> 64mb is a similar proportional difference as 850mb -> 2.6gb.

I gave several examples of things that might be getting cached (Leaderheads, terrain graphics, unit graphics, sounds.) Stuff that should be getting cached depends on available memory, and is a design decision, that doesn't necessarily have a correct answer. In particular, a lot (I'd say a vast majority) of loaded games from ingame are going to be of the same game being played at the time, so it makes sense to keep things like leaderheads cached.

That's not really the point though, the fact remains that simply seeing process memory usage increase from 850mb to 2.6gb in a session based on quitting/starting a new game is not enough to diagnose a memory leak. Sure, there might be a memory leak present, but from this information, without seeing the code, we can't know for sure.

If you can get the game to increase memory usage to any arbitrary amount in a way that is reproducible, that is a memory leak - if the memory usage increases by 1.8gb every time you start/play/quit/start a game, that would be a problem.

Did you read what you started arguing with? This is exactly what I said it was doing. You actually quoted me.

My mistake, I missed the "using each one in turn".

 

[Abremms]

just curious, but has anyone on a 32bit system with 3.whatever gb of ram tried making civ5 large address aware? did it help?

as far as leaks and slowdowns, the civ5 engine was essentialy built from scratch for the game. compared to other games that have built thier own engines instead of liscencing a known functional engine, the civ5 engine is pretty remarkable. not perfect, but its in decent shape for a 1.0 release.

 

[skrewler]

As Zelig stated, an app using more memory the longer that it's being used isn't indicative of a memory leak. Most programs will do this.

In fact, diagnosing a memory leak is pretty much impossible without access to the code.

With that being said.. you can use perfmon in Windows to check some stuff. Start/run/perfmon add counter Process/CivilizationV

Leave the app running overnight. "Working set" shouldn't be increasing without "Working Set - Private" also increasing. Total threads shouldn't be increasing without bounds either.

Run task manager, goto processes, find CivV, View/Select Column/Working set Delta

You should be seeing positive numbers, but you should also see negative numbers. If you never see any negative numbers, there might be a memory leak.

My quick analysis: there's no conclusive evidence of a memory leak.

 

[Abegweit]

I did do this. I have an i7 860 overclocked @ 3.80ghz, I disabled HT and 3 cores, leaving only one. I also played using all cores with HT and then just setting the affinity to one core. The only thing I noticed was that the starting video was very choppy with one core, or the affinity set to one core. I didnt really play the game long enough to see how else it was effected, and only loaded the same save for comparison. With all cores/HT everything was smooth.

I agree about the opening video. It does seem that the game uses multiple cores for that. Although, to be more accurate, it is likely that this effect is actually DirectX in action. The game is simply gaining a benefit from the optimization that Microsoft and the GPU manufacturer built in the system.

Cpu usage with one core was 79%
Cpu usage with all cores/HT was 8%
cpu usage with all cores enabled but affinity set to one was 14%

Right. This is proof that the game is not multi-threaded. If it was, you would see lots of activity across the entire system when you enable multiple cores. Not just when you only have one running.

No other single thread program I own is able to spread it like that. The game is able to break thing up into smaller jobs, and it works well at that.

You think 8% CPU usages is doing the job well? I sure don't. In any case, it's Windows (in conjunction with your task manager options) that chooses where to run the thread; the game has absolutely nothing to do with it. If the game was indeed multi-threaded, you would see heavy usage everywhere.

 

[sleepingbear]

I've left civ running on my machine for over 24 hours at a stretch and not noticed any issues. I'm lazy and don't bother to shutdown when I go to work. My machine is set to sleep after an hour of no use.

 

[skrewler]

Right. This is proof that the game is not multi-threaded. If it was, you would see lots of activity across the entire system when you enable multiple cores. Not just when you only have one running.

You think 8% CPU usages is doing the job well? I sure don't. In any case, it's Windows (in conjunction with your task manager options) that chooses where to run the thread; the game has absolutely nothing to do with it. If the game was indeed multi-threaded, you would see heavy usage everywhere.

I don't know if you're trolling or what, in any case, stop making stuff up..

http://img840.imageshack.us/img840/9729/singlethreadedlol.png

 

[tokala]

It may indeed be "heavily multithreaded" but it seems the performance critical parts aren't or are bottlenecked by something else.

 

[civhawaii]

Theres something fishy about loading games. Try loading a game, then loading another game, eventually my system just freezes.