Memory Leak?

[Harkonnen]

in my example for 1. I assumed 512Mb of RAM, so 300-400Mb is the limit. For 1GB something like 800-900Mb would be a limit when the game swaps itself. Swapping is maintained by OS, so not only civ4 will see some of its data swapped out, but other processes too. That's the reason of slow alt-tabbing for instance. If you run task manager after alt-tab, you'll see most of your processes at 10-100kb of memory)

 

[oldStatesman]

in my example for 1. I assumed 512Mb of RAM, so 300-400Mb is the limit. For 1GB something like 800-900Mb would be a limit when the game swaps itself. Swapping is maintained by OS, so not only civ4 will see some of its data swapped out, but other processes too. That's the reason of slow alt-tabbing for instance. If you run task manager after alt-tab, you'll see most of your processes at 10-100kb of memory)

I can verify this limit for 1GB - at least on my machine. 780 -790MB to be a bit more exact. Also, processes do exhibit said behavior after alt-tab.

 

[I_batman]

I_batman

3. Yes. But that nasty ~e0005.exe (or alike) process is not only CD protection. It also guards main executable from penetrations like mine. So in a few seconds I see all my API hooks erased. I know ways to fight this, but this'll take time. My aim here is to bring this "patch" earlier than Firaxis brings its own. Just for the sake of my own Too many waiting, and too much effort so far...

So what you are trying to fix is how much memory the game tries to allocate, but the fundamental problem of some Python scripts not releasing memory still exists.
Would that be a fair statement?
I can only imagine the nightmare the Firaxis guys are having poring over code looking for memory release issues, if indeed it is a root problem, and if they are trying to fix it.

 

[BlueLikeYou]

The game is definatly using more memory as time goes on...but its difficult to say what that means. Because I would expect a greater memory usage as the game progresses (more things to keep track of: units, cities, diplomatic deals, etc). So I guess the question would be is the memory usage increasing by more than it should? Not sure how we could tell...maybe a comparison with civ3 would help.

Here's a couple of graphs:

Here you can see the memory usage slowly increasing (cyan line is kind of hard to see though ). The game crashed at the point where the data ends (I have 1 GB RAM).


In this one, I started by loading a saved game (sometime in the 1900s) then started a new game (that one wasn't going so well). It looks like it releases all the used memory once you get out of the game itself, but then jumps to 500+ mb once the new game starts, and climbs from there. (used a red line this time ) Both of these were on Huge worlds.

 

[alexti2]

I_batman
1. Yes, you are right. First it reaches ~200Mb free physical memory and starts to swap out other running processes since they were least recently used (LRU caching). Then when it reaches 300-400Mb of physical memory, there are no other processes to swap out, and it starts swapping out itself, i.e. what it loaded a few seconds (minutes) ago. When you scroll to those units/terrain, they are loaded back and some other stuff is pulled back onto the disk. Advisor screens occur rarely, so they are likele to be swapped out first, thus entering them becomes a pain. Same stands for globe view (clouds and stars are new, previously unused textures when you enter it).

Just to add few things here. If there was a simple memory leak and Civ4 were using OS's allocator, the problem would be curable by increasing swap file (what windows calls 'virtual memory') because the lost memory blocks wouldn't be in use and OS would swapped them onto the disk, so the game would always have enough memory mapped to RAM to run well. But this doesn't seem to be the case in Civ4. And memory increase happens in big jumps. I think they use some other memory manager that just grabs a big chunk of memory from OS and manages it on its own (which is not a bad idea considering how slow windows memory manager is). But now, if the leak happens, and if this leak is not a one big block, but a lot of small blocks, the lost memory get fragmented, so no particular memory page is being idle, so from point of view of OS all pages are in use (because some parts of them are in use), so it can't swap them out onto the disk completely. And from point of view of memory manager, the programmer still needs those blocks. One would have thought that it's virtually impossible to have a memory leak in C++, but apparently one would be wrong (unless the leak is in Python). Hopefully, it's just some stupid bug in frequently used class...

 

[BlueBomberDude]

Great charts BlueLikeYou. Confirms what many of us have thought... and experienced.

 

[Harkonnen]

A little more exploration... I've put away one of my 256Mb sticks and stayed with 256Mb. For me my heuristics didn't work, so it consumed same 700Mb (I had to take some sleep before it loaded).

Probably it does not work below 512Mb, and those 690Mb is just a floor, and the game can't run below. I will try my executable with some 1Gb systems to check if it will help at least to 1Gb systems. It will be funny to see that it helps everyone except me....

Also during one of my tests I managed to make Civ4.ex dying slowly (pressed exit to desktop, alt-tab AFAIK), and in task manager I noticed that it frees its memory in 10-20 chunks... Mostly looks like very ineffective API calls where simple 'malloc' or alike would do. BTW, Python uses malloc and malloc only, but still Pythong is the only facility here conceptual enough facility to claim entire RAM for it. Well, probably there are two instances of it are running...

You may help me testing if you want. I am particularly interested in VM size when the game loads (my fix is attached). It reports 4 times less used/total RAM. Just start 'Civ4Fix.exe' from civ4 folder instead of Civilization4.exe on some heavy savegame of yours.

I have attached source files (for VC++ 6.0) to assure that I spread no bulls**t into your system. Personally I would be afraid launching unknown exe

I recommend to stop any spyware/adware removal services before testing because this program affects Civilization4.exe the way these "wares" usually do. It does not affect executable itself, it changes only its image loaded into memory.

It will work only under 2K/XP.

 

[Chaotic42]

I ran a test of this myself. Over the course of 6 hours, my memory usage stayed the same.

I didn't take any turns in that time, because that would have changed things. If it was a leak, it would happen as long as the game was running, right?

 

[Chopperhead]

Well you guys would probably be the ones who could figure this out.

When I first started playing the game when it first came out think I got it on the 27th it ran fine now that I have been playing for awile the game is so slow it barley moves. It takes like 20min just to loas my save game which is only in the Industrial right now but it took just as long in previous persiods as well and take around 10min in between turn and longer if an AI trys to contact you. when you scroll around the map it's like a slide show. It has also started to freeze often but it doesnt crash to desktop just locks up and I have to manually shut down the computer. I have defraged, and ran the disk cleanup, delted old programs that I didnt use any more changed the grapic settings to medium shut off some of the features, I have restarted the computer, I have completly shut it down and left it for awile nothing seems to help at all still just as slow. It's at the point where it's unplayable now. earlier today it took over an hour to load and play 2 turns and I just dont have the patience for this. I noticed that it was slowly getting slower and slower. It's almost just as slow even in the begging of a new game so I have no Idea whats happening Im not computer savy I figured it has something to do with the memory leak people are talking about although I dont really know what that means but they say that when you restart the computer it should reset it self well it doesnt seem to be happening here. oh ya I dont have any other programs running in the background besides virus protection and all that stuff. My computer also sounds like it's about to blow up lol but seriously it doesnt sound good. My confusion comes from the fact that it used to run beautifully but for some reason it just got slower and slower untill now It's just so slow It's not possible to play.

so does anyone know what this is all about? will a patch be able to fix it? plz help whats happening here?

 

[Harkonnen]

My observations show that it renders a lot when it sees buildings, forest or cotteges. In my savegame (see other thread) I have very few units, still the ocean is the fastest scrollable part of the globe.

It is not a memory leak, this game just eats too much memory for simple things. The good news is that it does not do so deliberately (e.g. badly coded), it looks like some conceptual mistake in one or two places, so it can be fixed by Firaxis or by me or by someone else in reasonable time (e.g. tonight ).

 

[aneeshm]

Lol - I meant 256 M, of course.
That said, back in the good old days someone managed to run a chess program on a pocket calculator. Today it takes 100 K to say "Hello world."

How can anything compete with NetBSD on a toaster ?


And I'm not joking - it actually has been done .

 

[phalzyr]

I ran a test of this myself. Over the course of 6 hours, my memory usage stayed the same.

I didn't take any turns in that time, because that would have changed things. If it was a leak, it would happen as long as the game was running, right?

Not neccessarily, it depends where the memory leak is. Like I said I haven't encountered one myself. Just major memory mis management.

That is good news harkonnen, maybe the patch will be out sonner then Though I hope they get the only bug I encountered fixed in it...I only had performance issues when I made a super huge map...huge is a bit slow but nothing annoying...

Maybe this weekend I will have time to chart my mem usage so those that still think it is a memory leak can at least see it isn't happenning to everyone...

 

[I_batman]

Well you guys would probably be the ones who could figure this out.

When I first started playing the game when it first came out think I got it on the 27th it ran fine now that I have been playing for awile the game is so slow it barley moves. It takes like 20min just to loas my save game which is only in the Industrial right now but it took just as long in previous persiods as well and take around 10min in between turn and longer if an AI trys to contact you. when you scroll around the map it's like a slide show. It has also started to freeze often but it doesnt crash to desktop just locks up and I have to manually shut down the computer. I have defraged, and ran the disk cleanup, delted old programs that I didnt use any more changed the grapic settings to medium shut off some of the features, I have restarted the computer, I have completly shut it down and left it for awile nothing seems to help at all still just as slow. It's at the point where it's unplayable now. earlier today it took over an hour to load and play 2 turns and I just dont have the patience for this. I noticed that it was slowly getting slower and slower. It's almost just as slow even in the begging of a new game so I have no Idea whats happening Im not computer savy I figured it has something to do with the memory leak people are talking about although I dont really know what that means but they say that when you restart the computer it should reset it self well it doesnt seem to be happening here. oh ya I dont have any other programs running in the background besides virus protection and all that stuff. My computer also sounds like it's about to blow up lol but seriously it doesnt sound good. My confusion comes from the fact that it used to run beautifully but for some reason it just got slower and slower untill now It's just so slow It's not possible to play.

so does anyone know what this is all about? will a patch be able to fix it? plz help whats happening here?

When you machine sounds like it is ready to die, is your HD going nuts, making all the noise?
I mean, during that huge load time, is your HD being accessed for that entire time?

 

[tegilbor]

Not neccessarily, it depends where the memory leak is. Like I said I haven't encountered one myself. Just major memory mis management.

Which is bad enough.

Maybe this weekend I will have time to chart my mem usage so those that still think it is a memory leak can at least see it isn't happenning to everyone...

But it seems to be happening to some - there was a post by someone a while ago who claimed that Civ's memory usage increased by 2MB (if I recall correctly) in every turn. Of course, that could be because the game is adding additional objects (e.g. units being built) in every turn, but 2MB would be quite a lot for one single turn. Also, at some point late in the game, the memory usage should remain fairly stable, because you don't just build an infinite number of units - at some point, you go to war and the total amount of units in the game decreases again. It seems to me like there may be a memory leak that is triggered only in very specific situations (which is a rather obvious assumption, because otherwise the developers would have found and eliminated it much earlier).

 

[ChuckLe]

@Harkonnen,

Hi again mate.

I tried your fixed .exe but I didn't notice any difference here I'm afraid sorry.

Used your AD 1961 save, and Venger's AD 504 save for testing.

The VM used was still 1.10 Gb or so. As you know I'm using 1Gb RAM here btw.

I wonder if you noticed something odd. The Civ 4 .exe used about 600 Mb or so for itself and looking in the list of processes the total there was about 50 Mb or so. So, 650 Mb total. However, when you look at the performance tab, the amount of 'Physical Memory' available was just a few Mb only. So, I'm not sure where the rest of the RAM is being used? Funnily enough, on a smaller save where the Civ IV.exe was using about 200 Mb, the amount of RAM available diminishes down to a fraction of the total also!

Anyway, the overall result was that the game is still completely unplayable with those saved games. Sorry.

@Chopperhead,

Very important. Try turning your virus scanner OFF!

I'm willing to bet you'll find a 10 fold speed-up in loading times.

 

[Control Group]

One would have thought that it's virtually impossible to have a memory leak in C++, but apparently one would be wrong

??

#include ostream
#include someClass

void someFn()
{
someClass* classInstance = new someClass;
}

int main()
{
someFn();
cout << "I've just leaked memory";
}

 

[mrgenie]

http://forums.civfanatics.com/showthread.php?t=138932

 

[Harkonnen]

Control Group
I think he meant that one would use vector<>, list<> or own templates instead. Actually, there are no 'new' statement used anywhere in the code I write at work... Whenever I need a heap access to object (i.e. I don't care when it dies), I use ref-counter pointer wrapper for that, i.e. garbage-collection-on-demand...

I have made some more research. The problem is actually memory fragmentation which is no surprise with python. It has no stack to hold objects, instead it uses 'malloc' for everything.

What I observe now being deeply within game process, is that there are about 300Mb of total malloc allocations, while system HeapAlloc calls sum up to about 600Mb. I think we face checker pattern of free-busy blocks, so anything bigger than 12 bytes eats more memory. And since windows paging is 4Kb, all that unused space is cached back and forth...

Currently I am writing personal memory manager targeted towards small objects in favor of allocation speed. That's my last hope...

 

[I_batman]

Control Group
I have made some more research. The problem is actually memory fragmentation which is no surprise with python. It has no stack to hold objects, instead it uses 'malloc' for everything.

What I observe now being deeply within game process, is that there are about 300Mb of total malloc allocations, while system HeapAlloc calls sum up to about 600Mb. I think we face checker pattern of free-busy blocks, so anything bigger than 12 bytes eats more memory. And since windows paging is 4Kb, all that unused space is cached back and forth...

Currently I am writing personal memory manager targeted towards small objects in favor of allocation speed. That's my last hope...

So I guess the question is if you can't figure out a memory manager , do you think the Firaxis guys will be able to? Would they have access to data/code you don't?
And this brings back the question I asked a couple days ago.
If this memory allocation problem is inherent with Python (can't utilize a stack), can this problem be fixed by re-writing the app in a language that does manage a stack. Or would this be a monster job that would have ripple effects throughout the entire core engine?

 

[Control Group]

I think he meant that one would use vector<>, list<> or own templates instead. Actually, there are no 'new' statement used anywhere in the code I write at work... Whenever I need a heap access to object (i.e. I don't care when it dies), I use ref-counter pointer wrapper for that, i.e. garbage-collection-on-demand...

Absolutely. I was just pointing out that, of the two, it's easier to write a memory leak into C++ than it is Python. In C++, you have to make sure you use appropriate extensions to the language to take care of GC, while Python does it natively (or so I believe; I haven't actually used Python, so I could be convinced I was wrong). All I meant was that if we're comparing which language is more inherently prone to memory leaks, it has to be C++. C++ is more prone to all sorts of low-level errors, since it's really quite close to ASM.

I think we face checker pattern of free-busy blocks, so anything bigger than 12 bytes eats more memory. And since windows paging is 4Kb, all that unused space is cached back and forth...

If that's actually what's going on (and you come across as competent, so I suspect it is), that is seriously remedial. If that's not an unavoidable consequence of implementing in Python, somebody should be fired*. If it is an unavoidable consequence of implementing in Python, then easy modding isn't worth the design decision.

*Whether that somebody is a developer who was too lazy to do it right or a marketroid who rushed the title too much to allow for better is a question I can't answer.