Memory Leak?

[Harkonnen]

As I observe, the more memory you have, the more civ4 consumes (with my 512Mb it stops devouring memory at ~600Mb). It does not look like a memory leak, but rather like some wrong expectation about when it will swap. I think this is garbage collector issue with Python, probably due to its *nix-oriented expectations. I tried to play with GC in game code, but it didn't help (still I am sure, lines I added were actually called).

BTW, Firaxis doesn't control garbage collector AT all, for two reasons:
1) There are no 'import gc as ...' statement anywhere in the code
2) 'gc' is GlobalContext variable in every module (game state), should they've known about garbage collector module name, they would name this variable differently. That's why 'as ...' required in (1).

Executable is actually packed, and its original size seems to be ~40Mb, and it seems that EVERYTHING is python-coded, even graphical engine. No wonder it's so sluggish... Python is the most uncompileable languags since every identifier is resolved by the time it is used. The only thing it can do is to turn strings to their hash values, but still this is a lot of overhead. Not to mention memory overhead which is primary issue with this game.

I don't know why Firaxis went Python. They had their code working in simplest C (if not ASM) in civ1, and I guess C++ in civ3, so they had codebase to continue civ4 same way.

I don't think either that they try to market Python this way. I just see no reason for commerce-oriented company (Firaxis) to market free-of-charge module (Python). Probably they wanted to make modding easier... that can be the only explanation, but IMO the cost is too high.

 

[tegilbor]

it seems that EVERYTHING is python-coded, even graphical engine.

No wonder it's so sluggish...

Damn straight!

 

[ledhed]

Look, there's definately a major memory leak:

Why so many things running ? I tried to right click on that with my mouse to turn them off .. Cripes I thought I was bad with 33 .

 

[Lmtoops]

My game was running fine until the Sistine Chapel movie started to run.(I'm not saying the Sistine Chapel is the cause) Suddenly, it slows to a crawl and then dies. I checked my virtual memory, and my peak usage of the virtual memory (paging file) was 93%. So I bumped my virtual memory, but the same error occured (peak usage at ~60%). In fact, the same error occurs every time. I cannot get past the wonder movie.

I holding out hope that the patch will address the issue.

 

[xguild]

Look, there's definately a major memory leak:

Seems like a lot of people are having trouble understanding what a memory leak is, the fact is that it's more then one thing which is why it's confusing.

For starters a memory leak is an application failing to appropriatly use and release memory at the appropriate (or intended) times within the program. As such memory (and virtual memory which is seperate from hard memory) can misplaced, misused or are marked "in use" by the application ensuring that no other applications or even the application that caused the problem know it is available. (The screen shot showing the excessive memory use clearly shows that the application fails to release memory and is obviously miss using it.) The result is usually slowness with an eventual crash. Having a lot of hard memory or increasing your virtual memory size can prolong this and even prevent you from crashing but it doesn't change the fact that clearly their is a memory leak.

Some other characteristics of memory leaks is a slew of "unidentifiable" bugs. Memory leaks tend to create bugs that aren't really their, sort of like ghost bugs that are only appearing because the application fails to allocate appropriate resources. Graphics glitches, unexplained errors, random crashing or predictable crashing in the same exact spot might all be fixed when the memory leak is corrected.

Memory leaks can also have significant impact on your PC. Memory leaks for example can make your anti-virus software believe your PC is infected, falsly discovering and quarantining files. The result can be the quarantining of a system file your operating system needs in order to function (Hence why some people have hard crashes only fixed by re-installing their operating system).

 

[Harkonnen]

Holy @#@#$@#$$#@#$&!!! Did anyone opened WorldBuilder savegame in notepad???

 

[phalzyr]

Actually I think people who post things that are completly and utterly wrong (like "Virtual memory is disk space, not memory.") aggrevate the situation because people who don't know whats going on believe it and suddently you have half a dozent memory leak threads where everyone blindly joins the crusade started by a few ppl who posted a few technical terms without knowing what they mean.

Actually in essence his statement was correct though he could of stated it better... how about this. Virtual Memory is actually a temporary file stored on the hard drive, not physical memory.

And Yes, if you accept windows default setting to manage its own VM then defragging and such could help improve its speed, due to less fragmentation.

And I didn't patronize anyone, I pointed out some wrong info and I won't argue any points about who is qualified to draw conclusions or who isn't because online, everyone suddently turns into an expert for everything... so you can either believe me or not.

And I still won't preach the inner workings of PCs here, I gave up doing that online quite a while ago.
But Harkonnen seems like he is still enthusiastic enough to explain some of the stuff in detail (And he seems to know that he is talking about)

If you are not going to bother saying why something is wrong (though it wasn't wrong just worded in a misleading way), then don't bother saying it is wrong. that is.....

Just remember: memory leaks don't cause slowdowns and having more main memory doesn't really help either (at least on the average windows system).

They dont?!? IT doesn't?!? Looks like you need to study some more. The lack of avaiable memory or not having enough physical memory causes the swap file (Virtual memory) to be used, and the Hard drive's access is much slower than physical memory. Thus if all your main resources are spent on unreleased memory then you ahve to use the slow swap file for memory instead... Having more memory helps in that it takes longer before the effects are noticed, because more memory has to be wasted before goign to the swap file.

On another note: Having a map take up 1GB of memory doesn't indicate memory leak, just a memory hog. Take one of the examples above, where he went into a diplomancy screen and it now took more memory and wasn't released after leaving the screen. That ISN'T A memory leak but poor memory management if your system is Memory limited, if not memory limited then it saves hard drive access next time you go to screen. IF it took even more memory the next time that screen was opened and consecutive times it was opened then it'd be a memory leak.

Personally I looked at ram before starting a game, play for hours then got back to main menu, and they mached pretty closely. Also a good way to see if there is a memory leak is to play a few hours or until slowness, save game, check memory usage then get completely out of game and reload. IF there is a memory leak the amount of memory will be greatly different. Of course take into consideration of any screens you opened like the diplomancy screen that might only be put into memory AFTER the first time viewing them. I tried this and came out with almost exact memory usage after playing for a few hours...

So if there is a memory leak it is system specific, because I don't get the issue myself, Or possibly with a feature of game I don't use...

This game seems to be a big memory hog. I created a custom sized map that is about the size of a normal huge, and just to create that map and in game it took most of my 1GB of memory. A large map took 700MB... If you only have 512MB of memory that is going to force the program to use the much slower hard drive swap file/virtual memory... I've only noticed slowness on huge maps myself and then not enough to bother me...

 

[Harkonnen]

You're both are talking about the same thing, but virtual memory is the whole 32-bit address space which can be mapped onto either physical RAM or onto disk - OS decides.

Memory leaking is tricky issue with python. It's more to loop-references with garbage collector, then to forgotten 'free' call. And it's harder to find, especially being in tight relations with foreign C libs.

 

[chris.giddins]

So guys, should I go and buy more memory and a beefed up graphics card, or should I wait for the patch. My machine handles it fine....at first.

 

[aneeshm]

There is no memory leak, as far as I can see. The game runs an entire day on my 256K RAM.

Wow . . . . just wow .

How did you manage to get even Windows to run using only 256 K RAM ? Even Civ 2 cannot run with so little memory .

 

[I_batman]

Memory leaking is tricky issue with python. It's more to loop-references with garbage collector, then to forgotten 'free' call. And it's harder to find, especially being in tight relations with foreign C libs.

This may be an idiotic question, but here goes.
Once the SDK is released next year, I am wondering if there will be a possibility of taking some of the Python scripts and re-writing them into C, where they can be compiled, instead of interpreted in the current system.

I am wondering if this will help speed up the game. And this may also help catch any memory release issues that Firaxis can't fix.

Is it worth even considering? The reason I ask is it will likely take months of examination of the Python code to see how this stuff really ties into the C libraries, which are still a black box to anybody until the SDK is released.
And even then, the libraries may still be black boxes if Firaxis deems the C code something they don't want to share.

It has been a long time since I was coding in C, but I am started into the Python this week. Sort of reminds me of my Perl coding days, which was another interpreted language, which are all pigs compared to a complied executable.

 

[phalzyr]

You're both are talking about the same thing, but virtual memory is the whole 32-bit address space which can be mapped onto either physical RAM or onto disk - OS decides.

Yes we are, it can be in either but when mapped to Physical memory it is called.... Oh well lets not get into a computer term warfare LOL!

I was primarily saying that he shouldn't complain about others giving false info if he gives just as much or more misleading info himself...Not to mention saying info is false when it isn't....

I haven't encountered a memory leak with this game YET, but I'll keep my eye open for one. Hopefully I don't have any trouble though... This is a three day weekend coming up

 

[Sevenhertz]

So guys, should I go and buy more memory and a beefed up graphics card, or should I wait for the patch. My machine handles it fine....at first.

You should wait for the patch Chris. I have run nearly every hi-graphics spec game on my pc with settings for visuals at the max with no problems, and Civ4 is the only game to mess up big. We are all having the same problems...

 

[Harkonnen]

No matter, I might be wrong too... This is terms war, not meaning. And what Windows calls page file can be named virtual memory in Linux, and vice versa...

What is different, is that one of you speaks about comp. terms, and another speaks about attitude and talking in the forum, that's also true Just sign a peace treaty and demand his PC if it's not colored red.

I_batman
Actually I've started studying Python today from python.org because of three things:
1) I waited for this game too long
2) I consider myself as skilled enough programmer (heck, I coded some asm when civ1 came into the world).
3) I think I know what's the problem with this game.

The fact is: THE LESS MEMORY YOU HAVE, THE BETTER

It tries to accomodate its needs to amount of physical RAM you have. Normally programs don't do that. When you have 1Gb ram, it uses up to 1.2-1.3Gb whole memory (inevitably swapped). With my 512Mb it is 620mb of mem. 2gb folks mostly catch CTD because 32bit architecture can't grow any further (2^32 = 4GB with high 2GB being devoted to kernel, drivers, etc...)

It actually calls WinAPI GlobalMemoryStatusEx. I don't know any other documented way to find amount of physical RAM (except GlobalMemory which it does not call). So that should be the narrow point where I can trick it.

Actually I have tricked it (it was damn hard API hook since it can be called from DllMain, NTDLL was in effect and a lot of lowest-level coding and googling). I did it, but... it didn't help. I am still on the issue. Perhaps, that dang ~e0005.exe removes my hook and constantly watches after civ4.exe, so that no **** affects it.... I'll know after this night.

I have downloaded python source and saw that it's not it making these calls. Something within civ make it. That thing tried to adjust itself for amuount of physical RAM, but actually it always overpasses it. I.e. it allocates not 50% as it would expect, but 120% always.

Maybe digit CD-Protection from Alchohol 120% messed up with mem-percentage, so it's 120%, I dunno. Actually, this was a joke, but I'm too mad with it not to finish. So, expect news...

P.S:
Python allows creating a single executable and all of this is well-documented in python docs. It's called freezer. I think that civ4.exe is actually such freezed application with about 40Mb of raw python code in it.

The problem is that that exe is packed, debugging-protected and I doubt there is byte-to-source-code decoder for it exists.

Still, you may play with some of game logic (e.g. with camera) by performing ATi step about unpacking Art0.FPK file. As far as I remember, Assets\Python folder appeared only after that. In that folder camera and screen-input controls are directly used by game engine. When I wrote 'import sys ... sys.exit(0)' upon camera movememnt, it did exit when I moved a camera, so it works... Though my efforts to affect python garbage collector still fail and fail.

I wonder if Firaxis coders came to what I've came without any original source code...

 

[phybre]

Actually I think people who post things that are completly and utterly wrong (like "Virtual memory is disk space, not memory.") aggrevate the situation because people who don't know whats going on believe it and suddently you have half a dozent memory leak threads where everyone blindly joins the crusade started by a few ppl who posted a few technical terms without knowing what they mean.

I don't know what your problem is, or why you're posting to this thread, but virtual memory is disk space. It shouldn't take however many semesters of CS for you to learn that. It is storage memory being used to emulate volatile memory, and is inherently slower and less useful. That's simple physics, not CS. It takes longer to access a spinning disk than it does a solid-state memory module.

If you have nothing useful to say, this is not a great place for you to be sophos moros about knowing what's wrong with the game. People are having problems. A disproportionate number of people, compared to other game releases. Can you fix them? Do you have any suggestions for fixing them? Do you have problems of your own to report? If you can't answer yes to these questions, the Bug Reports area isn't for you.


To return to the topic, I can verify that on my system at least, Civ does not return the memory when it crashes to desktop. So it's a particularly non-graceful crash. Which explains why windows doesn't even get an error condition to pop up. And the longer the game runs, the slower it gets. More evidence for a memory management issue. Its memory footprint should be more consistent than it is.

At least one person has also reported that running Civ causes other, previously installed programs to spawn. So it might very well be that Civ has that type of memory leak, too. It could be making unwarranted assumptions about the state of the memory and is spilling over into other memory spaces.

The real point here is that we, as consumers, shouldn't have to wonder. The developers have still not released any statement that I can find on their website, and their support section still has nothing about most of these problems. The fact that this game was released in such an unstable state can only be ascribed to pre-holiday profiteering, and it's despicable. I didn't pay to beta test a game. I paid partially for the assurance that it was already properly tested and the quality was assured. I am disappointed that this is not the case, and it will affect my future purchasing decisions significantly.

 

[Harkonnen]

Other programs are probably crashing due to overused swap file, it affects everyone in the system. Even drivers might not expect such situation just because it's too rare to happen, too hard to handle, and pain in the a** to test.

 

[chris.giddins]

Has there been any response from firaxis yet?

 

[Ribannah]

Wow . . . . just wow .

How did you manage to get even Windows to run using only 256 K RAM ? Even Civ 2 cannot run with so little memory .

Lol - I meant 256 M, of course.
That said, back in the good old days someone managed to run a chess program on a pocket calculator. Today it takes 100 K to say "Hello world."

 

[I_batman]

I_batman
Actually I've started studying Python today from python.org because of three things:
1) I waited for this game too long
2) I consider myself as skilled enough programmer (heck, I coded some asm when civ1 came into the world).
3) I think I know what's the problem with this game.

The fact is: THE LESS MEMORY YOU HAVE, THE BETTER

It tries to accomodate its needs to amount of physical RAM you have. Normally programs don't do that. When you have 1Gb ram, it uses up to 1.2-1.3Gb whole memory (inevitably swapped). With my 512Mb it is 620mb of mem. 2gb folks mostly catch CTD because 32bit architecture can't grow any further (2^32 = 4GB with high 2GB being devoted to kernel, drivers, etc...)

It actually calls WinAPI GlobalMemoryStatusEx. I don't know any other documented way to find amount of physical RAM (except GlobalMemory which it does not call). So that should be the narrow point where I can trick it.

Actually I have tricked it (it was damn hard API hook since it can be called from DllMain, NTDLL was in effect and a lot of lowest-level coding and googling). I did it, but... it didn't help. I am still on the issue. Perhaps, that dang ~e0005.exe removes my hook and constantly watches after civ4.exe, so that no **** affects it.... I'll know after this night.

I have downloaded python source and saw that it's not it making these calls. Something within civ make it. That thing tried to adjust itself for amuount of physical RAM, but actually it always overpasses it. I.e. it allocates not 50% as it would expect, but 120% always.

Maybe digit CD-Protection from Alchohol 120% messed up with mem-percentage, so it's 120%, I dunno. Actually, this was a joke, but I'm too mad with it not to finish. So, expect news...

P.S:
Python allows creating a single executable and all of this is well-documented in python docs. It's called freezer. I think that civ4.exe is actually such freezed application with about 40Mb of raw python code in it.

The problem is that that exe is packed, debugging-protected and I doubt there is byte-to-source-code decoder for it exists.

Still, you may play with some of game logic (e.g. with camera) by performing ATi step about unpacking Art0.FPK file. As far as I remember, Assets\Python folder appeared only after that. In that folder camera and screen-input controls are directly used by game engine. When I wrote 'import sys ... sys.exit(0)' upon camera movememnt, it did exit when I moved a camera, so it works... Though my efforts to affect python garbage collector still fail and fail.

I wonder if Firaxis coders came to what I've came without any original source code...

OK, Harkonnen.
Your coding skill levels are way deeper than mine, but a couple things you said I want to try understand.

1. Are you saying that Civ IV code automatically examines a system for its RAM, then wants to create a memory block made up of actual RAM and virtual memory on the HD that equals 120% of RAM?
Am I correct that is what you are saying?
An example: Someone has 1gig of memory. Civ IV fires up and says "I want to use 1.2 gig, since 1 gig is the total memory on this machine."
But the OS is using say, 300 MB, and other programs using another 100 Mb.
That means Civ IV would then have 600 MB available in memory, and would start swapping in the HD for another 600 MB.
Would that be an accurate scenario, based on what you have seen?
If true, this sounds like a truly stupid design.

2. Why in the coding gods' names would the Civ IV programmers build the core engine in Python as opposed to something like C, or C++. What advantage would this give them? To me it sounds like madness.

3. Are you thinking of building some app that will be called within WinAPI GlobalMemoryStatusEx, that will fake out Civ IV into believing there is less memory on the machine and therefore the swap file issue goes away? I think I am reading above that you are going to try that.

 

[Harkonnen]

I_batman
1. Yes, you are right. First it reaches ~200Mb free physical memory and starts to swap out other running processes since they were least recently used (LRU caching). Then when it reaches 300-400Mb of physical memory, there are no other processes to swap out, and it starts swapping out itself, i.e. what it loaded a few seconds (minutes) ago. When you scroll to those units/terrain, they are loaded back and some other stuff is pulled back onto the disk. Advisor screens occur rarely, so they are likele to be swapped out first, thus entering them becomes a pain. Same stands for globe view (clouds and stars are new, previously unused textures when you enter it).

2. The only possible advantage is widely spread modding, but the price is a bit too high... AFAIK, I wrote this somewhere in this thread. From my programmer's point of view I see some strange thing here... It's hell to tight all those modules together, and then to debug them. C/C++ is native for everything - DirectX, their GUI module, even Python can be used for "game code" from C code, but it looks like Python is main engine core here, and C plays secondary role to access DirectX and other too low system stuff. This more looks like marketing Python (what for? it's free anyway) or like a.... should I say this... student's project about module programming (dunno which smile to put).

3. Yes. But that nasty ~e0005.exe (or alike) process is not only CD protection. It also guards main executable from penetrations like mine. So in a few seconds I see all my API hooks erased. I know ways to fight this, but this'll take time. My aim here is to bring this "patch" earlier than Firaxis brings its own. Just for the sake of my own Too many waiting, and too much effort so far...