Memory Leak?

[Harkonnen]

No. I'm patching it via python, and I have downloaded 2.4.2 for this purpose, and it didn't help on itself. Also, Python eats 7-22Mb of entire memory in hugest case, so it is not the source of problems.

I have finally tracked down source of problems, and already saved ~50Mb, gonna save ~200Mb more later today, making it playable with my 512Mb config

 

[Harkonnen]

Also I tried new binkw32.dll (was hard to find since newest release links statically) and new mss32.dll (Miles Sound System) - no effect.

Well, with knowledge of what affects it, I can assure that no DLL will fix the problem (except Firaxis DLL/EXE). I'd like not to be more specific unless I release my own patch.

All of fatal problems (CTD, visual bugs) - are due to overloaded memory usage and/or address space usage. Increasing swap file size to fixed ~2Gb should help.

BTW, that was the reason of Art0.FPK solution. They file-map it when it exists, so it eats about 300Mb of address space (not memory) all the time, and this caused some ATi drivers to CTD due to lack of free address space.

 

[Harkonnen]

The last word - kind of data which consumes roughly 250Mb should never consume so much, neither its size should grow with amount of physical RAM.

So it is definitely a bug, and with either my patch (less likely since this bug originates in civ4.exe) or the one from Firaxis the problem will be solved. Civ4 is not resource hog by-design, phew...

 

[pygbe]

Go on, Harkonnen, we are with you!



Keep up the good work.

 

[CyberChrist]

@Harkonnen: If Firaxis/2K have any sense then a proposal to join their ranks should already be in the mail - along with a fairly large motivational check

 

[Fallblau]

That's some righteous debug work, Harkonnen... You are to be congratulated. Count me in as a willing tester should you manage to nail this thing down...

 

[I_batman]

Thanks!

If you contact him, please tell him that his writing style is very accessable..a non-programmer such as myself (I am system engineer/sdmin type) was able to comprehend it easily. I found it very helpful - now I really know what Harkonnen and you and others are talking about when they discuss mallocs and allocs and such...big thumbs up!

OK, closing the loop on my email with Evan Jones.
Nothing helpful here, but he echos what Harkonnen is saying.

Following is the email exchange:

On Nov 11, 2005, at 19:51, Paul Batte wrote:
> Do you know if this issue was ever addressed in later releases of
> Python?

My patch has not yet been committed to any branch of Python. However,
there has been interest in committing it, but I just don't have the
time or energy to try and "push" the patch through the Python developer
community. Since it touches a VERY critical section of the Python
interpreter (memory allocation) it needs very careful review and
testing.

Additionally, it probably no longer applies cleanly against the
development branch.

> The reason I am asking is like you I am a computer games addict,
> though really addicted to one game, Civilization.
> Civ IV came out about 2 weeks ago, and the bulk of the game was
> written in Python.
> The game appears to have some severe memory problems.

Hmmm! Interesting. The chances are that my patch would not solve the
problem. My patch only solves the problem IF the Python program
allocates a LOT of objects and then frees them, and never uses them
again. I would think that Civ 4 is creating a lot of objects and
keeping them around, or is creating and freeing a lot of objects fairly
quickly.

> BTW, when did you write that paper?

At the end of 2004.

> Oh, and it is nice to see that someone from my old alma mater is being
> noticed on a games web site.

It is nice to get noticed on any web site! That Python memory
management article is the most read thing I have written.

If you have any further questions, let me know.

 

[I_batman]

The last word - kind of data which consumes roughly 250Mb should never consume so much, neither its size should grow with amount of physical RAM.

So it is definitely a bug, and with either my patch (less likely since this bug originates in civ4.exe) or the one from Firaxis the problem will be solved. Civ4 is not resource hog by-design, phew...

Harkonnen, do you think the civ4.exe was built in C++?

Just doing some musing.
IF Firaxis does not fix this, (I have read other threads indicating that some Firaxian's (Sirian) don't think this is a problem), I am wondering if the SDK will allow "fan" programmers to access the code for civ4.exe. I imagine that this will be core code not made acccessible by Firaxis, but one can hope. It would mean they have pretty much open-sourced the entire game if they do.

 

[oldStatesman]

Great Job!

Also I tried new binkw32.dll (was hard to find since newest release links statically) and new mss32.dll (Miles Sound System) - no effect.

Well, with knowledge of what affects it, I can assure that no DLL will fix the problem (except Firaxis DLL/EXE). I'd like not to be more specific unless I release my own patch.

All of fatal problems (CTD, visual bugs) - are due to overloaded memory usage and/or address space usage. Increasing swap file size to fixed ~2Gb should help.

BTW, that was the reason of Art0.FPK solution. They file-map it when it exists, so it eats about 300Mb of address space (not memory) all the time, and this caused some ATi drivers to CTD due to lack of free address space.

The last word - kind of data which consumes roughly 250Mb should never consume so much, neither its size should grow with amount of physical RAM.

So it is definitely a bug, and with either my patch (less likely since this bug originates in civ4.exe) or the one from Firaxis the problem will be solved. Civ4 is not resource hog by-design, phew...

Glad to see this is a BUG...of coarse, you may be getting a lot of mail from fanbois disputing this still...lol.

But since it is a bug, it IS fixable, and most all of us should eventually be able to play the game without fear of BSOD's and CTD's due to address space limitations. If 'the complainers' had all not kept banging on this point, and simply kept quiet, it possibly may never have been addressed.

Thank you for being proactive, and for your hard work.

 

[Bast]

Well done, Harkonnen!

 

[DrewBledsoe]

Also I tried new binkw32.dll (was hard to find since newest release links statically) and new mss32.dll (Miles Sound System) - no effect.

Well, with knowledge of what affects it, I can assure that no DLL will fix the problem (except Firaxis DLL/EXE). I'd like not to be more specific unless I release my own patch.

All of fatal problems (CTD, visual bugs) - are due to overloaded memory usage and/or address space usage. Increasing swap file size to fixed ~2Gb should help.

BTW, that was the reason of Art0.FPK solution. They file-map it when it exists, so it eats about 300Mb of address space (not memory) all the time, and this caused some ATi drivers to CTD due to lack of free address space.

Firstly Hark your a star for all your effort

Secondly Ive been playing non stop for the last 2 weeks since I got game , and I mean non stop, as I'm off work long time atm. I increased my VM to 2Gb 4 days ago (must be reading your mind) , and havent had any crashes since, and slowdown is overall less.

I play on large maps with random civ and all random map settings.

Some points which may help:-

1) Slowdown is firstly most evident after you discover astronomy and map trading(which generally comes with more opening and closing of diplomacy screens).

2) Sometimes this slowdown stays , in which case I save game, quit out and then reload, this ALMOST ALWAYS cures the problem.

3) If it doesnt I reboot and then reload, which always works.

4)Your right about the "land" causing game lag, on maps with a lot of water its noticably less.

5) Advisor screens are not properly being cleared from memory , after loading them.


Just some observations, hope they may be of some help...

Keep up the good work..your doing a hell of a job


p.s. Didn't state my system :- P4 3.4gz, 512mb DDR Ram, 128mb pci graphics

 

[warpstorm]

Enigma, large memory usage does not mean memory leak. This game does have large memory requirements. (I do not know if there is or isn't a memroy leak although I suspect tehre is)

 

[markcogan]

First -- mammoth kudos to Harkonnen for digging into this.

Hark, I just want to be sure I'm clear on what you've found. You say that python allocates no more than 22MB of memory? Does that meen that all of the "game state" data -- map, until locations, cities, research, culture, borders, alliances, etc -- is that small? That's roughly what I expected ... which means the remaining 95-98% of the memory usage -- including the cpu-chugging memory allocations -- is for handling the graphics and UI, correct?

People keep claiming that Civ IV is a CPU/memory hog, but there's really no reason for it to be; the actual game itself uses only a tiny fraction of the memory allocated, and any modern CPU should be able to do thousands (or more) pathfinding operations without any perceptible delay from the user. And aside from pathfinding and other AI operations, updating the game state between turns is totally deterministic and linear; it should be very quick.

 

[oldStatesman]

Enigma, large memory usage does not mean memory leak. This game does have large memory requirements. (I do not know if there is or isn't a memroy leak although I suspect tehre is)

Agreed, by itself, using a lot of memory is quite okay...as long as that memory is managed by the application properly. Memory is meant to be used.

Probably not a memory leak in the classic sense...more like a 'soft' leak...and if the engine is not releasing/managing used address space properly, as Harkonnen seems to have found, this could be called a 'soft' type leak.

That is why it may be so difficult to find, (would have been tough to catch even in Beta)...a hard leak would be readily apparent, while a soft leak may only be noticable in certain conditions. The inability of the ATI drivers to work properly on some configutrations, but not all configurations, is an example.

Good news is that this behavior looks like a 'bug' ...and not by design. This is a huge relief for me, as it means that it is patchable.

 

[CalvinHobbes]

this might help, http://www.gamespot.com/pc/strategy/civilizationiv/download.html?sid=6139723

Is apparently an official patch. I may be wrong however

 

[inertiadriftsc]

Holy @#@#$@#$$#@#$&!!! Did anyone opened WorldBuilder savegame in notepad???

uhmmmm of course its long, it has to have all the coordinates of everything in a civ 4 map. Sorry that was pointless, it has nothing to do with a memory leak or any other problems. The only thing I have noticed is that the game's graphics engine slowly slows down, until its very jerky then I Alt-tab it and its fine again. And if your complaining about peoples misconceptions of virtual memory, explain it, dont act like a butt and put on a haughty superior, im educated attitude, its just annoying and doesnt accomplish anything. I dont believe Civ 4 has a memory leak, it is just memory intensive and is built to utilize as much memory as is available

 

[oldStatesman]

this might help, http://www.gamespot.com/pc/strategy/civilizationiv/download.html?sid=6139723

Is apparently an official patch. I may be wrong however

This is a patch for those who have downloaded the game over the internet via D2D. Not for the retail box version.

 

[AsnoT]

this might help, http://www.gamespot.com/pc/strategy/civilizationiv/download.html?sid=6139723

Is apparently an official patch. I may be wrong however

that's for the version you can download from Direct2Drive (Civilization IV D2D Hotfix), NOT the DVD version sold in shops.

 

[NKVD]

I dont know but I have 512 MB of ram 1.2 Ghz duron and a 64 MB geForce 4 and the game is not smooth when it cycles through units but I put all the graphic to low and the game runs fine and crashed only once on a big map at the end of a game where there was a lot of cities. and I know I've spent like 6 hours playing to civ 4 without getting anything bad. I once read a thread to let the OS manage the virtual memory in the options of the system instead of the numbers decided by the programmers and it worked for me...

 

[oldStatesman]

uhmmmm of course its long, it has to have all the coordinates of everything in a civ 4 map. Sorry that was pointless, it has nothing to do with a memory leak or any other problems. The only thing I have noticed is that the game's graphics engine slowly slows down, until its very jerky then I Alt-tab it and its fine again. And if your complaining about peoples misconceptions of virtual memory, explain it, dont act like a butt and put on a haughty superior, im educated attitude, its just annoying and doesnt accomplish anything. I dont believe Civ 4 has a memory leak, it is just memory intensive and is built to utilize as much memory as is available

Did you read this whole thread?????

Harkonnen does not just state his beliefs...he and others here have been constructive and worked hard over several weeks to try and solve an issue a lot of us have. You don't "believe"...well, how about backing that up with some factual discussion...opinion is not enough.

Sheesh.