Memory Leak?

[melatonin]

Unfortunately the ini fix didn't work for me. When trying to load a large map late in the game (~1960s) my screen goes black. Then after awhile it looks like this till I reboot:



That was taken with a digital cam since my PC is all locked up.

 

[spa]

The ini fix worked for me too. I actually quit my game after two hours of play instead of having it crash on me multiple times. Thank you so much Kangaroo.

 

[Yrth]

Kangaroo-
Thanks!

Your suggestions made the game playable for me; very glad you shared them.

 

[gunnergoz]

Well, I don't know if it's your fix, but no crashes in this game yet; large map and the year is 1870...keeping my fingers crossed.

 

[Gav]

This thread was and still is (I hope) about the serious problems with extensive memory (ab)use/slow down of CIV and NOT other bugs/settings that may cause blue screens or CTDs.

And while it is nice if the suggested INI file changes help those who have been experiencing severe blue screen and CTD issues that are not related to memory (ab)use/slow down, these suggestions/discussions really belong in another thread as they do in fact do NOTHING to help with the memory (ab)use/slow down problem - quite on the contrary.

Posting about them in this thread is IMHO straying far from the main issue of this thread - bordering on threadjacking.

i had all the symptoms described in this thread. Civ was using way too much memory. besides, the thread is titled wrong. its not a memory leak.

 

[Tormunda]

This thread was and still is (I hope) about the serious problems with extensive memory (ab)use/slow down of CIV and NOT other bugs/settings that may cause blue screens or CTDs.

And while it is nice if the suggested INI file changes help those who have been experiencing severe blue screen and CTD issues that are not related to memory (ab)use/slow down, these suggestions/discussions really belong in another thread as they do in fact do NOTHING to help with the memory (ab)use/slow down problem - quite on the contrary.

Posting about them in this thread is IMHO straying far from the main issue of this thread - bordering on threadjacking.

Hi CyberChrist.
I don't think anyone was intending to hijack the thread with the .ini fix. We are all trying to find things that solve the problems and I for one would like this .ini fix to be noted on this thread.
Why? Because things are never what they seem, as Harkonnen has already found out several times.
The .ini fix should make things worse but.... it does not.... Why?
What is it doing that solves so many of these issues?

I hope that Harkonnen will get time to see the .ini fix and take it into account when doing his hack, as it may reveal some more hidden riches!!

 

[phalzyr]

Harkonnen, I just sent you an email with my logs and a dxdiag. Just in case it gets lost in your spam you can also get it at the belwo URL.

http://209.174.48.132/downloads/hark.rar

Assuming I read the log correctly the patch cleaned up (for Harkonnen's save game) a bit over 100MB for me. Harkonnen am I reading it right or forgot a decimal point ?

 

[Harkonnen]

phalzyr
Thanks, got it. Though the patch reduced C7 usage, D3D usage remained almost the same... Firaxis seems to fix wrong things. At least there is an easy programmatic way to save 50Mb described by me in this thread

I'll explain meaning of values with the next post.

As for .ini file.

SyncInput might save from CTD. As I noted, civ4 doesn't create D3D as mutlithread-aware device, so concurrent calls to draw something might cause crashes. SyncInput forces input to stay in the renderer thread, so this might help if civ4 attempts to make concurrent D3D calls with SyncInput=0.

Anyway, in my patch I currently add D3DBEHAVIOR_MULTITHREADED, so it won't matter when I release it.

As for 'D3DQueries = 1'. I saw no effect of this thing. The game works via D3D9 anyway (no d3d8 calls made).

I am more interested in "DynamicAnimPaging" or such. This key should affect A LOT from its name, but I haven't noticed anything meaningful changing its value.

 

[Harkonnen]

Values of my logger:

civ4_mem.log:
As per DLL making a call:
VA - VirtualAlloc/VirtualFree totals (committed pages).
C7 - malloc/calloc/realloc/free totals for MSVCR71.DLL (modules compiled with VC++ 7.1) - most of modules including civ4 use these
C6 - malloc/calloc/realloc/free totals for MSVCRT.DLl (modules compiled with VC++ 6.0) - bink for example
PH - HeapAlloc/HeapRealloc/HeapFree for process heap
OH - HeapAlloc/HeapRealloc/HeapFree for other heaps
LA - LocalAlloc/LocalReAlloc/LocalFree
GA - GlobalAlloc/GlobalReAlloc/GlobalFree

negative values mean that memory was allocated before my logger took effect, but was freed later. So it detected decrement, but didn't catch increment.

There are two major devourers: civ4.exe for C7. I believe this is game state and other stuff which I can't affect since I don't know its nature.
Another one is LA of d3d9.dll. These are D3DPOOL_MANAGED resources. Textures/vertex_buffers/index_buffers - and that's my primary headache for past 2-3 weeks.

civ4_d3d.log:
list of textures, vertex buffers and index buffers with total size as per pixel format (hexadecimal D3DFORMAT, size is decimal like in civ4_mem.log).

I account only D3DPOOL_MANAGED resources. This almost totals to d3d9.dll LA in civ4_mem.log.

So far I could save ~50Mb on index buffers and 50Mb on vertex buffers. Also, those 50Mb on vertex buffers cost a lot of performance, so I'm speeding it up and releasing then.

Saving more from vertex buffers will require damn another approach, so I won't delay anymore what I have done for now. Around ~100 mb savings. Not much, but with my 512Mb system it became much more playable even after 50Mb index buffer savings.

 

[Stele]

Hi all,

Just signed up to thank Harkonnen and the rest of the 'team' for all the effort put into analysing and attempting to solve the problems Civ IV's been throwing at us. The detailed analyses by Harkonnen (and some very constructive suggestions/comments by some members, sorry I can't recall your nicks) throughout the 15-page saga are nothing short of amazing. I don't claim to know a lot of programming (I've done C and ASM, but I'm by far a hardware type, being a microelectronics engineer )... yet I've learnt a lot about Civ IV's inner workings from the programming perspective, shuttling between the informative and detailed posts of Harkonnen (and others), and online resources on Python.

I've had endless problems with CTDs, though thankfully no BSODs yet. After installing 1.09 and updating my nVidia drivers, things got even worse, at least for me - I could not play for more than 2-3 turns without a CTD, and the latter was almost always with my desktop kicked to 800x600x4-bit colour as some others here have also experienced. All needing a total reboot to clear up. Extremely frustrating and annoying beyond words.

This thread is a godsend for two reasons: first and foremost because I like to know more about the problem and the *process* of tackling it, even if it may or may not lead to an instant solution; and secondly because Kangaroo's ini suggestion worked wonders for me and I can now actually quit the game voluntarily! While I have to admit that I do agree with CyberChrist to a certain extent about keeping the thread focused, on the other hand having the .ini 'stopgap' allows people like me to have something to get civ IV working ... without lessening my interest in any way in the original thread's aim in tracking down the pesky memory issues. Although my game works great now (so far!), I'm still very keen in getting to the very root of this whole fracas and hence eagerly follow the progress of Harkonnen and 'team' It is quite an unpleasant surprise that Firaxis/2K has not responded specifically to these findings, damning as they are, and above all to release a patch which, it seems, makes things even worse for a lot of people.

And so, once again, great work Harkonnen and thanks for taking the trouble to do what you've done, and for sharing it all with the community. Keep it up!

p.s. D3D9 seems to be getting more and more a suspect eh? which probably ties in with the large number of directx renderer failures (pak issue or no pak issue). Sigh.

 

[phalzyr]

Values of my logger:
.

Thanks Though I pretty much guessed what was what, I appreciate the clarification. I look forward to your patch, though my machine runs it great I have played it little MP with my wife and choose standard maps during those sessions because of the memory issues I didn't want to risk it. Maybe I can convince her to play on a huge map after your patch Darn I didn't even think to run the test on my secondary machine also...

 

[CyberChrist]

The .ini fix should make things worse but.... it does not.... Why?
What is it doing that solves so many of these issues?

Higher levels of synchronization almost always equal increased stability, but normally also at the cost of some performance (which appears to be the case with the INI file changes suggested).

Also, reading Harkonnens recent posts, it would seem to support this and that the SyncInput INI change is in fact the only one of the 4 suggestions that actually does anything - but nothing that improves on the slowdown problem or memory (ab)use in general though.

 

[blue3c]

Nevermind, Mindless I need coffee dribble. I am removing my previous post.

 

[phalzyr]

HEy I just hit quote and got totally different text LOL good change blue3c. OH and btw if you read the whole thread you'll realize it isn't a memory leak that we are discussing.

 

[blue3c]

In the other thread I am in I tend to be a little quick tempered and spout off a little. Here I think, I should not do that. This thread is more for Helping with the mem problem. Whether its a leak or whatever. I don't think it matters. WHat I do know is Hark is doing a great job. I just sold three of my machines for testing, but I still have two p4's. Let me know if you would like me to log anything or help in anyway. I am a hardware geek and a os geek. Not so much on the programming side. Well at least since the stopped using basic. LOL

 

[Harkonnen]

I tried SyncInput not so long ago, but it didn't improve much on me. It just doesn't allow to scroll farther away until current frame is rendered, so logical reaction is faster, but framerate is as slow. Anyway, unsynced input looks better for eyes, but causing CTD outweights these "benefits"

Also I have noted that nVidia users catch CTD more oftenly. Probably because ATi synchronizes in any case, no matter that D3D initialization flag.

I'll put a version today which will put D3D in-da-sync, so you may check if 'SyncInput=0' still causes trouble with that version of my DLL.

 

[Harkonnen]

And thanks all of you for your kind words and respect

I would abandon it and continue playing quake2 online or something else after work, should community just sit and wait until that mad coder releases something useful

 

[oldStatesman]

Thanks for the explanation Hark - it is nice to know what your logs show!

I just suffered the killer BSOD - got a stop error 0x24, followed by an unbootable system with a stop error of 0x7b. Short explanation, The first was the file system became corrupt, the second no valid system files because of the corruption. Civ 4 itself did not cause this directly, but indirectly becasue it corrupted a system write when it CTD, causing the first error, which caused the second.

I had to re-install from ASR backup to recover. Oh well, the silver lining is that I have a clean OS again - since the most recent backup I had was from early October when I did a re-install.

Because I keep My Documents and apps and other data on different partitions I suffered no data loss, just have to re-install the apps.

Anyway, back on topic, I have a request for someone (who is a techie and knows how to do this - others, I appreciate it but if you do not understand the registry, please do not try as if you mess it up you can crash your system, I do not want that to happen)who is not experiencing problems - I would like to compare the registry values in XP Pro sp2 of the following registry key to mine to see if anything is different there as far as XP memory management.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\

(Ignore the extra space in the word 'control' above, for some reason when I post this it adds a space)

You can just export the key values in Regedit and post it here or PM it to me in a text file.

TIA!

 

[phalzyr]

Anyway, back on topic, I have a request for someone (who is a techie and knows how to do this - others, I appreciate it but if you do not understand the registry, please do not try as if you mess it up you can crash your system, I do not want that to happen)who is not experiencing problems - I would like to compare the registry values in XP Pro sp2 of the following registry key to mine to see if anything is different there as far as XP memory management.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\
TIA!

Does it have to be Pro, I'm not sure if I have, home or pro, at home. redone it so many times...though someone else that know how can get it to you faster I'm sure....I don't have internet at home....

 

[oldStatesman]

Does it have to be Pro, I'm not sure if I have, home or pro, at home. redone it so many times...though someone else that know how can get it to you faster I'm sure....I don't have internet at home....

No, 2000 would be okay - Pro would be ideal, but 200 or Home should not be that much different. I have a 2000 machine running to check, but it is way below spec and will not run Civ4.

I have a 'clean registry' right now to use as a benchmark - I'd like to compare to see if civ 4 makes any changes, or if those who are running fine have different values. I do have the old registry from pre-crash to compare too, but since I was having issues I'd like to see if this key is different. I'd like to do this before installing Civ 4 again - I am trying to benchmark everything since the system right now is clean. I will be installing the game over the weekend, and see what changes, if anything. I am also going to log all registry changes the game makes to compare before and after.