Memory Leak?

[DavidBeoulve]

Tegilbor, you are my own personal hero.

I've been having this problem too - I haven't finished a single game of Civilization 4 because of this problem -- that and I'm addicted to BigHuge Maps thanks to two years of playing Rise of Nations & RoN: Thrones & Patriots (and many, many, many other games, but even Dawn of War: Warhammer 40,000 failed to wrest RoN's claim as king of non-micromanaging RTS's).

But I digress.

I absolutely LOVE Civilization 4. I was such a big fan of the original Civilization, I once went through the whole game finishing a Conquest victory while losing one unit, by reloading the game every time one died (back in DOS days, reloading the game was a snap, and I was around 14 years old). I learned to play realistically as well, and just ate it up.

Civ 2 still had squares for graphics. In 1996? Come on.

Civ 3, I never heard great things about from my Civ-fan friends.

Civ 4 is like rediscovering Civ 1, better than it was before. And yet I can't freaking finish one lousy game.

 

[Smidlee]

DrewBledsoe
Hmm... Can you send me your 1.2Gb savegame to headden@karelia.ru please? Just my 512Mb system uses approx. 650-750Mb while 1Gb systems use 1.2Gb with the same map.
My hypotesis was that civ4 estimates something based on amoung of free RAM, but it looks like some other thing is affecting it... Probably CPU or video... What video do you have?

It can save same 150-200Mb improving playability, but not much, or it can save 600-800Mb (if savings will be proportional too) making it much more playable.

This may be true on an empty huge map (as i notice this too) but with a fully loaded and revealed Huge map there no way around using 1.2-1.5 gb of memory. I got three pc; one has 512 , another has 1gb while my newest has 2gb in which I was playing Rhye's 18civ earth map . I load this saved game on all 3 and every one of them used 1.3 gb of total memory from the start. The pc with 512 took forever to load up as well continue loading the whole time I scrolled (very choppy). On my 1gb pc which is my oldest doesn't lag as much when scrolling since it relys on less hd paging but lock up for 2-3 minutes if I scroll around too much eventually crashes to desktop.(pc travel jams) If continuing playing and scrolling the memory will peak over 1.5 gb. On my newest PC I notice at first it's a little more choppy until it reaches 1.5gb mark.

The only other game I got which reaches over 1.5 gb is Pacific Fighters 4.02m with the graphics on the highest setting. This game can also get laggy if there too many bullets and bombs going off at once.

P.S. I tried my oldest,slowest pc (Athlon 2400+ with Fx 5500) once more and it crashes when scrolling around which the total memory peaked at 2GB.My HD was running wide open trying to keep up as i scrolled from Rome to LA ,USA. I newest pc (Athlon 64 3800+; GF 7800 gt) hasn't ever reached this high (highest so far has been 1.55 GB).

 

[oldStatesman]

Report on my earlier post #180.

I have played all night - and have not had a CTD yet. I have not tried large maps yet; but this made no diffeence in the past.

I am using the Blue Marble mod, and....

...I have replaced two core 3rd party files, per this thread:

http://forums.civfanatics.com/showthread.php?t=141417

I updated the zlib.dll and the python24.dll. Besides no CTD, Civ4's page file memory usage seems to be peaking at around 775-780 MB. This is appreciably less than it was before making these changes. It had been as high as 990MB to 1GB. (Actuall physical memory used is peaking around 384MB in addition to the page file usage. Again, this has held true the whole time.)

I am not noticing much improvement in lag time though. It still sometimes takes units up to 3-4 seconds to respond to a command.

These results are preliminary; I have not yet given them a workout on a large map with lots of water and islands. I wanted to be cautious tonite; I'll try to put a bigger load on tomorrow.

But this does deserve more study; especially using the updated zlib.dll and python24.dll; especially since the ones released with the game were known to have bugs and in the case of the python one cause CTD issues per it's web site.

 

[alexti2]

I did some experiments and it seems that the lag problem has a very nasty source -- rendering engine is right in the main thread (I've tried to suspend all threads except one and rendering was still working). If you look at the list of Civ4 threads, there's one thread that works actively (60-70% CPU) even when Civ4 windows is not active. (Interestingly enough, frame rate at that time is satisfactory - 200 fps ). Now when I activate Civ4 window, the same thread starts to use about 95% CPU and fps drops into 12-13. This doesn't make much sense to me. What this thread is doing now? By at least it clear that the lag appears due to main thread doing something and has nothing to do with graphic capabilities of the computer. Another interesting screen is "Custom Game". When it opens I can notice that the first line is first set to AI before switching to player's name. With a good timing you can even click on it before it changes and see [default] as a name in the opening dialog. I don't see any sane reason why this dialog would be so slow. Neither map no AI "thinking" exist yet...

 

[warpstorm]

What this thread is doing now?

Quite possibly, animating the objects.

 

[alexti2]

Quite possibly, animating the objects.

I don't see any animated objects on "Custom game" screen. Besides, objects are still animated when you bring focus to other [then civ4] window.

 

[oldStatesman]

See this thread

http://forums.civfanatics.com/showthread.php?t=142735

Version 1.08

Not on the official site yet - only available through the game itself so far.

 

[DavidBeoulve]

Hmmm, downgrade my Nvidia drivers? DoH! I upgraded them. Silly me, come to think of it, but I assumed that since Civ4 used a DirectX that Microsoft hasn't pushed I should upgrade my drivers as well.

I'll try the patch first, and if that doesn't work, I'll downgrade my Nvidia drivers.

 

[warpstorm]

I don't see any animated objects on "Custom game" screen. Besides, objects are still animated when you bring focus to other [then civ4] window.

I do. I see a moving star field, multiple layers of moving 3d cloud layers, a world that goes through day and night cycles (including cities lighting up at night), a sun flare, and a blinking button. (In a static state but still needing to be re-rendered each frame is the UI that is over the background).

 

[alexti2]

I do. I see a moving star field, multiple layers of moving 3d cloud layers, a world that goes through day and night cycles (including cities lighting up at night), a sun flare, and a blinking button. (In a static state but still needing to be re-rendered each frame is the UI that is over the background).

That's the same animation that runs when another windows is active. Why would making Civ4 window current make this animation suddenly much harder to render?

 

[oldStatesman]

See this thread

http://forums.civfanatics.com/showthread.php?t=142735

Version 1.08

Not on the official site yet - only available through the game itself so far.

False alarm. Patch pulled by T2 - bugs slipped through.

 

[alexti2]

False alarm. Patch pulled by T2 - bugs slipped through.

Apparently, in their rush to outrace Harkonnen Firaxis went slightly off the track

 

[oldStatesman]

Blue Marble, and updating the 3rd party drivers did not help. Only delayed the advent of the BSOD.

When I tried to build an academy with a Great Scientist...

BOOM! :suicide:

BSOD with the following stop code:

0x00000050: PAGE_FAULT_IN_NONPAGED_AREA

Fault was in the atiduag.dll module. This is definately a memory issue with how Civ4 is using address space and interacting with the video card. (MS Techniet will tell you this error MAY be caused by bad memory modules or spyware, neither is the case here - clean machine, and I checked the physical memory with diagnostic software- no issues.)

 

[Harkonnen]

Smidlee
Can you please send it to headden@karelia.ru? I mean that savegame that eats 1.2-1.3Gb with all 3 configs.

I just saw with my own eyes when my savegame ate 600-700Mb with my system and ate 1.2-1.3Gb with the other system.

Even if I don't release my patch in time, it'll push Firaxis to bringing theirs sooner, so at least anything good from me Or probably it'll be applicable to whatever they release, we'll see...

 

[Private_pAuLa]

first thing:
great work you are doing here!!

perhaps this helps you to complete your patch:

i was playing via LAN with a friend of mine.
huge map (i dunno if it is called huge, 'cause i have the german civ4, at least it is the biggest mapsize), a total of 16 civs (incuding me and my friend).

my PC:
P4 3.0 GHz, 768 MB DDR-RAM, nvidia pci-xpress gf5-dunno-what 5600 i think

my friends pc:
amd 64 3600+(not shure) 1GB DDR-RAM ati radeon 9xxx

at first all went fine and we had a lot of fun.
then i finished stonhenge-wonder so i knew how big the world is.
here the first lags came up. but still playable.
then about 1700 AD it started to get more and more laggy.

even so slow, that my friend got the message "waiting for response of"
quit annoing! 'cause as we kept on playing he somtimes got the voting-screen.

but it got even worse.
as he invented the satelite-tech both of our pc's hung up and only thing to do was push reset-button.
loading game, payed a round or two, same problem^^
my hdd including swap-file was busy quite a long time. when scrolling busy all the time.

i thougt, why not trying more RAM? kicked out me 256 MB stick and put in a second 512MB. (since adam riese was right, this makes a total of 1 GB RAM )

good thougt, 'cause me hdd was no longer busy when scrolling...
at first... but after a few rounds both of our PC's had a busy swap-HDD

so we saved game and alt-tab out of game and checking taskmanager:
civ4.exe using all(!!!) free RAM. i kicked out some running processes back to game and alt-tab back to taskmanager.
(ab)use even got worse! first "only" 870MB then about 900MB! (dunno why me system used about 30MB )

so we kept our savegame and tried a smaller world (can't remeber wich size exactly but i thing it was "standard")

we played , had lot of fun, and then same prob... but this time much later.
first lag as usual when the original size of the map is known i.e. finishing stonehenge and lags getting worse the more of the map is known. this time no crash inventing sats-tech.

interesting:
as me friend quit game, 'cause he had to go home, i kept playing with much fewer lags!
not for long, but 3 or 4 turns nearly lag-free.

as far as i can remember, the realy annpying lags started each time we played, when we reached the 1700...
for me it looks like, the graphics for every century (antic, medi, etc) of all units, citys, and stuff like mines and huts are kept in memory, even when they are no longer in need (new graphics are shown, 'cause of new time), witch fills up the RAM and forces the OS to put data to swapfile.
AND they are loaded again after a few turns if you start a saved game!

hope it is usefull, even i wrote a small novel

p.s. i have the same problem when plying SP. sometimes even earlier...

 

[phalzyr]

Hark: I also noticed a few memory usage items that just didn't make sense. Maybe you can look into it.

What I done is copy all the TEXT XML files to my custom directory...load game now it seems to have increased 6MB usage (size of combined files) this is even at menu options so it has both versions loaded?

Next I edited out some of the german...french... non-english stuff about 2MB worth and put everything back into the main directory (so it wouldn't double load) I got a gain (use less) of 6MB not the 2MB(this is 6Mb different than orignal figure since these files are no longer in custom directory )...

I assume it double loads any XML file maybe even python ones? but the weirdness of it taking 6MB less instead of 2MB I edited out was strange...

I know meager amount of memory difference...but just curios why the triple return, is it possibly loading everything or just xml file in more than once or compiling them in memory and keeping both source and compiled in memory...

 

[Harkonnen]

Private_pAuLa
phalzyr
Thanks both of you.

From my info I had the following with overall 750Mb RAM used (huge (biggest) terra map near game end, 1 player others are AI):
Python24.DLL (and boost-python): 7-11Mb
MSXML.DLL: 30-40Mb
Civ4.exe: ~300Mb
d3d9.dll: ~350Mb

So, we have most of usage coming from civ4.exe and d3d9.dll. With my memory manager page fragmentation was around 2-8%, so it won't help with any improvement. Also I can't change civ4.exe since it's machine code black box from my "no-source-code" point of view.

Then I started investigating d3d9.dll memory usage.

240Mb - vertex data
60Mb - index data
40Mb - texture data

By hashing and sending index buffers with same contents into a single copy I reduced 60Mb of index data to just 5Mb, so I have saved around 55Mb here with small performance penalty only during loading (buffers don't change frequently during gameplay).

Vertex data wasn't as good, it allowed to save only around 40Mb (out of 160Mb).

Vertex data consists of two major parts:
FVF=0x202 (XYZ, texture1, texture2) - around 140Mb, 40Mb saved with hashing
FVF=0x112 (XYZ, NORMAL, texture) - around 50Mb, 10Mb saved with hashing
(other FVF - too few of each kind to make any use of hashing)

When I tried to fill those major 0x202/0x112 FVF type buffers with garbage, I have noticed that 0x202 corresponds to terrain, and 0x112 corresponds to improvements (but not cities).

I was badly surprised that 0x202 buffers shrinked only 40Mb, I was expecting to win around 120Mb there. Probably this is due to roads. Probably rivers and random coastline make them so different, I dunno... Even hashing just on coordinates, leaving texuring aside still made same ~40mb space victory.

I still can win 100Mb more with 128Mb video memory by not backing up video-memory loaded resources, but this will force me to back up that data in files for proper alt-tabbing (at the price of alt-tabbing speed).

That's the final step I am currently implementing.

P.S: Most buffers are completely rendered, so there is no allocation waste or it is neglectable.

P.P.S: Still I'm darn confused what makes its allocation so different with 1Gb of RAM against my 512Mb of RAM with same savegame... Probably they draw some things twice, but this is hard to track and even harder to fix from d3d9.dll point of view... and Firaxis has some patch on the horizon, so I gotta hurry

 

[Smidlee]

Smidlee
Can you please send it to headden@karelia.ru? I mean that savegame that eats 1.2-1.3Gb with all 3 configs.

I just saw with my own eyes when my savegame ate 600-700Mb with my system and ate 1.2-1.3Gb with the other system.

Even if I don't release my patch in time, it'll push Firaxis to bringing theirs sooner, so at least anything good from me Or probably it'll be applicable to whatever they release, we'll see...

I have seen this also at the beginning of a huge map game. the 512mb of ram pc starts around 700mb while the one with 2gb starts around 1GB. I can't continue running this huge map on the 512 late in the game. Only the 2gb pc can handle when the map gets loaded and revealed after trading maps with AI. I was surprised it was my 1gb pc which reached over 2gb of total memory which in turn crashed.

P.S. I was one of those who happen to download the patch before it was pulledyesterday. I saw no difference in the amount of ram used so far with this patch. Yet I've only got the patch on my 2gb PC.

 

[Fallout2]

I haven't got the time or the patience to read all these posts about this thread. I will just talk about my experience with the game...
First of all I have a AMD 2600; 512 RAM; 120GB HD SATA (game disk); and a Nvidia 6600 GT...
Not a bad system, but this game really overloads it! And for what I read, it doesn't matter if I have a super PC, it will get slugish!

There is defenetly a memory issue here, and if what I read about the game being true (coded in Python) it will not be easy to improve it with a patch...

Some experiences I had with the game: The game only starts to slow down noticebly mid way into the game if the world is almoust all occupied and you discovered a substancial part of it. I only tried to play huge and large maps. I cannot play huge maps because the game almoust stalls with all that memory and non stop HD reading!!! I tryed 2 different graphic cards on the game and I can tell there is an enourmous difference between an FX5200 and a 6600 GT... but that is expectable... what is not expectable is that I can play games which in theory are more demanding graphically much more smoothly with this system than Civ IV! this is for sure a problem with the "core codding" of the game...

Just like to add that with an Nvidia 5200 the sound during the wonder movies plays fine dispite image hickups. However with the 6600gt the movies play fine but the sound gets the hickups...

This is a neat game with smart ideas but I'm sad with the lack of testing and poor programing... I'm sure we'll have to wait for the expantion to seriously play this game

P.S. I turned off anti aliasing and the game improved a bit

(what have you been doing Sid Meyers?!?! )

 

[Harkonnen]

Fallout2
To save you some reading: this is not Python. This is wrong Direct3D access. Some inconsistence which in turn depends on system condfiguration. So far I suppose that amount of video memory is the most important thing.