Memory Leak?

[Harkonnen]

I_batman

Control Group is right. If this is true, then this is conceptual mishap with Python invalidating most of game code. The only thing Firaxis can do in such case is to limit map size to Large and post 1Gb as recommended.

Not only heap allocations do consume space, but it's just too overheady... Above I wrote about WolrdBuilder save file... (one is created each time you load a game, so you definitely have one). Just check it... you'll be damn surprised of its format. I suppose the whole engine is overloaded with constructions like that save-file.

Probably they track entire game replay and keep it in Python memory. And since every junk of that replay is heap-allocated, it fills in between those small blocks, and never becomes cached out altogether, just because its memory pages are mixed up with required content...

 

[I_batman]

I_batman

Control Group is right. If this is true, then this is conceptual mishap with Python invalidating most of game code. The only thing Firaxis can do in such case is to limit map size to Large and post 1Gb as recommended.

Not only heap allocations do consume space, but it's just too overheady... Above I wrote about WolrdBuilder save file... (one is created each time you load a game, so you definitely have one). Just check it... you'll be damn surprised of its format. I suppose the whole engine is overloaded with constructions like that save-file.

Probably they track entire game replay and keep it in Python memory. And since every junk of that replay is heap-allocated, it fills in between those small blocks, and never becomes cached out altogether, just because its memory pages are mixed up with required content...

Geez, Harkonnen, are you and Control Group saying what I think you are saying....this is a virtually fatal flaw in the code that cannot be fixed without gutting virtually the entire engine?

And I want to make one thing clear, I have done a fair bit of coding in my life, but none professionally or to the depth that you know this stuff. I have fiddled with memory allocations , but that was a long, long time ago in university.

And Civ IV craps out on my laptop as soon as I try to run. (likely the T&L issue on the video card), so I can't even look at the save files in World Builder.
At least not yet, until my gaming machine arrives. The horrible thing that I think you are saying is this: If you have 512 Mb memory, the game crawls. If you have 2 GB memory, you risk Crash to Desktop.
You need precisely 1 Gig memory to play the game in its present form, and even then it will lag regardless.

Is this a good paraphrase?

And I do have another question? How do you respond to the people who all say the game is working perfectly on their machines? I read there are a lot of them, and I am wondering if they are simply adjusting to the slow speeds, or is there something more going on that allows them to play?

 

[Saad]

The memory problem is more accute in a hotseat game. A friend and I were playing and over time as the visible map became larger, the game would crash more often. When playing by myself, crash frequency was very low. What pissed me off was that the game does not save backups everyturn like it used to and I couldn't change this setting.

Because it happens more frequently in hotseat, I believe the crash is related to rendering the terrain map, as each time we switch a new terrain is loaded. I also found a sure fire way to crash the program was to trade for a world map and click on a random location.

My PC specs are 512 RAM (I'm planning to get another 1GB just for this game), 3.0 GHz PIV and Nvidia 256 RAM card.

 

[Harkonnen]

I_batman
Well... heap allocations are balancing between three issues:
1) Fragmentation
2) Performance
3) Amount of used memory (actually derived from 1)

Generic allocator (and civ4 uses one) are balanced according to world's software written in C (python uses C allocator and one of its own, but with close to C behavior).

That kills it because python programs are very unlike C programs, they have no stack. Stack is the ideal allocator, it suffers from neither of those 3 issues, but its limitation is that you can delete only items that you most recently allocated. I.e. if you assembled a pile bottom-up, it's best to disassemble it top-down if rules allow such order.

Civ4 is outballanced. It has huge performance capacity (I hope...), but memroy use is away from reality. So I try to fake its allocator (fortunately, not many functions to patch) in order to drive it towards less fragmentation.

I am not sure that every CTD is about this problem. Perhaps, nVidia did make a mistake in their drivers. Perhaps their new driver is more tolerant to low-memory situations, I don't know exactly.

What I see from this site and from my own experience:

256Mb - consumes as much as with 512Mb (~700Mb swap), barely playable...

512Mb - consumes 700Mb overally, if you're addicted, you'll play, but it's not so pleasant...

1Gb - consumes 1.2Gb which is only 20% more then system memory. With 512Mb this I have 50% overhead, so it's most playable with 1Gb

1.5Gb - not enough info

2Gb - almost always CTDs since can't grow to 2.x Gb. Someone offered /3Gb switch for Windows, but noone replied if it helped to escape CTD with 2Gb system.

I am not yet sure that all those people are playing huge world, and most who tried, changed their opinion. Those who discuss in strategy forum say that they are finishing their 10th game. I don't think this would be true with each game being completely finished huge game.

Anyway, I started writing this patch primarily for myself because my system does not allow cheap upgrade, and I want this game too much. With huge map For instant gaming I prefer shooters...

 

[ledhed]

The game got too slow to enjoy so I exited out to desk top . I checked the task manager and it was running at 239,621 memory usage...this climbed to 244,863 over a few minutes AFTER I closed the program , before tricling back down..whats going on ? I never saw a program use so much memory .

 

[Chopperhead]

When you machine sounds like it is ready to die, is your HD going nuts, making all the noise?
I mean, during that huge load time, is your HD being accessed for that entire time?

yes it does all the way through and it does the same thing in between turns also

 

[Harkonnen]

Many modern games eat as much. Even civ3 was at about 300Mb sometimes as far as I remember....

How much memory do you have? Looks like 256Mb or even 128Mb if 244,863 that sounds much...

Tiny map with my system starts at 150Mb, start menu - 80-90Mb.

 

[ledhed]

512 ram..rams expensive because its a dell . all my specs exceed or meet what the game recommends . It seemed like alot to be using AFTER I closed the program..during game play it sometimes reaches 2.3 million k..at any rate it just crashed when I used a spy to do sabotage .

 

[Chopperhead]

ya i gota dell to and I just dropped $300 on a new vid card. Now im told I might have to double my RAM also this game is going to end up costing me like $500

 

[alexti2]

Absolutely. I was just pointing out that, of the two, it's easier to write a memory leak into C++ than it is Python. In C++, you have to make sure you use appropriate extensions to the language to take care of GC, while Python does it natively (or so I believe; I haven't actually used Python, so I could be convinced I was wrong). All I meant was that if we're comparing which language is more inherently prone to memory leaks, it has to be C++. C++ is more prone to all sorts of low-level errors, since it's really quite close to ASM.

I didn't mean to say that it's hard to leak memory in C++ if that's what you want to do. I was intended to say that you don't need to do any special efforts to avoid leaks. Unlike some languages where you always sort of leak memory (meaning that you don't need some memory anymore, but you have no way to release it and have to rely on gc to eventually (like at the end of application) release it).

 

[Harkonnen]

In python 'gc' is immediate as soon as ref-count reaches zero. Though, there is another issue - loop references, such nodes will hang forever... Either 'stderr' redirection is invalidated whenever game starts, or there are no such situation in civ4, checked with gc.collect().

 

[ShyLion]

I had a little trouble understanding this concept and any posible solutions. When running the game I have had a couple CTD on larger maps (the Earth map) with my computer saying afterwards there was a virtual memory problem.
Is this something I can fix? I have 512 RAM, will getting more solve the problem?
As I understand it the game may be suffering from a memory leak in which the game does not released unused memory, which eventually causes the game to crash.
Once the game crashes, should I restart the computer to get the memory back? or is it released when Civ4 crashes?
Thanks

1. Check free space on volumes containing swap files.
2. Check surface of HDD's containing volumes containig swap files.
3. Check RAM
3. What about other, memory consuming applications?

 

[Mazruk]

Quick question for those experiencing late game slowdown and trying to track the cause:

I play mostly multiplayer with my wife, on huge terra maps so far. I've got 1gb and my wife has 768mb mem. Beyond a couple of seconds extra in late game turn ends (and I mean, only a couple of seconds), I haven't seen much evidence of a slowdown (playing 12 hours straight, no exits or reboots).

I run windowed and my wife runs fullscreen.

The few single player games I've played I havent gone far into. However, I DID notice that some features are in single player that are NOT in multiplayer: Wonder movies and the extra Leonard Nimoy comments about each tech - also more help/advice pop-ups occur.

Do people see more problems in single player?

Is it worth someone trying MP and playing it AS IF it was single player (ie only 1 person, all others set as AI)? Dont forget that the game was designed MP from the bottom up - maybe they added SP feature last so they are more buggy?

The only "problem" I've encountered in my 5 days of heavy playing is the load times of saved games - 30-60 seconds to load.

 

[Harkonnen]

As I observe, the game swaps each time another civilization is scrolled through the screen. Do you play 1-vs-1 or with a lot of AIs?

 

[ledhed]

I cant play hot seat because of crashes to desk top. I gather you are playing LAN ?

 

[Mazruk]

My MP games with my wife usually include 5-6 other AI civs.

I play LAN MP on my home network.

I'm just 4 hours into a SP game at the mo and no slowdowns, but its using 100mb more than at the start (start 420mb, 4 hours in 520mb). Huge pangaea map, me + 6 AI civs.

 

[Commander Bello]

Harkonnen, some information about a 1.5 GB system:
The game consumed 912.000 K (measured by taskmanager) yesterday evening, before a war consuming quite some units happened to start. At this time total memory useage of my computer was 1.51 GB (Norton Antivirus, ZoneAlarm, Taskmanager and 2* IE running), so it started swapping as for the first time during moving units I heard my HD start rotating.
Later during that war the memory usage dropped to around 780.000 k.

I am playing on a 32*24 map (gigantic setting) with 17 nations left.

Furthermore, I noticed a considerable amount of time needed if you call the military advisor for the first time in the game, and then it may be called like one would expect. So, it indeed seems that all the data (which??? just to display some unit locations?, btw) are to be loaded and then are kept in whatever kind of memory.

Hope, this information will be useful for you. Seems, you guys are doing a great job here in analyzing what actually is going on with the game engine and I would like to ask you go on with that.
Unfortunately, on Thursday night I will have to leave for a project abroad, so most probably I won't be able to provide you with more information, if needed.

 

[oldStatesman]

Not to throw cold water at anyone working on this issue, myself included, but read this thread.

http://forums.civfanatics.com/showthread.php?p=3292816#post3292816

Seems that at least the Map maker knew that civ4 would have issues with larger maps. And that it would use a large amount of resources. These memory issues may not be fixable as far as the response times...hopefully the CTD issues can be fixed.

But please read it and the linked Apolyton threads to judge it for yourself.

 

[Sevenhertz]

Is this part of the config file relevant?

; Break on memory allocation order #
BreakOnAlloc = -1

 

[Harkonnen]

There are more interesting ones (in order of decreasing interest):
DynamicAnimPaging
ClipTreesRegion
D3D9Queries

BTW, I am now spotting invalid memory usage. When the game loads, it calls something lik HeapAloc( -16 bytes ). My memory manager failed to handle such until I saw it really happens. Segregating own address space via VirtualAlloc always failed here and there, dunno what kills my regions... They are alive until any real allocation.

There is one more thing to blame - that ~e0005.exe process. Macrovision say about some memory protection in their recent Safedisk release, so that no "hacked" can read the code... Probably it's theirs memory works (BTW I found NOCD solution the same night the game arrived to me, 27th october). Wrote nothing, just searched the web... so Macrovision probably provides more trouble than does any really working protection. At best it doesn't affect anything, at worst it is the fragmenter.

Also, when I VirtualAlloc'ed too much for my own needs, DirectX renderer failed during initialization like in those ATi drivers issue.