Debug build (no compiler optimization, debug CRT) ???

[Harkonnen]

I suspect very much that Firaxis gave us a debug build of their executable. I don't mean just a beta version, folks here are almost sure of this, and I am too. I mean version suitable for debugging, setting breakpoints, making a lot of darn asserts to assure everything is consistent in game state. Well, asserts are actually not so darn - they reduced number of threads here, but they are darn when I have constant swap with them... Well, probably there are no asserts, but the code is unoptimized by the compiler. Why? Read on...

I was among those lucky who doesn't get the newest drivers until faces problems, so I avoided all of CTD and other issues. The BIG reason is LAG. Riddick and PainKiller work waaaay better and consume muuuuch less memory. This thing consumes about as much as doom3/quake4 with high-to-ultra settings (given ENTIRE artwork is about 300-400Mb in size). Civ4 consumes something about 1Kb of memory per tile to my expectations. The word "overhead" is too weak to describe that.

I thought this could be some some DirectX issue. What I've found.... damn, what I've found. I've found d3d9x_26.dll in my SYSTEM32 folder. This DLL is used only in debug builds (release version links D3DX statically, so it stays as part of code within EXE).

Also, 11Mb file size is just... ergh.... I saw such size only with 2 progs: MS Word and some version of 3D MAX. Under release build executable size usually comes to about C++ code size. I don't believe Firaxis has 11Mb of C++ engine code there, especially with all "business logic" in python, aside from exe.

I mean that just building it as 'Release' will chop down memory requirements and speed it up about 2-4 times in case CPU becomes an issue.

I don't know why Firaxis didn't complete Release build before shipping out the game in a rush. Perhaps there is actually some memory problem which becomes access violation under Release build (this sometimes happens), but...

Anybody at Firaxis, is that true?

I don't think this is memory leak. Yes, VM size grows as the game proceeds, but in my case it stops growing after reaching 524Mb (which accomodates with caching new units, new improvements, etc...) . Well, it explains memory usage growth capped at some limit, BUT IT DOES NOT EXPLAIN ABSOLUTE VALUE OF THAT MEMORY USAGE (300-400Mb of WHOLE artwork, remember...).

P.S: I still like this game; and I want to like it because it will give me a lot of pleasure once it doesn't annoy me with 1-second delays to scroll a single tile. I understand that tech issues shouldn't affect my thoughts about taste of entire civ4, and actually these were coded by different people. Still, it's like a hot beer. It might be tasty when it's cold, I might want it very much, but still I can't drink it hot. And I probably won't buy it anymore if it will be sold hot every time and everywhere.

P.P.S: Actually this games costs not $60, it costs $110 if you need a memory upgrade ONLY FOR IT or $210 if you need video upgrade just for it. Why? Because normally I would buy those pieces within a year or two when they will be actually needed by MOST games and their prices will drop (video will drop for ~$200 in such a period of time), but civ4 forces me to do so now, so I am forced to spend these $200 not to Firaxis, but anyways away from my pocket. I won't... So those $60 are wasted up until now. I still have a little hope that these issues are from Debug build - it's fixable. But if it's architecture flaw (e.g. python memory use or engine programmer left the building or whatever...), then we're all out of luck.

 

[alexti2]

11Mb is probably too small for complete debug build. You can check it easily though by attaching debugger and checking if you'll see names of the functions etc. But in fact, depends report civ4 as "Debug", however this will happen even if you have just one function with debugging info linked in. My suspicion is that some part of the code is with debug info, and that's not accidental. Perhaps, release build kept crashing and they ran out of time to get to the root of the problem, and leaving some debug info seemed to fix the issue. Of course, this plan backfired and the problems came out on many machines even with debug build. That's just my guess, of course, but then perhaps, we can expect a lot of problems cured in the patch.

 

[Harkonnen]

I think they stripped away debug info (I saw no PDB reference, in case they used VC++ of course), though executable looks packed, DUMPBIN outputs some junk for its imports.

Debug info on itself does no harm. Unoptimized code does. And debug CRT/D3D3DX not only use same unoptimized, stack-frame-everywhere, no-inline-functions code, but also migh have considerable memory overhead just to perform stronger checking, especially D3DX.

I agree that some memory-issue could be appearing within Release build or this happened to them a lot of times during development, so they decided not to risk and shipped Debug build deliberately, just for safety.

My worst feares are for GFX UI library (that 'S' label along with Miles, nVidia and others). I didn't see it in any other game with it, so memory leak might reside there. In this case it will take aeons to fix, they can't simply change entire UI from the ground.

 

[Harkonnen]

My another worst fear is that that game could silently install Debug DirectX runtime on my PC. If so, that's not funny...

 

[oldStatesman]

My another worst fear is that that game could silently install Debug DirectX runtime on my PC. If so, that's not funny...

YIKES!!!!!

Let's HOPE not. The only reliable way to get THAT monster off is a complete OS reinstall.

I agree with you in your speculation. I too do not believe it is a 'classic' memory leak - the perfmon logs I have run support this so far. civ does use a lot of memory - but it does plateau out - in my case at about 30% of total memory - and stays there. But it does act like a memory leak...it is very puzzling to me, I have not seen anything quite like it.

The code is definatly not optimized - for example it makes no sense why Civ4 should reserve 94% of the CPU time even though it is not always using it. It does release it to other apps - but then gobbles it back up again. Your theory that it may be using debug code may be an explanation for this...and I hope if so it is only because they did not have time to optimize it to Release level, not because there is any inherent issue that can't be solved unless it is run in Debug mode.

Anyway, great observations...the theory does meet a lot of the evidence. I'll keep an eye on this thread and hope to learn more.

 

[Harkonnen]

What makes it strange is that plateau is close to available physical memory. In my case I have 512Mb installed, so it gets somewhere by 520-560Mb. Folks having 1Gb stop growing at ~1.2Gb. Folks with 1.5Gb sometimes catch CTD just because with 1.5Gb RAM it stops devouring memory at address 32bit space limit (2Gb per process data), there stays no more addresses for stack, and it silently quits like any other program with stack overflow does.

Actually this thread would produce no tweaking recommendation if what I told here is true. In this case we just should wait for executable that Firaxis would normally ship out witout deadline oppression.

I think I will try to write forwarder d3dxd_26.dll based on its static implementation for Release build and feed it to civ4. Probably this will fix the issue or at least part of it.

One guy in some other thread recommended to upgrade mss32.dll (Miles Sound System) and binkw32.dll (Movie player) to their most recent version, this helped some people over here, didn't try it yet because I still couldn't found the lastest binkw32.dll (new RAD game tools offer static compilation like what I told about D3DX library). I think I'll borrow some from Blizzard games, those use bink too, but it does not freeze. Well, provided all required functions are in place with that old version...

 

[Harkonnen]

P.S: 100% CPU usage is ok for almost every game. Normal windows apps like photoshop act only when you ask them to do something, all other time they wait for mouse/keyboard input to invoke that action. 3D MAX rendering on the other hand is along action, so it consumes 100% CPU as well.

If game does not consume 100% CPU, this means either tremendously slow video card or constant swapping. I think during swaps at huge map civ4 might consume less then 100% CPU, but anyways - <100% CPU of a game is bad, it means some bottleneck which is not a bottleneck with other games.

 

[Harkonnen]

Another example is several open flash players. They don't consume 0% CPU, but 5 of them may eat up to 30-40%. This is because they need CPU horsepower, but still some time they wait on timer for the animation (framerate is fixed). With unlimited game framerate (VSync off) CPU and Video spend as much time with graphics as they can. If you normally have 160fps somewhere, and turn vsync on to make it 75-100 (Hz of your monitor), this might reduce CPU load as well as video-chip load.

 

[Mercade]

Folks with 1.5Gb sometimes catch CTD just because with 1.5Gb RAM it stops devouring memory at address 32bit space limit (2Gb per process data), there stays no more addresses for stack, and it silently quits like any other program with stack overflow does.

This can be fixed if - and only if - Civ4 is large address aware.

Edit the C:\BOOT.INI file on your PC and add the /3GB flag to it. This will cause the process space to increase from 2GB to 3GB while decreasing the Windows space down to 1GB for that process.

Example:

[operating systems]
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Microsoft Windows XP Home Edition" /noexecute=optin /fastdetect /3GB

 

[Harkonnen]

Mercade
Yeah, probably they have 'int' all over the place where 'size_t' would do. Well, anyway, 3Gb address space will just make it more difficult to start task manager and shut this thing down when it's time to

I made some research. It uses MSVCR71.DLL, so it's coded in VC++ .Net or 2001 or 2003, and it DOES NOT use debug CRT, good. At least memory allocations don't contain file/line records to track memory leaks under IDE. Actually very very bad because this would mean easy workaround. If this is not memory leak (with my system it isn't), then it is a resource hog just on itself, not due to debug CRT.

I've got another idea. The fact that it consumes memory close to actual physical memory, but does not grow further (512Mb systems have no CTD, only 1Gb+ systems do as I see on the forum), tells me that it allocates virtual space on itself, up to physical memory size, for which there is an API function to find this size. I will try to trick civ4 and report like I have 64Mb or 128Mb of RAM, so it VirtualAlloc's less memory for its storage (which I believe would be enough). Probably this will help on the issue...