Memory-fix by Harkonnen is out!

[Harkonnen]

Thrallia
RAM monitor means RAM (system memory), not video memory which is probably shown in dx-diag as "available texture memory". Though, the last one is mixed up with AGP aperture.

I don't get this kind of message (but I didn't play 1.52 for too much). With my fix I simply get some sort of access violation right after the intro movie. The bad thing is that it happens in civ4/d3d/kernel32, so I can't directly spot problematic line of my own code. Those line(s) of code do not cause access vioaltion when they are executed, but they do something that civ4 crashes later. These bombs are hard to find in undebuggable code

 

[Llewen]

Just thought I'd post back with some hopefully useful information. Civ IV is back to an 8 out of 10 for me. I have 1GB of ram so I thought I'd be smart and turn off my paging file. Apparrently that was what was causing the problems for me. Since I reactivated my paging file I haven't had a single crash (mind you that is only for maybe a dozen turns, but still much better than it was). So it would appear that probably the problems I was having were my fault. Apologies to Firaxis...

 

[Officer Reene]

Just thought I'd post back with some hopefully useful information. Civ IV is back to an 8 out of 10 for me. I have 1GB of ram so I thought I'd be smart and turn off my paging file. Apparrently that was what was causing the problems for me. Since I reactivated my paging file I haven't had a single crash (mind you that is only for maybe a dozen turns, but still much better than it was). So it would appear that probably the problems I was having were my fault. Apologies to Firaxis...

what is "paging file"?

What does it do?

I have 2 GB of ram

how do I turn it off?

will it affect other applications/programs if I turn it off?

thx

 

[quixadhal]

The paging file is the system file Windoze uses to implement virtual memory. Under Windows XP, the settings for it are in My Computer -> properties -> Advanced -> Performance -> Advanced -> Virtual Memory.

With 2G of RAM, you should be able to turn it off entirely if you want.... if anything that isn't an RDBMS or Photoshop with 200 layers open needs more than 1G of RAM, it's a big fat bloated pig that needs to die.

All turning it off and then back on *should* do is cause the paging file to become less fragmented. That might reduce latency for a time, but unless you also set the minimum and maximum sizes to the same value, it will gradually refragment itself.

So, while it may help.... it still isn't a valid solution, since Civ IV was coded to run in a normal WInXP environment and should not perform badly on a 1G machine with swap enabled.

 

[Falken65]

Just wanted to thank Harkonnen for his excellent work. I'm using 1.09 with Harkonnen patch and only having problems in late game now when revealed world size (extra graphics to process) and sudden jumps to other spots (by clicking on world map to "jump" to a spot that needs my attention) overwork the video memory.

I have a 1GB RAM system and use Cacheman XP to free RAM and control the pagefile size. I can see in that instance where turning off the pagefile is not a good idea.

Hopefully, things will be better once my new computer gets shipped and I have a dual-core, 64bit AMD CPU w/2G RAM and 2 6600/256 (total of 512 vid mem) in an SLI configuration.

Current "beater" is AMD Athlon XP (1.8 ghz equivalent), 1G RAM and GeForce FX5200 (in case system specs will help Harkonnen). 1.52 is unplayable on it, but 1.09 + Hark patch + autosave every 2 turns makes the inevitable CTDs more bearable.

 

[Thrallia]

Thrallia
RAM monitor means RAM (system memory), not video memory which is probably shown in dx-diag as "available texture memory". Though, the last one is mixed up with AGP aperture.

I don't get this kind of message (but I didn't play 1.52 for too much). With my fix I simply get some sort of access violation right after the intro movie. The bad thing is that it happens in civ4/d3d/kernel32, so I can't directly spot problematic line of my own code. Those line(s) of code do not cause access vioaltion when they are executed, but they do something that civ4 crashes later. These bombs are hard to find in undebuggable code

Harkonnen:

The reason I went ahead and posted this error(it isn't the usual error I get, usually I get a virtual memory error), is because you should remember, I've never had any kinds of errors before, only immense lag in late game. So to me, it means that v1.52 is using memory in a different way entirely from v1.00 and v1.09. In those games it got laggy, but even on huge maps with every graphics level at max, I never got a CTD or out of memory error(vid mem or RAM or virtual)
Plus, if I'm running out of vid memory now, I should have been running out every other time I played as well, cause I've actually turned down all the graphics to medium, and was playing a standard map when this happened. So something is odd about how v1.52 is 'fixing' the memory issues. it is almost like they patched a sloppy version of your fix onto the game.

Not to mention, I've never had a 'ran out of vid memory' or 'ran out of virtual memory' error before with any other game I've played...and I've played several with computers that were below the 'required' specs in the past.

 

[Officer Reene]

With 2G of RAM, you should be able to turn it off entirely if you want.... if anything that isn't an RDBMS or Photoshop with 200 layers open needs more than 1G of RAM, it's a big fat bloated pig that needs to die.

All turning it off and then back on *should* do is cause the paging file to become less fragmented. That might reduce latency for a time, but unless you also set the minimum and maximum sizes to the same value, it will gradually refragment itself.

thx for the info...

now my question is...

do I turn virtual mem off and leave it off when I play CIV4

(turn it back on when not playing CIV4)


and if I turn it off do I need Harkonnen patch? (should I remove his patch?)

will it solve that mem problem...?

(and maybe install the 1.52 patch too?)

 

[Officer Reene]

With 2G of RAM, you should be able to turn it off entirely if you want.... if anything that isn't an RDBMS or Photoshop with 200 layers open needs more than 1G of RAM, it's a big fat bloated pig that needs to die.

and what is a RDBMS? (sorry not really computer literate)

 

[Harkonnen]

Thrallia
So something is odd about how v1.52 is 'fixing' the memory issues. it is almost like they patched a sloppy version of your fix onto the game.
Well yes, and they say this themselves publicly. No, they don't say "we used Harkonnen's patch", but rather they say about principles behind which are similar to mine and were told in this forum.

They did save more free memory, and that was expectable since they know which "brick" is which before they house is built. What I don't get is how could they make it working slower (even without swapping). I have to calculate hash function over entire geometry buffers each time they are changed, and even with this kind of overhead my patch worked faster.

I had performance similar to theirs in 1.52 until I have optimized my hash function. Does it mean that Firaxis hashes own data? I doubt but this might be the case If so, it's like searching for own goods in the own shop on item-per-item bassis, not on per-box/shelf/room bassis, also not looking at how the item looks, just watching at its barcode/serial.

 

[SithlordDK]

thx for the info...

now my question is...

do I turn virtual mem off and leave it off when I play CIV4

(turn it back on when not playing CIV4)

You need to restart the PC after changing any option on virtual mem.

 

[Smidlee]

You need to be careful turn off all virtual memory even with 2GB of ram. I've read that some programs won't run if it's totally turn off.

 

[Llewen]

Ya, I've heard that turning it off completely is a bad thing as well, but I had to try it... One thing I would recommend is something called page defrag, I think... *looks it up*

Yep found it, Page Defrag. It's freeware, completely painless, and I wouldn't run a 2000/XP box without it.

http://www.sysinternals.com/Utilities/PageDefrag.html

 

[phubar]

Llewen: Does PageDefrag help with gameplay? Doesn it last longer without crashing (understand that crashes are still inevitable)?

Phu

 

[SithlordDK]

Llewen: Does PageDefrag help with gameplay? Doesn it last longer without crashing (understand that crashes are still inevitable)?

Phu

I think it just rearranges the fragments from the virtual memory, so that Win (or Civ or the prog that uses Virt Mem) can access the pagefile faster. So it could help playing a game (a little bit), but it won't prevent it from crashing.

 

[Dornadair]

hm, well, I'll post it for you anyway, just in case you get a different one than me...it took longer on a standard world, but I got it to crash on me

First message:
Your system has run out of video memory.
Please try reducing your memory settings.
file:\main\civilization4.bak\SDK\Gamebryo2_0\CoreLibs\NiDX9Renderer\NiDXVBManager.cpp; line: 1018"

funny, my RAM monitor showed I still had 100MB free memory when that went off...and it was followed, as always, by this one:

"Runtime Error!
Program: Civilization4.exe
The application has requested the runtime to terminate it in an unusual way.
Please contact the application's support team for more information."

lol that's a joke, right?

I have the exact same issue - same error message.
I have a radeon 9550 with 256MB and 1GB RAM.
I have tried various things eg overclocking the video card etc.
It seems to happen more towards the end of the game when there are more things on the map. I am patched to v1.52.

Please help as I am tearing my hair out!!!!!!!!!

 

[Llewen]

Does PageDefrag help with gameplay? Doesn it last longer without crashing (understand that crashes are still inevitable)?

I don't know if it will specifically help Civ IV run longer without crashing, but it will definitely help your computer run faster and smoother, and it might possibly prevent some kinds of crashes. It is probably even more important than keeping your hard drives defragmented generally because your paging file and registry are the two most accessed data fields on your hard drive(s).

A couple more tips, if you have the memory to support it, try increasing your graphics aperture to 256mb, in spite of other opinions which I had read which stated it should never be higher than 128mb, I can state categorically that it has prevented one specific glitch, and a late game crash that was related to that glitch. However, I'm guessing you need the ram to support this. I expect that if you have less than 512mb of ram, there is no point setting your graphics aperture that high, and you will be better off with 128mb (will Civ IV even run on less than 512mb of ram?).

edit: ...and tip number two is having a senior's moment two blocks from where I left it sitting on the curb...

edit: ...tip number two caught a cab and made its way back to the home...

Set page defrag to run every time you boot up your computer. If it is done regularly it will only add maybe 30 seconds to your boot up time. If you've never run it before the first time you boot up with it it will take awhile I expect to run, but we're talking in the order of a few minutes, not the hours that it can take to defrag a large hard drive (depending on the program you are using).

 

[quixadhal]

You need to be careful turn off all virtual memory even with 2GB of ram. I've read that some programs won't run if it's totally turn off.

Impossible. Well, ok, this is Microsoft we're talking about... so I suppose it might be possible. However, the whole point of virtual memory is that no application program should be able to tell that the memory it's using isn't "real" memory.

When a program gets loaded into memory, the CPU (actually the MMU) gives it a virtual address space large enough for itself. That space gets mapped to either real RAM or chunks of disk space behind the scenes, and when the application asks for byte 138,765 of its address space, if it isn't already in memory, the OS will swap someting out to load it. There shouldn't be any way for any application to know about paging, since when it's running, it IS paged in.

The only reason virtual memory should ever be used is if you don't have enough physical RAM to keep everything loaded at once. If you disable it, programs that would have swapped will instead run out of memory. Same result as filling up swap, just happens a bit faster.

Oh, and RDBMS is Relational DataBase Management System... Oracle, PostgreSQL, MySQL, etc... databases like to have lots of memory for doing complex queries.

 

[Harkonnen]

quixadhal
As one instance when it may crash is memory-mapped files with NULL passed as file handle - in this case this memory-mapped file is actually backed up with system swap file. I am not sure that OS will reuse system RAM for this in case there are plenty and/or swapping disabled.

This is pretty usual situation when two separate processes have to exchange huge portions of data measured in MBytes (for example transferring bitrmaps captured in one process into another at real time). Though actually it works like physical memory sharing due to caching of all those reads/writes, it still needs some physical storage.

Well, this is for another forum, but I mean that swap file means a little more than fault-gap in case you run out of physical RAM. There are always some 200 out of 512 Mb free RAM with my system (when I don't play civ4 ), but still swap file usage is about same 200-300Mb. I think that OS prewrites at idle times memory onto hard-drive, so that once swapping is involved, it has only to read since everything was written beforehand, but it may have some other use causing unaware applications to crash when they have no swap file at all.

I think there are some expected allocations strategies behind this... and looking at how well XP performs under both Celeron 500 / 96Mb, P3-1.3Ghz / 512Mb and P4-3.4 / 2Gb, I claim that these "disk-first" strategies tend to be useful and almost overheadless.

 

[alexti2]

Impossible. Well, ok, this is Microsoft we're talking about... so I suppose it might be possible. However, the whole point of virtual memory is that no application program should be able to tell that the memory it's using isn't "real" memory.

It's not really that hard. As soon as application starts it tries to calloc() block of about to 2Gb (whatever is left in virtual address space, assuming 32-bit build), sees that allocation fails, reports the error and quits

 

[SithlordDK]

Is the Fix dead??? ...or just the Thread