1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Memory-fix by Harkonnen is out!

Discussion in 'Civ4 - Utility Programs' started by Harkonnen, Dec 7, 2005.

  1. Harkonnen

    Harkonnen Prince

    Joined:
    Sep 13, 2005
    Messages:
    532
    Thrallia
    RAM monitor means RAM (system memory), not video memory which is probably shown in dx-diag as "available texture memory". Though, the last one is mixed up with AGP aperture.

    I don't get this kind of message (but I didn't play 1.52 for too much). With my fix I simply get some sort of access violation right after the intro movie. The bad thing is that it happens in civ4/d3d/kernel32, so I can't directly spot problematic line of my own code. Those line(s) of code do not cause access vioaltion when they are executed, but they do something that civ4 crashes later. These bombs are hard to find in undebuggable code :(
     
  2. Llewen

    Llewen Warlord

    Joined:
    Jun 5, 2004
    Messages:
    274
    Just thought I'd post back with some hopefully useful information. Civ IV is back to an 8 out of 10 for me. I have 1GB of ram so I thought I'd be smart and turn off my paging file. Apparrently that was what was causing the problems for me. Since I reactivated my paging file I haven't had a single crash (mind you that is only for maybe a dozen turns, but still much better than it was). So it would appear that probably the problems I was having were my fault. Apologies to Firaxis...
     
  3. Officer Reene

    Officer Reene It hates you all

    Joined:
    Jan 21, 2006
    Messages:
    698
    Location:
    In the middle of nowhere

    what is "paging file"?

    What does it do?

    I have 2 GB of ram

    how do I turn it off?

    will it affect other applications/programs if I turn it off?

    thx
     
  4. quixadhal

    quixadhal Chieftain

    Joined:
    Jan 17, 2006
    Messages:
    8
    The paging file is the system file Windoze uses to implement virtual memory. Under Windows XP, the settings for it are in My Computer -> properties -> Advanced -> Performance -> Advanced -> Virtual Memory.

    With 2G of RAM, you should be able to turn it off entirely if you want.... if anything that isn't an RDBMS or Photoshop with 200 layers open needs more than 1G of RAM, it's a big fat bloated pig that needs to die. :)

    All turning it off and then back on *should* do is cause the paging file to become less fragmented. That might reduce latency for a time, but unless you also set the minimum and maximum sizes to the same value, it will gradually refragment itself.

    So, while it may help.... it still isn't a valid solution, since Civ IV was coded to run in a normal WInXP environment and should not perform badly on a 1G machine with swap enabled.
     
  5. Falken65

    Falken65 Chieftain

    Joined:
    Feb 6, 2006
    Messages:
    23
    Location:
    New Orleans, LA
    Just wanted to thank Harkonnen for his excellent work. I'm using 1.09 with Harkonnen patch and only having problems in late game now when revealed world size (extra graphics to process) and sudden jumps to other spots (by clicking on world map to "jump" to a spot that needs my attention) overwork the video memory.

    I have a 1GB RAM system and use Cacheman XP to free RAM and control the pagefile size. I can see in that instance where turning off the pagefile is not a good idea.

    Hopefully, things will be better once my new computer gets shipped and I have a dual-core, 64bit AMD CPU w/2G RAM and 2 6600/256 (total of 512 vid mem) in an SLI configuration.

    Current "beater" is AMD Athlon XP (1.8 ghz equivalent), 1G RAM and GeForce FX5200 (in case system specs will help Harkonnen). 1.52 is unplayable on it, but 1.09 + Hark patch + autosave every 2 turns makes the inevitable CTDs more bearable.
     
  6. Thrallia

    Thrallia Prodigal Staffer GOTM Staff

    Joined:
    Dec 16, 2005
    Messages:
    2,836
    Location:
    Maryland
    Harkonnen:

    The reason I went ahead and posted this error(it isn't the usual error I get, usually I get a virtual memory error), is because you should remember, I've never had any kinds of errors before, only immense lag in late game. So to me, it means that v1.52 is using memory in a different way entirely from v1.00 and v1.09. In those games it got laggy, but even on huge maps with every graphics level at max, I never got a CTD or out of memory error(vid mem or RAM or virtual)
    Plus, if I'm running out of vid memory now, I should have been running out every other time I played as well, cause I've actually turned down all the graphics to medium, and was playing a standard map when this happened. So something is odd about how v1.52 is 'fixing' the memory issues. it is almost like they patched a sloppy version of your fix onto the game.

    Not to mention, I've never had a 'ran out of vid memory' or 'ran out of virtual memory' error before with any other game I've played...and I've played several with computers that were below the 'required' specs in the past.
     
  7. Officer Reene

    Officer Reene It hates you all

    Joined:
    Jan 21, 2006
    Messages:
    698
    Location:
    In the middle of nowhere

    thx for the info...

    now my question is...

    do I turn virtual mem off and leave it off when I play CIV4

    (turn it back on when not playing CIV4)


    and if I turn it off do I need Harkonnen patch? (should I remove his patch?)

    will it solve that mem problem...?

    (and maybe install the 1.52 patch too?)
     
  8. Officer Reene

    Officer Reene It hates you all

    Joined:
    Jan 21, 2006
    Messages:
    698
    Location:
    In the middle of nowhere

    and what is a RDBMS? (sorry not really computer literate)
     
  9. Harkonnen

    Harkonnen Prince

    Joined:
    Sep 13, 2005
    Messages:
    532
    Thrallia
    So something is odd about how v1.52 is 'fixing' the memory issues. it is almost like they patched a sloppy version of your fix onto the game.
    Well yes, and they say this themselves publicly. No, they don't say "we used Harkonnen's patch", but rather they say about principles behind which are similar to mine and were told in this forum.

    They did save more free memory, and that was expectable since they know which "brick" is which before they house is built. What I don't get is how could they make it working slower (even without swapping). I have to calculate hash function over entire geometry buffers each time they are changed, and even with this kind of overhead my patch worked faster.

    I had performance similar to theirs in 1.52 until I have optimized my hash function. Does it mean that Firaxis hashes own data? I doubt but this might be the case :D If so, it's like searching for own goods in the own shop on item-per-item bassis, not on per-box/shelf/room bassis, also not looking at how the item looks, just watching at its barcode/serial.
     
  10. SithlordDK

    SithlordDK Chieftain

    Joined:
    Dec 17, 2005
    Messages:
    8
    Location:
    Germany

    You need to restart the PC after changing any option on virtual mem.
     
  11. Smidlee

    Smidlee Deity

    Joined:
    Jul 10, 2003
    Messages:
    3,348
    You need to be careful turn off all virtual memory even with 2GB of ram. I've read that some programs won't run if it's totally turn off.
     
  12. Llewen

    Llewen Warlord

    Joined:
    Jun 5, 2004
    Messages:
    274
    Ya, I've heard that turning it off completely is a bad thing as well, but I had to try it... One thing I would recommend is something called page defrag, I think... *looks it up*

    Yep found it, Page Defrag. It's freeware, completely painless, and I wouldn't run a 2000/XP box without it.

    http://www.sysinternals.com/Utilities/PageDefrag.html
     
  13. phubar

    phubar Chieftain

    Joined:
    Jan 17, 2006
    Messages:
    10
    Llewen: Does PageDefrag help with gameplay? Doesn it last longer without crashing (understand that crashes are still inevitable)?

    Phu
     
  14. SithlordDK

    SithlordDK Chieftain

    Joined:
    Dec 17, 2005
    Messages:
    8
    Location:
    Germany
    I think it just rearranges the fragments from the virtual memory, so that Win (or Civ or the prog that uses Virt Mem) can access the pagefile faster. So it could help playing a game (a little bit), but it won't prevent it from crashing.
     
  15. Dornadair

    Dornadair Chieftain

    Joined:
    Feb 8, 2006
    Messages:
    2
    I have the exact same issue - same error message.
    I have a radeon 9550 with 256MB and 1GB RAM.
    I have tried various things eg overclocking the video card etc.
    It seems to happen more towards the end of the game when there are more things on the map. I am patched to v1.52.

    Please help as I am tearing my hair out!!!!!!!!!
     
  16. Llewen

    Llewen Warlord

    Joined:
    Jun 5, 2004
    Messages:
    274
    I don't know if it will specifically help Civ IV run longer without crashing, but it will definitely help your computer run faster and smoother, and it might possibly prevent some kinds of crashes. It is probably even more important than keeping your hard drives defragmented generally because your paging file and registry are the two most accessed data fields on your hard drive(s).

    A couple more tips, if you have the memory to support it, try increasing your graphics aperture to 256mb, in spite of other opinions which I had read which stated it should never be higher than 128mb, I can state categorically that it has prevented one specific glitch, and a late game crash that was related to that glitch. However, I'm guessing you need the ram to support this. I expect that if you have less than 512mb of ram, there is no point setting your graphics aperture that high, and you will be better off with 128mb (will Civ IV even run on less than 512mb of ram?).

    edit: ...and tip number two is having a senior's moment two blocks from where I left it sitting on the curb...

    edit: ...tip number two caught a cab and made its way back to the home...

    Set page defrag to run every time you boot up your computer. If it is done regularly it will only add maybe 30 seconds to your boot up time. If you've never run it before the first time you boot up with it it will take awhile I expect to run, but we're talking in the order of a few minutes, not the hours that it can take to defrag a large hard drive (depending on the program you are using).
     
  17. quixadhal

    quixadhal Chieftain

    Joined:
    Jan 17, 2006
    Messages:
    8
    Impossible. Well, ok, this is Microsoft we're talking about... so I suppose it might be possible. However, the whole point of virtual memory is that no application program should be able to tell that the memory it's using isn't "real" memory.

    When a program gets loaded into memory, the CPU (actually the MMU) gives it a virtual address space large enough for itself. That space gets mapped to either real RAM or chunks of disk space behind the scenes, and when the application asks for byte 138,765 of its address space, if it isn't already in memory, the OS will swap someting out to load it. There shouldn't be any way for any application to know about paging, since when it's running, it IS paged in.

    The only reason virtual memory should ever be used is if you don't have enough physical RAM to keep everything loaded at once. If you disable it, programs that would have swapped will instead run out of memory. Same result as filling up swap, just happens a bit faster.

    Oh, and RDBMS is Relational DataBase Management System... Oracle, PostgreSQL, MySQL, etc... databases like to have lots of memory for doing complex queries.
     
  18. Harkonnen

    Harkonnen Prince

    Joined:
    Sep 13, 2005
    Messages:
    532
    quixadhal
    As one instance when it may crash is memory-mapped files with NULL passed as file handle - in this case this memory-mapped file is actually backed up with system swap file. I am not sure that OS will reuse system RAM for this in case there are plenty and/or swapping disabled.

    This is pretty usual situation when two separate processes have to exchange huge portions of data measured in MBytes (for example transferring bitrmaps captured in one process into another at real time). Though actually it works like physical memory sharing due to caching of all those reads/writes, it still needs some physical storage.

    Well, this is for another forum, but I mean that swap file means a little more than fault-gap in case you run out of physical RAM. There are always some 200 out of 512 Mb free RAM with my system (when I don't play civ4 :)), but still swap file usage is about same 200-300Mb. I think that OS prewrites at idle times memory onto hard-drive, so that once swapping is involved, it has only to read since everything was written beforehand, but it may have some other use causing unaware applications to crash when they have no swap file at all.

    I think there are some expected allocations strategies behind this... and looking at how well XP performs under both Celeron 500 / 96Mb, P3-1.3Ghz / 512Mb and P4-3.4 / 2Gb, I claim that these "disk-first" strategies tend to be useful and almost overheadless.
     
  19. alexti2

    alexti2 Prince

    Joined:
    Oct 29, 2005
    Messages:
    316
    It's not really that hard. As soon as application starts it tries to calloc() block of about to 2Gb (whatever is left in virtual address space, assuming 32-bit build), sees that allocation fails, reports the error and quits :)
     
  20. SithlordDK

    SithlordDK Chieftain

    Joined:
    Dec 17, 2005
    Messages:
    8
    Location:
    Germany
    Is the Fix dead??? ...or just the Thread :)
     

Share This Page