Memory-fix by Harkonnen is out!

[Harkonnen]

Current version of the fix is not compatible with 1.52. It either does nothing or crashes. Update addressing this issue is on its way.

The patch I was coding during last month is finally out.

The patch is compatible with 1.00 and 1.09 versions of civ4.

So, here we go.

You will need 3 files to be placed into your civ4 game folder
(where Civilization4.exe resides)

A single zipped pack is attached to this post (see bottom of the post), or you may visit this URL (thanks to phalzyr for this mirror):
http://209.174.48.134/downloads/harkfix.zip

Or you may want to download those files separately:
http://www.sampo.ru/~headden/zlib1.dll
http://www.sampo.ru/~headden/Patch.v...yHarkonnen.dll
http://www.sampo.ru/~headden/Patch.v01/Harkonnen.ini

'zlib1.dll' just loads 'PatchByHarkonnen.dll', otherwise it was compiled from most recent zlib source from zlib.net

Note that it is necessary to download and replace zlib1.dll (either via .zip or solely), otherwise memory fix won't have any effect because PatchByHarkonnen.dll won't be loaded.

I recommend to turn AGP on (SMARTGART for ATi cards for example) and set AGP apperture size to maximum in BIOS (if you don't know how, just don't care). Your AGP setting might be off because it helped some people earlier to keep game non-crashing, so they still might have it off.

This patch is primarily for those who have at least 64Mb (better 128Mb) of video memory with preferably TnL card above GeForce2. You may give it a try even with lower specs - some people reported it had effect on very low-end machines.

First of all please follow these steps. Note that some steps are about decreasing graphics quality. This is not required, it is just to make sure that your first start with my memory-fix to be success, so you may increase graphics quality later when you get it running for the first time.

So, let's go:

1) Modify "Civiliation4.ini" - set "D3D9Query = 1", "DynamicAnimPaging = 0". If you have trouble finding/editing this file, don't care and skip to step 2.
2) If you have 1 Gb of memory or above, skip past step 9.
3) Run the game (if it doesn't run, follow past step 9)
4) Set it to windowed mode (not must-do, but preferable to get stable first launch)
5) Set anti-aliasing to 0 (also not 100% required)
6) Set all low/high settings on the left to 'low' (same as above)
7) Check 'Low resolution textures' (same as above)
8) Then you may check anything on the right, i.e. effects and animations.
9) Exit the game

If the game hangs during 'Initializing Python' step, just restart it and hold down 'Shift' key, so it updates its files cache. This thing is not about my patch, this is something about fresh python24.dll sources...

If you could do all of these steps, try loading some of your huge savegames. If the game crashed or graphics becomes damanged (main menu globe and sun), set 'insane_mode = 0' in Harkonnen.ini file and try again.

!IMPORTANT!

If the game crashes during loading (and crash comes from PatchByHarkonnen.dll if you click details), try recommendation from this post on this thread (post #50):
http://forums.civfanatics.com/showpo...0&postcount=50
It says to turn off ATI Tray Tools software. RivaTuner appears to be ok with my fix (thanks to Kolyana for pointing that out).

Also, xFire coming with 1.09 patch is also causing this crash according to a lot of data sent by filterban per my mail requests. This will be fixed with the next release, and thanks to filterban for his help! It's enough just to quit xFire, no need to uninsntall it. And, again, it's temporary solution until next release of my fix.

!END-OF-IMPORTANT!

Subscribe this thread if you want to get email notifications about new releases. The first update is on its way.

Some explanations:

Windowed mode increases chances that the game won't crash in 'insane' mode (read below).

Antialiasing = 0 dramatically increases speed - the game uses way too detailed geometry, so it turns out around a triangle per every 3-4 pixels. So setting AA to 4 makes your video card workign at actual ~2048x2048 resolution, no wonder it's so slow... Old cards like GeForece2 always increased resolution (supersampling), modern video-cards increase resolution only in places of triangle edges, but these are just too frequent with this game... so multisampling works like supersampling.

I didn't see textures quality affecting anything except globe in the main menu But it allows winning ~50Mb more memory, so I recommend setting them to 'low'. BlueMarble textures might actually suffer with 'low' settings so I leave decision up to you... Just if you have 512Mb or even 256Mb of physical memory, every detail setting counts.

Animations/Effects decrease performance, but they are worth seeing, so turn them on if you want.

Settings described above made my hugest Indian 1961 AD savegame playable with my 512Mb and my patch in insane mode. If you have 1Gb of memory or 256Mb of video memory, you do not need reducing detail level.

If you are interested in my 1961 AD savegame, here is the link:
http://sampo.ru/~headden/Harkonnen%2....Civ4SavedGame

-----

Now, if the game ran fine, please look at 'Harkonnen.ini' and set 'vid_mem_size' as commented there. Basically you may want to set it to 180 if you have 256Mb of video memory. Also please read attentively comment above 'insane_mode' in Harkonnen.ini. Some old card users (or some not-so-popular cards / drivers users) will probably need settings this to '0'.

Memory savings (as for my 1961 AD savegame, low texture settings):

1) patch_on = 0 - ~700Mb

2) patch_on = 1, vid_mem_usage = 80, insane_mode = 0 - ~600Mb

3) patch_on = 1, vid_mem_usage = 80, insane_mode = 1 - ~520Mb

4) patch_on = 1, vid_mem_usage = 180, insane_mode = 1 - ~420Mb
(I didn't try this last one. 256Mb vid-ram people should try that!)

So far, that 1961 AD savegame is the most memory-consuming thing I ever faced, so most

other problematic savegames will work at around ~450Mb even with vid_mem_usage = 80.

I want to thank everyone from Memory Leak? thread for keeping my enthusiasm alive. When I receive some feedback on this patch, I will make final release out of it add all of you guys to CREDITS zone. Thanks!

From now on you can play... enjoy!

P.S: The most funny thing is that this patch acts as a layer between civilization4.exe and d3d9.dll. I mean it doesn't break any non-disassembly agreement. The only arguable effect on the game is changing window title and some messages I left behind to keep Firaxis/Take2 away from using it until they reply to me. Actually, this patch is rather universal, at least with overuse like that one of civ4. I'd like someone to try it with "Sid Meiers's Pirates". Thanks.

Original 'zlib1.dll' and 'python24.dll' (those you may need to overwrite) are located at:
http://www.sampo.ru/~headden/zlib.Original
http://www.sampo.ru/~headden/Python.Original

If after installing / rolling back the game says it can't find 'hapdbg.dll', you may either reinstall the game or download it from:
http://www.sampo.ru/~headden/hapdbg.dll

[Harkonnen]

P.P.S: In my last tests I reinstalled the game and didn't perform FPK unpacking step.

P.P.P.S: As Kangaroo said in 'memory leak' thread, "SyncInput = 1" might solve CTD problems. I could live with both 0 and 1. As I've noted, this helped primarily nVidia users.

[@rjen]

A big thank you, Harkonnen!
Your patch has a significant positive effect on the performance. My saved late games (1950+) on a standard map are running very smooth now! Even your 1961AD save on a HUGE map has become (at least) playable for me

Although I prefer to run the game in Full Screen mode, and with medium graphics and animations turned on; it proves to be of almost no effect on the new performance.

You're the man!

My specs: Celeron D 335 2,8 Ghz, 1 GB Ram, Radeon 9550 128 MB.

[Harkonnen]

@rjen
Thanks man! The first reply is good, I'm 50% less nervous since now

Did you use insane mode?

Full-screen is ok, just with 'insane_mode = 1' you might get corrupted graphics after alt-tabbing back to game. Also, all that "dancing" around low quaity graphics are more for 512Mb systems (like mine)

[@rjen]

You're very welcome

You're right about the insane_mode, when it's turned to 1 and I alt-tab and back to the game with Full Screen, the map becomes black. With insane_mode = 0 this problem is fixed! I didn't notice a decrease in speed yet with this particular setting.

When the US wakes up in a couple of hours I surely expect and hope there will be plenty of praising replies!

Good luck with optimizing your patch, if needed! But first enjoy your free time after what probably has been nights of work

[Harkonnen]

@rjen
Thanks Yeah, I'm currently at work (it's day over here) - I think this gonna grow large when I come home... I've posted a link in all threads related to this problem.

insane_mode is driven towards reducing memory usage at price of alt-tabbing stability (and it might even not work on nVidia) and at price of some speed, but speed penalty of insane mode appears to be negligible.

Its primary goal is to get below that thin threshold when the game swaps like a hell and when it's a paradise (512Mb systems). With 1Gb it isn't an issue as it doesn't matter if the game consumes 600Mb or 800mb, as long as it is within that 1Gb.

[Wolfwood]

Wow! For the first time in weeks I had to actually EXIT the game to stop playing! No crashes at all and my system only has 64Mb graphics memory (set the usage to 50Mb and using medium settings). 1GB of system memory may help as well.

Oh, now I cannot wait until the evening to game some more!

A big thank you to you, Harkonnen!

[Harkonnen]

Wolfwood
Glad for you BTW, Karelia (my part of Russia) is very close to Finland.

I am very interested if 'insane_mode = 1' works. Especially for those who has low-end(9200)/high-end(x800) ATi chips and any nVidia video chips.

(I will reply less frequently since now to keep this thread compact. I see it's gonna be big...)

[Nicos_Rex]

Thank you Harkonnen!

I have spent far too much time trying to get this game to run acceptably on my system. Following your instructions, I have just loaded my "guaranteed CTD" save and been able to continue for several turns at a rollicking speed.

Insane mode = 1 works fine on my GeForce Ti4200 (128Mb) card.

Your efforts and expertise are greatly appreciated.

[panzooka]

tell us what u did with dll files, what is the theory behind this

edit: sorry i didnt follow your other thread, so..

[The Great Apple]

Works wonderfully! Thanks very much.

I do get the black terrrain problem when I alt-tab the game, but I think that may be because I'm being stubborn and not using windowed mode...

I have isane mode on with a GeForce 5700LE - never had any CTDs before (except when I accidently pressed alt-F4 - very easy to do!), and I haven't had any since.

[Dragon67]

Good work Harkonnen.
And since the cdt affect a lot of users, it may bee an idea too make the thread sticky?

[Harkonnen]

panzooka
Basically I am reusing entities for which new copies are created by civ4. I am not 100% sure, but it seems that these entities (geometry primitives) become tangliable only at low level of Direct3D, thus only there they can be managed - that's what my .dll does. GameBryo engine works at meshes/models level, so I think civ4 can't detect duplicates there. E.g. even if ten houses are different, there are still a lot of duplicate items in them - bricks for example, but you must go down to bricks level to mess with them.

The Great Apple
Setting 'insane_mode = 0' will remove alt-tabbing issues if alt-tabbing is important for you in full screen mode. Though it will increase memory usage and if you are low on RAM, it might start swapping again.

[Harkonnen]

Dragon67
Well, in case memory usage was the reason of CTDs... Anyway, a lot of people suffered from swapping too (me for example), and so far I see it's solved.

What should I do to make this thread sticky? Notify moderators?

[phalzyr]

Yep notify mods and if you need a mirror http://209.174.48.134/downloads/harkfix.zip I put the first thread info in the hark_readme file within the zip... My Outlook was acting a bit weird like it does before T1 goes down(physically) but hopefully just something at email server side...(we don't handle our own email)

P.S. All files zipped are less than the main one unzipped, that has to be dloaded...872KB compared to 1.8MB...

EDIT: Now with zlib only 48K

[Dragon67]

Yes, you should notify them about it. Maybe as a sticky or in the same way they handled the ati solution earlier on. I believe it will affect a lot people, including me, so why not. Maybe after a few days’ ore so, just too make us 100% sure that none other issues turns up.

[Harkonnen]

I will write small loader later today, so we end up with something like 30Kb. This was a test release, I didn't expect so much downloads, and I didn't expect all of them to work so fine... Lemme just get to my hope PC...

[phalzyr]

hopefully my email issues are just the fact they been moving to new server, not foresight of trouble... Let me know when you got the new version out, though like I said on other posts I don't have internet at home... I also leave work early but if you time frame is off from ours that might not matter depending on how much "later today" is At worst I'l get the other version with installer up the next morning (my morning)

[Strawberry]

Sorry, silly question but I am not very techie. Can this be used with 1.09?

[Harkonnen]

Yes. I started it with version 1.00, and finished when 1.09 came out. It's independent patch, it will work with 1.10 and forth as well.