Memory-fix by Harkonnen is out!

atomant
Christmas greeting appears as a message when the game starts (and you have to click 'OK' to proceed) - it doesn't matter if the game runs in fullscreen or in windowed mode. It appears every time you start a game between 25th and 31th December inclusive (according to system clock).
 
Just a note Harkonnen, I hope that you do continue with your patch. I've been waiting for the installer version and I look forward to adding that on top of 1.52.

Thank you and Merry Christmas!
 
SenGruber
Yes, I will continue working on with it. Installer will be delayed a little since 1.52 is a little different (still dunno how much "a little").
 
Harkonen

I've tested your patch on several systems.
1) Travelmate 804 laptop, Centrino 1.7ghz, 512mb ram, Raedon 9000 64mb
2) Desktop PC P4 3.2ghz, 1gb ram, Matrox Pahrellia 256mb

Without the patch they both worked, without any crashing. The laptop was obviously slow.

On both systems the patch made 1.09 run faster ( though it was barely noticable on the desktop, which ran nearly without delay anyway ).

I installed 1.52 on both, and did *not* reinstall your patch afterwards.
They both ran fine, and they both still registered both your patch, and also the 1.52 patch.
They both ran at roughly the same speed as 1.09 with the patch.
The laptop now occassionally returns the "out of video memory" message.
This does not crash civ. It minimizes it to background, shows the prompt, and then returns to civ, and it works okay afterwards.

I adjusted your .ini to have vid memory set to 60 after the first few prompts,
and this improved the situation.
Certain things seem to spark it - and it's very rare. The sure fire way is to launch the military advisor.

Just giving you some feedback that might help you.

As everyone else has said, your work on this is really very appreciated!
As a programmer myself, I'm also very impressed with the method you used.

G
 
gbjk
1.52 replaces original 'zlib1.dll', so my patch is not loaded unless you deliberately reinstall it again. Are you sure my patch was in effect with 1.52? Did you use post #1, post #640 or post #842 version? Last two contain FPS meter, so it is always seen if my patch is in effect.

Also, post #1 version changes civ4 window title (visible in windowed mode).

Just with my system and MemSaver=1 my patch worked, but did nothing since allocations strategy changed in 1.52 With MemSaver=0 it crashed with my system right after intro movie or even before it.
 
I agree that 1.52 does work worse than 1.09 + Harkonnen fix.
They seem to have picked Hark's ideas, but not implemented them as well.

What really surprised me was the side effect (Alt-Tab) of advanced memory-saving option, being exactly the same as the one of Harkonnen. I though first that they re-used Hark's code.
but no.

I'm a bit pissed that Firaxis just comes here, take the ideas, and re-use them without even mentionning Harkonnen or talking to him. How rude.

These guys are always there to complain about software piracy, and when they get the possibility, they copy other's work without even giving the credit. Not a thing that will improve Firaxis credibility. At least in my own eyes.
 
Am I the only one here who's game runs better and smoother under 1.52 than Harkonnen's patch? I was installing it right, but the effects where minimal at best. If I did a map at Standard size my screen would randomly go black for a couple seconds here and there, and above it would just crash. Before Harkonnen's patch I would just lag like mad.

But now with 1.52 I can even play the huge map and it's smooth and stable as ever. And yes, I tried Harkonnen's patch numerous times under 1.00 and 1.09 just to see if I was doing something wrong, but either way it didn't help much... it just must be my system. And yes I have fiddled around with the settings, read this entire thread to get a better understanding of the patch, and so forth. But I never got the results others are saying they got.

Finally, the use of unofficial patches is not recommended and is largely redundant with the memory fixes in this patch. Unofficial patches may slow performance, cause graphical problems or conflict with Firaxis code resulting in crashes or other problems. It also makes it more difficult for us to respond to error reports and debug issues reported by users running unauthorized patches.

I only slightly agree with this, and I don't think it's right to scorn them for saying this. Harkonnen's patch won't work wonders for everyone. It managed to make my game nearly unplayable, and I am glad that 1.52 is here. I am happy it worked for many others though, and praise Harkonnen for his work that has given others a month of good gaming time. It's unfortunate the patch refused to work for me. :(

My specs are:
1.70 Ghz
512 MB RAM
Radeon 9000 series, 64 MB
 
i just registered to say that it's pretty sad that a customer has to fix the software companies' problems.
thanks to you harkonnen, i can finally play maps bigger than Standard without crashing after a few turns. much love to you!
 
Fanaza
Well, my fix has proven to be bad with 64Mb ATi cards (though, 64Mb nVidia cards were good, and required 'vid_mem_usage=100/120' or so). Also, 512+64 is not enough for my fix to make difference, and 1.52 saves more memory. So, you path is pure 1.52, and I totally agree to that (or 1.52 with some later version of my fix ;)).

Tractor
3Darklight
Thanks! :)
 
@ Fanaza
This seems to be a big problem with civ4. A fix to one's pc is a curse to another. I also got better improvement and saves more memory with 1.52 patch. (memory not a issue on my pc though)
Still I'm gald Harkonnen helps so many to get civ4 running. p.s. I wish Harkonnen did fix all the problems with civ4.
 
Tractor said:
I agree that 1.52 does work worse than 1.09 + Harkonnen fix.
They seem to have picked Hark's ideas, but not implemented them as well.

What really surprised me was the side effect (Alt-Tab) of advanced memory-saving option, being exactly the same as the one of Harkonnen. I though first that they re-used Hark's code.
but no.

I'm a bit pissed that Firaxis just comes here, take the ideas, and re-use them without even mentionning Harkonnen or talking to him. How rude.

These guys are always there to complain about software piracy, and when they get the possibility, they copy other's work without even giving the credit. Not a thing that will improve Firaxis credibility. At least in my own eyes.

Just to clarify a few points about this issue, Firaxis has almost 20 top-notch programmers and most of them have been in the game programming business for years. We have numerous experts in graphics technology who have also worked on major 3D titles at other game companies (including our director who was the head of the Xbox graphics group at MS). My point is that we are familiar with the tricks of the trade by now, especially general techniques like Hark's patch uses (which allows it work on any game). Don't assume that because Hark's patch or some other game came out before ours, that we stole their ideas. For example, the alt-tab side effect is caused by not using managed graphics resources (which saves system memory). This is common knowledge; here's a DirectX article about it:

http://msdn.microsoft.com/library/d...ctx9_c/Resource_Management_Best_Practices.asp

I assume that Hark's patch is doing something similar with managed memory, but he never said what his code does (other than sharing graphics buffers, which we also do, and is another general graphics trick). I understand that he was hoping to work out some kind of arrangement with Firaxis so he didn't want to explain his work. However, no one on the programming team received any email or call from Harkonnen; we haven't ignored his call. We've been busy working hard on the 109 and 152 patches which have been under development the past 2 months. Even though the game has been selling great we wanted to improve it as much as we could for the fans and for ourselves. It might have seemed like we were doing nothing while Hark and others were fixing Civ, but we were working long hours on the same issues and more. It just takes longer for us to make an official patch since we have to deal with installers, patching, testing, copy protection, etc.

So if we knew about these memory tricks, why didn't we use them earlier? We've been working on Civ IV for over 2 years (from scratch) - so we are experts on its source code and had a long list of optimizations to choose from. At launch, our focus was on making the game as stable as possible (even then we still had issues). Many of the optimizations in V152, such as the ones mentioned above, required extensive modification to our rendering engine, and usually had a downside (such as the alt-tab effect) so we didn't want to make those changes initially (we still plan on handling alt-tab in the future). V152 has ALOT of memory and performance improvements, much more than are mentioned here. We were able to reduce the size of (and optimize) models, textures, and interface elements, as well as implement more aggressive algorithms for releasing memory as the game progresses. Many systems were made more efficient and that allowed us to also reduce the RATE of memory usage, which is the most important factor to avoid late game problems. As a result of this, my tests show that V152 is the fastest and lowest memory version of Civ yet, even compared to V109 + Hark's patch.

In any case, if you get better results on your system from a previous Civ version or with Harkonnen's patch, by all means use it at your own risk. That especially goes for advanced Civ experts like many on these forums. Understand that Firaxis and Take 2 can not officially recommend or endorse external code patches since we don't know what's in the code. Instead of a friendly Merry Christmas easter egg like Hark's patch, the next unofficial patch that comes along could just as easily contain something destructive or offensive to people, resulting in big problems for Civ and the companies involved. Even if the patch coder thinks it's fine, it might not be appropriate from our perspective. As a legal issue, the last thing that Firaxis and Take 2 want is to be associated with another 'Hot Coffee' situation caused by unknown content in a recommended patch, which results in millions of $ of losses or other damages.

So please don't take this personally. Harkonnen, you're a good guy with a bright future as a programmer from what I can see and I've enjoyed our conversations this week. You've really helped alot of folks get Civ running well and that's the most important thing. So I'm not trying to take anything away from Harkonnen, in fact, he deserves plenty of applause. But please do the same for the Civ team, we've worked hard for a long time to make the game as great as we can, and we are proud of the improvements in V152. We plan to continue working on Civ and there are more exciting things in the works.

Thanks
Mustafa Thamer
Civ IV Lead Programmer
 
thamer said:
As a result of this, my tests show that V152 is the fastest and lowest memory version of Civ yet, even compared to V109 + Hark's patch.


Thanks
Mustafa Thamer
Civ IV Lead Programmer

There's still room for improvement though. My game bogs down as I get into the pre-industrial era. So far, sometimes talking to another civ or exiting the Civilopedia makes all my terrain go black and there's a horrible lag. It was happening more and more frequently as the game progressed, until I finally gave up. Sometimes the speed comes back, but then the screen is a kaleidoscope of various colours. If I save the game and reload, everything is normal again. I have everything set as low as it can go, I even reduced my desktop resolution to 16 bit and lowered the 3D settings of my video card as low as I could stand the quality loss.

And incidentally, the game won't even load if I disable my Norton Antivirus before it starts. I have to do it just as it starts loading, before it hits the main menu. And my specs are:

Athlon XP 1700+
1 gig RAM
Radeon 9550 128 meg

Since this falls somewhere between your minimum and recommended specs, it seems to me that I should be able to at least play a Standard map with my graphic detail set to low, but that's not the case. At least not towards the end of the game. So I hope you aren't sitting back on your laurels and thinking you have the problems fixed because there's still more work to be done. It's an excellent game, it would be a real shame if I wasn't able to complete it to the end.
 
Hey thamer, we seem to have discovered a problem with BFG Tech 6800GT OC graphics cards and Civ 4 are you aware of this problem? It is discussed under tech support forum under thread freeze/crash/reboot.
 
Well Thamer all I can say is a job well done to an issue well handled. All members of this forum can learn from the respect you displayed, although your company was 'verbally assaulted' by some members here.

Firaxis' efforts are much appreciated.
 
THAMER
Thanks for the frank and honest response, we programers understand the way a small change can effect the core code of any program. (much more many small changes). It was nice to hear that our thoughts and improvisations are being heard and implemented. Looking forward to getting this game in the fast lane. keep up the good/hard work.

HARKONNEN
I seem to be getting a little confused reading all these post. Im getting the impression that some are able to use your(latest?) patch with the 1.52 upgrade. Is this the case? I used one of your eirler patches (not sure which one) on my 1.09 version and it worked wonders. I have since done a clean install ( and manualy removed all the directories and extranius files, which if you dont do will give you the impression that your still using your patch and may cause other issues, which I also encountered.) and updated to 1.52. It is dog slow again. I guess I could try it but dont want to mess anything up just yet.:)
 
thamer
Nice to see you in this thread :goodjob:

All
thamer is right about no contact from me. I understood that majority of people wouldn't want to play with 'vid_mem_usage' and stuff, and this stuff appears to be the cornerstone of slowdowns and crashes or great speed improvements with my fix. I was going to make a contact after installer version, but this time 1.52 had put me back several steps (for the least rendered my patch crashing for some reason).

I think I underestimated Firaxis power, so I was expecting them to make the first contact just because I wasn't expecting them to come to their own solution. Well, they did. The score now is 1-1 :) (but still 1-0 or 0-1 for some specific configs).

thamer
The biggest problem with civ4 is dynamic allocation of video memory during the game... it causes most of the problems:

1) Video memory fragmentation, so you almost never can expect if some vertex/index buffer will go to video memory (OK), AGP memory (slow) or fail to allocate (CTD or message).

2) Inability to select textures / vertex buffers with D3DPOOL_MANAGED resources. That causes black terrain, cheshire cat, gray leader box and other problems. Though, it is possible to track situations when some resource fails to create, free some memory and try again, it is still not possible to know two things:

a) If it goes into pure video memory (not AGP memory). That's why reducing AGP aperture helped many to avoid slow-downs. Textures are fast with AGP, but geometry is not, especially with my AGP 4x.

b) If 'SetTexture/SetStreamSource/SetIndices' calls succeed or fail for D3D3DPOOL_MANAGED resources when these do not have video memory instance by the time of this call. That's a subject to reserving some 10-20Mb of video memory, but these reservations must take approx. 40-50% (!) of video memory to always work just because of fragmentation effects... AFAIK, DirectX7 had better resource management, at least one had better control over all these allocation problems.
 
Otterbear
Well, situation around 1.52 and my fix is as follows:

1) Some had my fix still working right after 1.52 installed. This means that some binaries were not overwritten properly, so this actually means that they are sitting back with 1.09, at least with some core parts of the game.

2) With MemSaver=1 (1.52 feature) my fix does not crash, but neither it improves anything.

3) With MemSaver=0 (1.52 fetaure) my fix crashes with my system, so I don't know if it improves anything there. I don't know if anyone managed to start my fix with MemSaver=0 because I am not sure if they were actually running 1.52... or 1.09 badly patched.

As for now, I have added a bold text at the begining of post #1 to avoid people installing my fix over 1.52.
 
I'm sorry if this has already been asked, but it's a long thread.

If I install 1.52 patch and turn harkonnen patch off in the config file, Civ4 is patched properly or not?
 
Harkonnen:
Actually, score is like 10:1, in your favour:)

Your patch allowed me to play civ on large and huge maps, it made diplomacy and even military advisor work (in vanilla civ it's just mega hddwhorage ****up).

Keep up the great work!
 
Hey Hark and Thamer. I've got a question for you guys. I'm not really a programmer and have only basic understanding of C++. (I took one course in college and did very well in it, but to put it in perspective I was ecstatic when I could write a DOS program that would tell you to "BURN IN HELL YOU DIRTY BASTARD" if you told it to perform a division operation that left a remainder. :lol: I've also gone through the MAME sourcecode and made some modifications with my limited understanding to suit my own desires. BTW, if you want to see some INCREDIBLE coding done by some random people, take a look at the MAME source code. Simply amazing stuff).

Anyway, when reading through all the posts about the movie stuttering and how the play fine sometimes but horribly other times, it made me think a bit. I've noticed the following things with how my movies play:

1): If I just turn my computer on the movies play fine for the first few, then stutter and slow down after a little while.
2): The first movie I play almost ALWAYS plays flawlessly. It's the second, third, fourth that progressively get worse.
3): When playing on smaller maps with fewer civs, the amount of time before the stuttering begins increases dramatically.
4): In version 1.52, if MemSaver=0 the movies play great for a good while even with AA. If MemSaver=1, movies begin to stutter and choke a great deal sooner whether AA is on or not.
5): If playing the videos outside of Civ4 in the Bink Video Player, they run blazingly fast and I have yet to come across any situation where they didn't zom along. (I'll have to try and run it along with some other very intensive CPU/RAM stress testing software to see if that affects them).

I guess my question to you guys is how are the videos loaded and to where? Are they loaded into physical RAM or are they loaded into the Video RAM. If it's the video RAM then it would kind of make sense how on larger maps or with AA enabled that the movies would stutter more. I'm just wondering if there is some way to empty out the neccessary RAM space prior to the turn starting where a Wonder is scheduled to be completed? When the movie is finished playing and the text items appear on the screen then the offloaded data could be loaded back in perhaps. I'm just not familiar with the processes of loading and offloading data into RAM so I'm not sure if it's possible or not. I guess I'm just kind of thinking out loud. :D
 
Back
Top Bottom