Memory Allocation Crashes

[Chanterive]

Mon dieu - we might be getting somewhere! I decided why not and gave dgvoodoo a chance. Under the output API there are several Direct 3D 11 options, I chose the (feature level 11.0) option, not sure if that was the intended one. Under DirectX I selected my VRAM to 2048 MB. Then I loaded my game and the dgvoodoo logo was showing in the bottom right. I then switched to RAMmap and used empty option 1&3. And then - I was able to play 3 full turns before I got hit with an ingame MAF error. Progress I guess
Any suggestions on how to push this even further (while keeping the graphic settings on high)?

3 full turns, how exciting !

I'm experiencing problem with my civ install because I tried to modify it too much. Then I tried to reinstall it cleanly from my iso (burned in 2011!), and this time it's not working, saying that I don't have admin rights (I do!!!). I spent the evening trying to make it work. (After playing 3 hours to my game, zero crash with RAMmap and dgvoodoo, actually very fluid)
So I guess since everyone in their right mind should be using the steam version and not an antic home copy, I'll switch to steam tomorrow.
But before to reinstall , I managed to load your save, play one turn with no bug and no RAMmap, only dgvoodoo. It's when I noticed the display was weird (bugging menus, my icone lost the image, etc, because I tried stuff in Windows register...)
So, more to come in the next days.
There's still plenty of stuff made to stabilize old software that i would like to try.

You could try all type of different options in dgvoodoo, also try the agressive emptying in RAMmap, (the 4th option), it did work for me sometimes, sometimes it lead to crash. You could try RAMmap several time in a row. "Throw everything at the wall and see what sticks".

What I'd like is to know exactly is what causes the crash. It needs a proper diagnosis to know the right treatment.

I was surprised by your screenshot, after cleaning your civ4.exe it's around 35Mo (extremely low), mine about 110Mo. And only after one turn you are back to 1,7Go, while after playing several hours I was still around 1,5Go. On another hand your virtual memory is same as mine, high but with some margin.

I will keep looking into it in the next days. If anything new don't hesitate to share.

Bis bald !

 

[giorgio1234]

I found this software, same purpose as RAMmap.

Mem Reduct Download and Install – Latest version 3.4

memreduct.org

It is not from Microsoft so maybe it is less efficient, but it seems more versatile and configurable.
It is possible to load it on system startup, start it minimized, start the memory cleaning immediately, and schedule new optimizations based on memory occupation or elapsed time.
So, It is possible to make it completely hidden and without user action required.
The same clean areas as in RAMmap are available.

I started a new game with giant map and maximum graphic details; I reached the year 915 and 2.8MB savegame size, and I got no MAF so far.

 

[Sinocpm]

A bit of an update after a few more tries:
My setup right now is dgvoodoo installed, loading the game and then using RAMmap. I have used either empty options 1&3&4&5, or just 1&3&5. With this it seems I'm again down to one full turn and then a crash on the second turn. While also having Project Lasso runnning I was able to push it to 3 turns before getting a runtime error.

There's still plenty of stuff made to stabilize old software that i would like to try.

Fantastique!

You could try RAMmap several time in a row

Yeah I think I will try that next, after every turn, just to see if that would help. Not really an elegant solution but for the purpose of testing might be useful.

What I'd like is to know exactly is what causes the crash. It needs a proper diagnosis to know the right treatment.

Anything else one could provide that would help you in your quest?

I found this software, same purpose as RAMmap.

Maybe this will come in handy if the solution is indeed to perform the emptying actions after every turn for example.

I started a new game with giant map and maximum graphic details; I reached the year 915 and 2.8MB savegame size, and I got no MAF so far.

Yeah those shouldn't really happen until your save file reaches about double that size, at least that's around the size when the crashes started happening for me. They got too frequent around 6-7MB I believe, which is when I switched to low graphic settings and that (in combination with Project Lasso) reduced them to an acceptable frequency.

 

[Sinocpm]

Quick update:
Using RAMmap after every turn did not help, still crash on turn 2.

However funny enough I tried RAMmap after the first term, plus Project Lasso running on medium graphic settings (except for high detail terrain turned on), and that seemed to work pretty well! I was able to play without a crash for just under an hour or so when I had to stop.
Just now I even tried the same setup on high settings (except anit-aliasing off, because with settings on high and that on it seems to crash after turn 1 or 2) and was able to play 6? turns or so until a crash.

 

[Chanterive]

Anything else one could provide that would help you in your quest?

Actually, your save helps because it's more extreme than mine.

I finally managed to reinstall Civ. Even the Steam version wasn’t working (!), but I found another ISO somewhere on the internet, and it's now okay.

First, I'm curious: when loading your save, I see something that looks like a city radius, except sometimes it's not centered on any city, and I can't deactivate it. What is that?

Spoiler :

I used more specialized software for diagnostics, and in your case, it's the virtual memory hitting the ceiling. What we saw in Task Manager was probably something else.

Here, I'm using VMmap (Virtual Memory Map), while RAMmap was showing what's going on in the physical RAM.

You can see that just after loading (with full graphics), the total size is close to 4GB (3,913,792 KB = 3.73GB), with only 280,448 KB of free space because it's 32-bit.

And after several turns, when the MAF happens, the virtual memory reaches 4,156,660 KB (3.974 GB) :

Then the process requests some free memory, can't find it, and crashes. RAMMap probably helps with some of the memory leaks. Also, when emptying the RAM, it forces memory to defragment a bit, creating larger chunks of free space, while the total virtual memory remains the same. But here, it's not enough.

As you can see, the two biggest memory areas are Heap and Mapped Files:

• Heap: Dynamically allocated memory used by the program to store variables and objects during execution. It is private to the process and can cause fragmentation issues if not managed properly.
• Mapped Files: Memory regions used to load files directly into RAM, allowing fast access without traditional read/write operations. They can be shared between processes.

Civ4 loads all the FPK files into memory (mainly textures, I think). That's not good. They get larger with each RI version. I don't know if modders have a way to compress them more, but that would be good and easy.

It's theoretically possible to force a software not to load certain files into memory, but that's not easy and requires special development since we don't have access to the source code. (I will try!)

Also in the Civilization4.ini, there's an option
; Disable PAK memory mapping (May affect performance)
DisablePAKMemoryMapping = 0

I think it's exactly what we need, but it doesn't change anything!!! Here too, maybe Walter knows a way to (re)activate that option for real ? Is PAK same thing as FPK ?

From the GPU side, with full graphics, we are not far from 2GB.

I would suggest to put 4096Mo in dgVoodoo, just to be sure.

So, i will try some devs and keep you informed !

 

[Chanterive]

I came up with something.

I decompress all the .FPK from \Beyond the Sword\Mods\Realism Invictus\Assets, except for RealisticTerrain.FPK.

Then i recompress them in one sigle .FPK (that can still be splitted, but thats one archive)

It went from 855MB to 477MB. Almost 400 less MB. I think that's the easy way to go, if it does save up 400MB, that will be a nice bonus for endgame.

Those FPK where not present in original version, they are part of the mod. I would like to know how i can made necessary changes to use the new, big FPK, instead of many small ones. Some path must be changed, somewhere.

Also, in some other FPK, some huge files seem useless. For example, in assest.fpk we find some "motionflow_aircraft.tga" 8MB. If i undestand well it's simply produce a jpg, "motionflow_aircraft.jpg"

Spoiler :

It's probably nice to have for modders, but it takes a lot of place in a Release version that suffers memory limitation. Maybe some additional "legacy" / useless files could be deleted too.

 

[Walter Hawkwood]

Then i recompress them in one sigle .FPK (that can still be splitted, but thats one archive)

It went from 855MB to 477MB. Almost 400 less MB. I think that's the easy way to go, if it does save up 400MB, that will be a nice bonus for endgame.

You "recompress" as in actually using compression? And it runs ok afterwards? Because from what I know, the fpk's for Civ 4 are not supposed to be using any compression. And in any case, they'd have to be decompressed back to be used by the game engine - so how does that impact the actual RAM use?

 

[Chanterive]

You "recompress" as in actually using compression? And it runs ok afterwards? Because from what I know, the fpk's for Civ 4 are not supposed to be using any compression. And in any case, they'd have to be decompressed back to be used by the game engine - so how does that impact the actual RAM use?

I used PakBuild.exe from Firaxis to open several FPK files and then rebuild them into one. The compression I used is the one provided by PakBuild.

I have zero knowledge about PakBuild and PAK/FPK.

Can I ask, how do you usually build FPK files ? How do you make them recognized by Civ ? In my case, they are not being loaded, as if they are not considered properly formatted files.

Also, do you know about the "DisablePAKMemoryMapping" option ? Did it ever actually do anything, and could it potentially be re-enabled ? Because as you can see from previous screenshot, all the FPK are being memory mapped. It was necesserary on old HDD, but today NVMe is faster than 20 years old RAM.

 

[Walter Hawkwood]

I can't help but feel relief - while it would have been great to have the advantages you've described, it would have meant I was going about it wrong all this time.

The proper settings to build fpks that will be recognized by the engine are "store full paths" and disabling any compression (setting compression level to zero). Since, from what I understand, an fpk is loaded into memory as a whole, it is a good idea to split them into "digestible" chunks - from my experience I found going over 50Mb exacerbates memory problems. In recent versions, I also try packing together stuff that would logically be loaded together, such as all units for a certain civ. Also, from what I understand, NOT packing stuff into fpks at all is best for memory use, but with lots of assets, it takes forever to load.

As for "DisablePAKMemoryMapping", I have no idea - feel free to experiment and see if it is of any benefit. I'd say if it solved the memory problems, it's likely that someone would have pointed that out already, but I'm happy to be mistaken.

 

[Chanterive]

Then, without compression, there are no benefits, unfortunately.

I tried DisablePAKMemoryMapping on Beyond the Sword without any mods—just to check if it ever did anything—and monitored memory usage with various tools. I didn't observe any difference.

However, I noticed something interesting: when you delete some of the FPK files, the game still runs fine, and the affected units simply become invisible, without any apparent bugs (I tested this for only two turns). I assume it's just textures, so if they’re missing, they’re simply not displayed.

This suggests that, in the endgame with only a few civilizations left, you could delete (after backup!) the FPK files of civs that are no longer in the game, potentially saving hundreds of MB to help finish your game.

Following the same logic, we could imagine FPK files being divided by civilization and era (or other criteria). That way, in the late game, we could simply remove everything that isn’t modern, keeping only the assets currently in use. It’s not the cleanest solution, but if it helps people finish their games…

I’ll keep looking for a way to trick Civ into not loading the FPK files into memory.

 

[Chanterive]

I have an additional question: why the massive use of FPK files ?

I'm trying to understand how BTS works without mods. It seems that textures are loaded directly from the file system.

For example, \Beyond the Sword\Assets\Art\Units\Charlemagne_HeavyCavalry\HeavyCavalry_128_Gloss.dds. In RI, however, these textures are packed into FPK files.

I'm not entirely sure yet, but in this case, I get the impression that only the necessary assets are loaded into memory.

The philosophy behind FPK files seems to be that core assets are always kept in memory, non-negotiable, because they are needed all the time, whereas files stored directly in the file system are managed dynamically, loading them as needed.

---

I tried it, i unpacked all RI_Units*.FPK file back into assets\art\units\

It took very long to load the game (why?)

Et voilà Over 700MB of free memory :

I played only several turns, didnt notice anything unusual. I havent touch the Structures*.FPK and Interface*.FPK, so there's margin for more.

 

[Walter Hawkwood]

I have an additional question: why the massive use of FPK files ?

I'm trying to understand how BTS works without mods. It seems that textures are loaded directly from the file system.

For example, \Beyond the Sword\Assets\Art\Units\Charlemagne_HeavyCavalry\HeavyCavalry_128_Gloss.dds. In RI, however, these textures are packed into FPK files.

I'm not entirely sure yet, but in this case, I get the impression that only the necessary assets are loaded into memory.

The philosophy behind FPK files seems to be that core assets are always kept in memory, non-negotiable, because they are needed all the time, whereas files stored directly in the file system are managed dynamically, loading them as needed.

I tried it, i unpacked all RI_Units*.FPK file back into assets\art\units\

It took very long to load the game (why?)

You basically just answered yourself - fpk files are used because loading times are really long otherwise. Vanilla assets are packed, just specifically BtS ones that came later are unpacked (I highly suspect that BtS was made by a different internal team or even a subcontractor, as a lot of assets "feel" very different from technical PoV, so they might have seen it works without packing and just called it a day). Using unpacked assets is ok while you don't have a lot, such as in the case of BtS, or if you don't mind long loading times (my guess is that Civ 4 engine creates a checksum of all the files inside its folders for MP compatibility or something like that, and creating that checksum takes time proportionate to the number of files). SVN version of RI is unpacked, and while it takes 10-15 minutes to load, I personally never experienced any memory-related issues while running it.

 

[Chanterive]

SVN version of RI is unpacked, and while it takes 10-15 minutes to load, I personally never experienced any memory-related issues while running it.

That packing is the reason people experience memory issues (in addition to poor memory management that was simply not designed for such a heavy mod).

Here’s an example: I went to the extreme and unloaded every .fpk file, including those from BTS, since they are loaded as well.

I then loaded a very large game (sent by Sinocpm—the save file is over 8MB) with 1.4GB of free space, using full graphics and high resolution. Before, the game was loading with only 300MB of free space, crashing very quickly (since the first turns after loading always take an additional 100MB each).

I could play 30 turns experiencing no crash, without using RAMMap or Process Lasso, but with dgVoodoo on (because it feels more fluid, so I'm keeping it). Then i went bed, seems stable. Before it was crashing on the 2nd turn.

Notice, "Mapped file" usage is now down to 29MB, whereas before, it was over 1.2GB. All of that within a 4GB limit—that's a massive memory reduction!

My loading time is about 6 minutes, but when you're planning to play for several hours, that's acceptable. And when the game crashes, you've probably already been playing for dozens of hours, so it's just a small inconvenience to continue. When you play civ you have time. I also think loading got faster after deleting the .fpk files (if not, then there's both indexing/checking of unpacked and packed files happening, but somehow the game understands and doesn't load the fpk).


So, the simple solution to finally fix the MAF issue would be to release an additional unpacked version that players could switch to in the endgame. With all the necessary warning about loading time. It would maybe be possible to have the unpacked version not overwriting the packed one, if placed in a different unpacked mod. Saves are compatible.

What do you think ?

 

[Walter Hawkwood]

I won't be "releasing" the unpacked version - it can be already checked out from the SVN repository, with an instruction on how to do that available as well, so it is available already. Alternatively, one can simply use Pakbuild to unpack the release version should one so desire. There is simply no reason to release something that is the same as the SVN version (which has the additional advantage of being up-to-date at all times, not just at release). Also, you'd be surprised (or not) by the average user, who'd be constantly reporting the fact that the unpacked version loads too slow (or "doesn't load at all", if the user in question has the patience for only 1-2 minutes), which would drive me insane - so I don't want to make the unpacked version "official" in any way.

But yes, this matches my experience that the unpacked version doesn't have memory issues (or has a much more subdued version thereof).

 

[[Y]]

And for what it's worth, civ 4 development best practices were developed for hardware in 2005, and computer performance improved a lot since then. Even 10 years ago when booting up an unpacked RI, I would go off and do something else for 15-20 minute. These days, on my M2 macbook, it's only 3-5 minutes to boot unpacked, which is much more tolerable.

 

[Walter Hawkwood]

Yeah, it does boot much faster on a modern SSD than on an old HDD, that's for sure. Still, I wouldn't trust the average user with an unpacked version - which is also why (as I sometimes get asked) I only provide installers, not a zipped-up version. I can trust an installer to put the mod where it needs to be; I can't say the same about some users. Present company (and generally most people posting here on CFC) obviously excluded, but dealing with a subset of users who have absolutely no clue about anything is an unfortunate downside of having a popular mod.

 

[Sinocpm]

Sorry I have not been able to play or even answer recently, so finally a quick reply:

Actually, your save helps because it's more extreme than mine.

Glad to hear it, I think that's also the furthest I've ever pushed a game of RI or probably Civ in general for that matter.

I finally managed to reinstall Civ. Even the Steam version wasn’t working (!), but I found another ISO somewhere on the internet, and it's now okay.

That is... weird? That the Steam version wasn't working for you I mean. You mean you weren't even able to run/start the freshly installed Steam version of BtS without any modifications? Maybe there were/are some leftover files somewhere from an old installation or something?

First, I'm curious: when loading your save, I see something that looks like a city radius, except sometimes it's not centered on any city, and I can't deactivate it. What is that?

What you are seeing and describing are the city radiuses you can manually place py pressing ALT+X. CTRL+X toggles them on and off. I use them to plan my (future) city placements. However what is strange in your screenshot is that none of the cities that are present there are displayed by any buildings etc. What's going on there?

I would suggest to put 4096Mo in dgVoodoo, just to be sure.

Yeah I already figured that might help so I went ahead and already selected that.

As for your interesting further experimentation I very much applaud you for that but I'm afraid I can provide pretty much 0 useful input here, as that is not something I am very familiar with or knowledgable of

However the possibility of using an unpacked SVN to continue a late stage game on high graphic settings sounds promising! So if I understand this correctly I can grab the 3.7 SVN (since my huge game is played on an official 3.7 release version), install that version parallel to my official 3.7 version and then load my save and continue, correct? Sorry for the silly question, but where would I find the SVN versions? I only see the official ones on Sourceforge... Thank you!

 

[Walter Hawkwood]

However the possibility of using an unpacked SVN to continue a late stage game on high graphic settings sounds promising! So if I understand this correctly I can grab the 3.7 SVN (since my huge game is played on an official 3.7 release version), install that version parallel to my official 3.7 version and then load my save and continue, correct?

No, as the mod names are different (deliberately from my side, for me to be able to have both side by side). You'll need a save started on the SVN version for that.

Sorry for the silly question, but where would I find the SVN versions? I only see the official ones on Sourceforge... Thank you!

See the SVN instructions at the end of the first post: https://forums.civfanatics.com/threads/realism-invictus.411799/

 

[[Y]]

Yeah, it does boot much faster on a modern SSD than on an old HDD, that's for sure. Still, I wouldn't trust the average user with an unpacked version - which is also why (as I sometimes get asked) I only provide installers, not a zipped-up version. I can trust an installer to put the mod where it needs to be; I can't say the same about some users. Present company (and generally most people posting here on CFC) obviously excluded, but dealing with a subset of users who have absolutely no clue about anything is an unfortunate downside of having a popular mod.

Absolutely. The point of my post was to give Chanterive more context on why the FPK files are relevant. I was playing RI on the computer that took 15-20 minutes to boot up until about two years ago.

Even right now, I'm dreading art updates because my current setup is using the SVN version, but removing all the art files and using the FPKs from 3.7 instead. My games don't often get to the point crashing from memory, so faster boot times is just more valuable to me.

 

[Walter Hawkwood]

Even right now, I'm dreading art updates

And art updates you'll soon get, mwahaha! I'm finally done with 3.7x update cycle, and now I'm back to my favourite pastime, which is painting my tin soldiers!