why doesn't civ fanatics allowAnd art updates you'll soon get, mwahaha! I'm finally done with 3.7x update cycle, and now I'm back to my favourite pastime, which is painting my tin soldiers!
as a reaction? 
Memory Allocation Crashes
- Thread starter Cruor
- Start date
as a reaction? Oh, I see
So I guess that's not an option. No way around that?Also I was wondering how a mod like Caveman2Cosmos handles this kind of problem, since that seems to be a mod of even greater scope/scale. On their mod page it highlights the option "Graphical Paging", which should virtually erase MAF problems (or so it says there). I know RI has that option too, but whenever I tried using it before it seemed to cause even more frequent crashes. Is that a common experience? Any tips on how to better utilise this option?
Walter Hawkwood
RI Curator
You can unpack the fpk files on your release version if you're attached to the particular save. Instead of deleting them, you can even put them somewhere else temporarily.
I don't think it ever worked in RI - it was ported a long long time ago, way before I started meddling with code stuff, and I have absolutely no idea what's wrong with it or even how it's supposed to work in the first place.
Asking naively and not meaning to opine on the extent or urgency of the problem: Might packing only small files accomplish the best of both worlds in terms of memory use and startup time? Inspired by this post,
90 KB struck me as a sweet spot. Not all too sweet, 670 MB is already 40% of the total. (To which RealisticTerrain adds another 48.2 MB.) Then I generated another script to move the files smaller than 90 KB into a separate folder hierarchy and packed that into a single FPK of 672 MB. The remaining 990 MB unpacked, 3,733 files. I wouldn't really know how to test the memory use of this scheme. The startup time was under 1 minute until the splash screen came up – on a 15-year old PC that never was quite new. Then about another minute for XML loading. Subsequent starts are taking fewer than 15 seconds until the splash screen (mysterious as ever where the caching happens that provides this speed-up). I don't know if this is really a promising result. Perhaps was worth a try. At least I've now updated my local copy of RI. 
... I've just downloaded the latest snapshot from SourceForge and ran an AI-generated script to find out how many files are smaller than a given number of kilobytes. For reference, the total number of art files was close to 32,600 and the total file size 1.62 GB. Here's what I got from the script:
Spoiler :
Walter Hawkwood
RI Curator
It's an interesting approach and I will keep it in mind for the next release - but for people in this thread, it would actually be cool if you ran the tests on whether it results in good memory savings and tolerable load times in the meantime (no rush, you have until Christmas!) - maybe even experiment to find the optimal size ratio. BTW, forgot to mention another benefit of packing - Civ 4 engine seems to have trouble unloading lots of loose assets all at once, so if you exit from a running game to main menu with unpacked assets, you'll more often than not get a CTD.
I see, thank you for the suggestion. I think I will just continue playing on medium or low graphic settings in that case, as that seems to be a bit too much effort for what I'm getting out of it.
But you are Walter, the immortal creator, wandering the forum since time immemorable! How can that be?!
I think I will continue my huge save game for a bit longer, the save file I already posted here, and if I can be of use as a guinea pig for further experimentation I'd be happy to help.
Chanterive
Chieftain
Sorry, I’ve been very busy lately—freelancing, though unfortunately (or fortunately?) not with a lance, but with software.I see, thank you for the suggestion. I think I will just continue playing on medium or low graphic settings in that case, as that seems to be a bit too much effort for what I'm getting out of it.
But you are Walter, the immortal creator, wandering the forum since time immemorable! How can that be?!
I think I will continue my huge save game for a bit longer, the save file I already posted here, and if I can be of use as a guinea pig for further experimentation I'd be happy to help.
Unpacking FPK and copying the output to the right place isn’t that complicated. I’ll try to create a script to automate the process. I think it’s worth it because it would allow to play the endgame with full graphics. And my friends and I could enjoy it too.
Download VMMap from Sysinternals, launch the first executable (not the 64-bit version) with administrator rights. Select the Civ4 process and check the Free section—the more, the better. With the right tool, it’s easy.
The issue lies in Mapped Files—this is where you’ll see which files are mapped into memory and their sizes.
I still believe that providing an unpacked, separated version as an optional endgame solution, with all the necessary warnings, would be nice. Even though a solution exists, the number of players who will come here to find it is extremely low, and it requires some technical knowledge. Also, since SVN is up to date, it can't be used to fix an existing save. Maybe a cleaner solution could come up later.
Many players have stopped playing this awesome mod due to MAF errors (I love big maps). When you invest 20 hours into a game only to be unable to finish it, it's frustrating.
At launch, the game scans every single file and creates an output in: AppData\Local\My Games\Beyond the Sword\cache
It opens, reads, and closes each file individually and sequentially, which takes an unnecessarily long time since there are tens of thousands of files. The output is below 3MB and consists of several .dat files. What could be in this files ?
I tried making them read-only to prevent them from being rewritten at every launch, but the scanning still occurs—proving that it is useless, except perhaps the first time.
Adding a DLL to prevent this (probably unnecessary) scan would be nice.
Or maybe modders already know which piece of code is responsible for triggering this scan ?
There are CIV4...Infos.dat files; that's the XML cache, which seems to get created and read only when launching without a mod. And there are catalog...dat files, which are probably the ones you mean. Opening them with a hex editor (or even a text editor) shows mostly file paths in ASCII. Likely being mapped to some other type of data. DisableFileCaching in CivilizationIV.ini sounds like it should suppress those files, but it has no apparent effect. This thread is probably the most productive investigation of this subject so far. It suggests that there is a scan of files with .dds and .nif ending, but it doesn't connect that scan to the catalog files. But your post sounds like you've been able to ascertain that by observing file access patterns? I guess this might be relevant when trying to find the binary code in the EXE that performs the scan. However:
The DLL being loaded only once the scan is done will make even a runtime hack of the EXE through the DLL impossible. One could at best alter the EXE on disk.
For the time being, one could also browse to the proper revision on the SourceForge page and to the Art folder and use the website's "Download Snapshot" function. One may have to resume the download manually a couple of times if it times out.
Thanks, but I feel that I should ultimately also test some large savegame with MAF errors to weigh in on which way of packing works best. That gets too laborious for me, at least for now.
What's discouraging about packing only small files is that the relationship between the number of unpacked files and the scan time is apparently linear. So packing just, say, 80% of the files will only reduce the start-up delay to one fifth. Given how long it takes (when launching the mod for the first time) to scan all files, the duration with partial packing will never be negligible. And the game showing no splash screen or other sign of life for nearly a minute is indeed a pretty bad way to greet new players.
Random thought about Process Lasso: Supposedly, the windows.h function SetProcessWorkingSetSize can do the same or a similar thing as the "trimming" by Process Lasso (edit: or maybe rather EmptyWorkingSet). I doubt that the more advanced optimization techniques mentioned in this thread could easily be implemented by the DLL, but, if well-timed trimming already goes a long way, then integrating that into the DLL might save players the trouble of seeking out third-party apps.
Last edited:
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Actually it seems to not be linear based on my observations. The reason is likely explained here: https://forums.civfanatics.com/thre...mecoredll-to-run-faster.686918/#post-16537892
(at least that is the working theory right now). Having said that, I will discourage packing files anyway because it makes the game less stable. Packing on one computer might work while the same packed mod might crash at startup on another computer. Packing files is not super stable and we have run into issues like packed files working only on 64 bit while unpacked working on 32 bit windows. However we have also observed that packed files can fail on one 64 bit computer and work on another one despite them running the same version of windows, so no idea why that happened.
On a side note, graphics in WTP doesn't work on 32 bit computers for unknown reason, so we have declared it a 64 bit only mod despite both DLL and EXE being 32 bit binaries.
Oh, two times 9 is 18, not 28. I misinterpreted this:
Although it might scale differently on the first launch. That, in my test, took nearly a minute with 3733 files, 11.5% of the total number. How long would it have taken with everything unpacked? Difficult to test now; 10 minutes seemed plausible to me based on past experiences with RI. But 15 minutes I could also imagine. For reference:
Well, if inefficient look-up is what makes it superlinear, then I'd expect that to make less of a difference on the first launch.
Chanterive
Chieftain
Ok, I have something working.
I noticed that unpacking the FPK from BTS doesn’t change the loading time... So, I realized that the long scanning process is actually comparing what's in the BTS Assets folder and what's in the RI Assets folder. If a file exists in the mod, it chooses the mod's version, if not it keeps the BTS version. Then probably creates a list of paths in .dat files.
So, after unpacking all the FPKs, I copied everything from the RI Assets folder into the BTS Assets folder, and then deleted the Assets folder from the mod !
Result: The game launches immediately, with all available free memory ! It lauches instantly, much faster than the regular RI. Of course, this messes up your normal BTS game, and other mods will not work, and BTS will not work since you're merging BTS and RI. So that's dirty, be sure to have everything needed to reinstall the full game.
But it does loads instantly and has absolutely all the free memory needed, as long as you play only RI
Also i only play 2 turns, so let's see if it's stable.
I noticed that unpacking the FPK from BTS doesn’t change the loading time... So, I realized that the long scanning process is actually comparing what's in the BTS Assets folder and what's in the RI Assets folder. If a file exists in the mod, it chooses the mod's version, if not it keeps the BTS version. Then probably creates a list of paths in .dat files.
So, after unpacking all the FPKs, I copied everything from the RI Assets folder into the BTS Assets folder, and then deleted the Assets folder from the mod !
Result: The game launches immediately, with all available free memory ! It lauches instantly, much faster than the regular RI. Of course, this messes up your normal BTS game, and other mods will not work, and BTS will not work since you're merging BTS and RI. So that's dirty, be sure to have everything needed to reinstall the full game.
But it does loads instantly and has absolutely all the free memory needed, as long as you play only RI
Also i only play 2 turns, so let's see if it's stable.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Looking through the thread, a lot of memory numbers have been posted, so just to be clear: what is memory usage as a mod vs copying everything into the game itself?
On a site note, it seems that the computer doesn't mind having more than one install of civ4colo (and presumably BTS), and just copying the folder of the GOG install seems to work. Telling people that having one install for each mod seems doable, at least as an option for people, who are seeking the faster load time.
Chanterive
Chieftain
What's saves memory is the unpacking of FPK files into the Assets folder. If you do both BTS FPK + RI FPK, the saving is aroung 1.2GB (on 4GB available). 850MB for RI alone. Each new version of RI being naturaly heavier.
Then copying mod Assets folder into BTS Assets folder solves the loading time issue. I don't know if there's other benefits or cons.
I played several more turns (new game, but start in future era since i install RI 3.72), works like a charm.
I agree that apparently only files from mods get scanned, and this is interesting news. Just moving/merging (anyone trying this should make backups) my partial RI Art folder (the 3733 unpacked files) to BtS\Assets\Art removes the initial delay, at least the one upon launching repeatedly. However, loading the Art via CustomAssets also does the trick. So I doubt that the file comparison hypothesis is correct. I used to think that a mod can block custom XML and Python assets while allowing custom art, but it seems that it's the other way around. So Realism.ini will need these two changes to enable custom art:
Keeping the RI art in CustomAssets permanently can interfere with other mods. The ini files that the EXE generates allow all custom assets. One can rename the folder to e.g. "RIArt" in CustomAssets when not playing RI. A launch script that waits for RI to exit could handle the renaming:
Code:
; Custom Art from user folder is not loaded
NoCustomArt = 0
; Custom XML and Python from user folder are not loaded
NoCustomAssets = 0Spoiler :
Chanterive
Chieftain
Whats CustomAssets and how does it work ?
Because now we have everything for both fast loading and no MAF, it works great on my computer. It's much more tempting to launch the game when there's no loading time and you know you will be able to finish your game !
Maybe it's possible to come up with something clean. I never modded civ4 (actually i did add new leaders being my friends and me !)
There's usually a Windows shortcut to the CustomAssets folder in the BtS install dir – though perhaps not for all editions of BtS. The folder should be under My Games\Beyond the Sword or something like that, where also CivilizationIV.ini is located. My understanding is that it's for asset changes that a player wants to apply regardless of whether a mod is loaded. Like changing the original files but without actually altering one's install folder. I haven't used CustomAssets much, but iirc the DLL can also be loaded from there. Some players certainly install BUG into CustomAssets, which keeps their savegames compatible with unmodified BtS (no mod name gets written into the savegame). But I think BULL (BUG plus DLL changes) also can go into CustomAssets. Blue Marble perhaps also gets placed in CustomAssets somewhat commonly. Mods can specify in their .ini file whether they want to block the loading of CustomAssets. I assume that the majority of mods do not block them though perhaps the majority of large, mature mods do. Load order should be base game before Warlords before BtS before CustomAssets before the mod before the mod's modules.
Aradziem
Warlord
- Joined
- Jul 14, 2021
- Messages
- 131
I don't know much about making programs (I can make simple ones at most), so my suggestion may be weird.
Is it possible to load some assets from folder selected by DLL (i.e.when you load a mod, exe loads files that are required, then when some more are needed because you e.g. load game DLL loads and manages files from folder different than standard using a much faster method)?
Is it possible to load some assets from folder selected by DLL (i.e.when you load a mod, exe loads files that are required, then when some more are needed because you e.g. load game DLL loads and manages files from folder different than standard using a much faster method)?
CivUser1800
Chieftain
- Joined
- Mar 8, 2025
- Messages
- 1
Created an account just to comment here - as I've discovered this mod and love it. Immediately went to play on gigantic map size with max factions & spawning (so 20+). My first game as Russia though got to ~1000AD on recomended speed and keeps carshing on the attached turn.
Is this memory related? If unpacking the files will solve this issue and I deal with a 15min startup time that's no skin off my back when I can just leave the game on after work and play turns as needed. Better than losing 20+ hours put into a save game.
Is this memory related? If unpacking the files will solve this issue and I deal with a 15min startup time that's no skin off my back when I can just leave the game on after work and play turns as needed. Better than losing 20+ hours put into a save game.
@CivUser1800: I can't easily check because the random SVN version I have now won't be save-compatible with anyone. (I also wouldn't know what's memory-related unless an error message explicitly says so.) You could try just moving all FPKs out of the Assets folder, maybe simply one level up, directly into the mod folder; I don't think they get loaded from there. If that gives you the same crashes as before, then I don't think unpacking could help you. For reference:
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I store the git revision as a string in an autogenerated header file. That string is only used as the very first variable in a savegame. That way when I open a savegame file in notepad++, the revision used when saving will be on the first line in plain text. This really helps when getting savegames from bug reports as it is clear which version should be able to load it. I imagine it can be done with svn too. In fact it should be even easier as svn has a simple revision numbering system rather than the git hash key.
