Memory Allocation Crashes

[Lantarh]

Hello, I did some tests.
I played a Huge Earth game on RI 3.72 until 1723AD, when I got the first crash. Save size is 5661KB and load time is around 45 seconds on my system.

Without any action, that's the memory situation immediately after loading the save:

I then created a PowerShell script that unpack the RI FPK and moves all the assets to the BtS folder (after backuping the Assets, both in the BtS and RI folders).

Spoiler Script :

# Personalize these
# You have to place PakBuild.exe from http://www.civfanatics.net/downloads/civ4/utility/PakBuild.zip inside the $PakBuilder_Path folder
$BtS_Path = 'C:\Program Files (x86)\Steam\steamapps\common\Sid Meier''s Civilization IV Beyond the Sword\Beyond the Sword'
$RI_Path = 'C:\Program Files (x86)\Steam\steamapps\common\Sid Meier''s Civilization IV Beyond the Sword\Beyond the Sword\Mods\Realism Invictus'
$PakBuilder_Path = 'C:\Users\User\Desktop\tests'

# Don't touch these
$BtS_Assets_Path = $BtS_Path + '\Assets'
$BtS_Assets_Path_Backup = $BtS_Path + "\Assets_Backup"

$RI_Assets_Path = $RI_Path + '\Assets'
$RI_Assets_Path_Backup = $RI_Path + '\Assets_Backup'
$RI_Assets_Path_Unpacked = 'C:\Assets_Unpacked'
$RI_Assets_Path_Unpacked_Art = 'C:\Assets_Unpacked\art'

$PakBuilder_Exe = 'C:\Users\User\Desktop\tests' + '\PakBuild.exe'
$PakBuilder_Input = '/I="' + $RI_Assets_Path + '"'
$PakBuilder_Output = '/O="' + $RI_Assets_Path_Unpacked + '"'
$PakBuilder_Options = '/U'

# Backup Asset folders in BtS and RI
Copy-Item -Path $BtS_Assets_Path -Destination $BtS_Assets_Path_Backup -Recurse
Copy-Item -Path $RI_Assets_Path -Destination $RI_Assets_Path_Backup -Recurse

# Unpack RI FPK file
Start-Process -FilePath $PakBuilder_Exe -ArgumentList $PakBuilder_Input, $PakBuilder_Output, $PakBuilder_Options -NoNewWindow -Wait

# Move all assets to BtS folder and cleanup the rest
Get-ChildItem -Path $RI_Assets_Path *.FPK | foreach { Remove-Item -Path $_.FullName }
Copy-Item -Path $RI_Assets_Path_Unpacked_Art -Destination $RI_Assets_Path -Recurse -Force
Remove-Item $RI_Assets_Path_Unpacked -Recurse -Force
Copy-Item -Path $RI_Assets_Path -Destination $BtS_Path -Recurse -Force
Remove-Item $RI_Assets_Path -Recurse -Force

After running the script, I launched again the save. Load times are the same, but the memory situation is drastically improved:

I'll play it further and will update you.

 

[Nightinggale]

Reading those numbers it seems that the difference is mapped files. In other words it seems that the exe unpacks the packed files and stores each file as a non-persistent file in virtual memory. This way when accessing a file, it can be done with open file code, be it from disk or memory.

So we can pack files to get faster startup time, but at the same time it is likely the mapping of files to track what is packed and what isn't, which is slowing down startup. Either way it seems the game keeps around 900 MB of files in memory when using packed files, so it's not really an option to do that given the 4 GB memory limit.

 

[Lantarh]

I only use RI, so merging all the assets inside BtS is not an issue.

For other players I think the solution could be researched mixing two things:

1. a RamDisk that presents the consolidated BtS+Mod assets folder
2. a wrapper (along the lines of Save-over-HTTP) that intercept call to the real BtS and Mods Assets folders and redirect them to the RamDisk

Unfortunately, that would be a bit too much for me to program...

 

[Nightinggale]

Unfortunately, that would be a bit too much for me to program...

Understandable. It seems that Save-over-HTTP relies on MinHook, but since the mapped files are generated prior to the DLL is even attached, using MinHook inside the DLL to alter how the exe behaves is pointless as it's too late. The mapped files will already have been created at this point. In other words it's not possible to do from within the MODS folder.

 

[Lantarh]

Uhm, I'm sure there are other options for wrapping the BtS exe and remapping I/O, but yes, it would be less elegant than doing that from the mod itself (as you'll need to switch manually the wrapper for each mod)

 

[Walter Hawkwood]

This would be a nightmare between versions of RI as well - files don't just get added, they get renamed, moved and deleted too. Updating a version would create a nasty mutant composed of both versions' assets.