FPKing files for faster mod loading times

[Sword_Of_Geddon]

For some people my mod takes about twenty minutes or so to load. I know FPKing files can drasticly reduce this. Does anyone know what program to use to implement this?

Thankyou in advance.

 

[Dancing Hoskuld]

PAKBuild is the program.

I am not sure it does reduce load time. It does make your mod smaller so decreases download time.

 

[Nightinggale]

I am not sure it does reduce load time. It does make your mod smaller so decreases download time.

I haven't managed to get the compression to work, meaning the only change to filesize is that it can put multiple small files in the same disk block. That will not really change anything with graphic files as they are so big relative to the blocks.

It will increase load time significantly because on start, the game opens a file, copies the contents into memory, closes it and moves on to the next. The open and close files produces a significant overhead in the OS itself and using a packed file means you only open it once according to the OS itself.

I will recommend against using packed files in svn/git or similar because it makes modding them somewhat harder. Packed files are for releases only.

Another issue is that it appears that unpacking at startup will use more memory. At least there is some memory related issues as packed files can cause the game to crash while unpacked files works just fine. To my knowledge crashes due to this has only been observed on 32 bit systems.

There is a claim that packed files are faster to load, but slower during the game where the performance is most needed. I haven't been able to confirm this, though not packing movie files is likely a good idea. It's also possible that file access will be faster if you reduce the number of files in each packed file and you then make say 10 of them. It all depends on how it stores the table of contents and how quick it can search it.

 

[Sword_Of_Geddon]

So it does reduce loading times, but only if there are multiple small fpk files?

I recall that Mamba's Diversica mod had drasticly reduced his loading times by using packing.

 

[Xyth]

Using FPKs will make your mod launch considerably faster, as it's quicker to load a few big files to memory than it is to load thousands of small ones. They will not make your mod smaller, as Civ4 doesn't seem to cope with PakBuild's compression option.

Using FPKs is not slower or faster in-game than not using them. However, using FPKs does seem to restrict Civ4's ability to manage memory, making MAFs more likely on larger maps. So there is a significant trade-off.

My advice (and what I do for my mod), is to have 2 or 3 FPKs: one for Art/Terrain, one containing both Art/Shared and Art/Structures, and (maybe) one for Art/Units. Leave everything else unpacked. I've done a lot of experimenting and this gives me the best balance of launch time and memory usage. I say maybe for the Unit art, because it depends how much your mod has. For most, it'll be the single biggest contributor to slow launch times and MAFs alike. I pack it when I release a new version, but tell those that prefer playing on large maps or with 18+ civs to unpack it (and expect much slower launch time as a result).

For reference, my mod has ~230MB of unit art. It used to be nearly 700MB but I did a massive clean up of my units folder a while back – deleting unused textures and models, baking a lot of textures together (why use 5 textures when it could all fit on 1 or 2, grr), even deleting unused alpha channels (presence of an alpha channel doubles the size of a .dds file regardless of whether it's used). Took weeks, but worth the half a GB of memory saved.

Important: Never put leaderhead art into an FPK. Leader art is loading from disk on demand, not loaded into memory at game launch. But an FPK containing leader art does get loaded into memory, meaning that you reduce available memory (increasing the likelihood of MAFs) for no benefit whatsoever.

 

[Leoreth]

Using FPKs will make your mod launch considerably faster, as it's quicker to load a few big files to memory than it is to load thousands of small ones. They will not make your mod smaller, as Civ4 doesn't seem to cope with PakBuild's compression option.

Using FPKs is not slower or faster in-game than not using them. However, using FPKs does seem to restrict Civ4's ability to manage memory, making MAFs more likely on larger maps. So there is a significant trade-off.

My advice (and what I do for my mod), is to have 2 or 3 FPKs: one for Art/Terrain, one containing both Art/Shared and Art/Structures, and (maybe) one for Art/Units. Leave everything else unpacked. I've done a lot of experimenting and this gives me the best balance of launch time and memory usage. I say maybe for the Unit art, because it depends how much your mod has. For most, it'll be the single biggest contributor to slow launch times and MAFs alike. I pack it when I release a new version, but tell those that prefer playing on large maps or with 18+ civs to unpack it (and expect much slower launch time as a result).

For reference, my mod has ~230MB of unit art. It used to be nearly 700MB but I did a massive clean up of my units folder a while back – deleting unused textures and models, baking a lot of textures together (why use 5 textures when it could all fit on 1 or 2, grr), even deleting unused alpha channels (presence of an alpha channel doubles the size of a .dds file regardless of whether it's used). Took weeks, but worth the half a GB of memory saved.

Important: Never put leaderhead art into an FPK. Leader art is loading from disk on demand, not loaded into memory at game launch. But an FPK containing leader art does get loaded into memory, meaning that you reduce available memory (increasing the likelihood of MAFs) for no benefit whatsoever.

Wow, that's a lot of information that I did not know or never really considered. Given that people with sufficient memory still run into MAFs with my mod that I can't really explain this could very well be the reason. Looks like I also have to reconsolidate my FPKs.

What you're saying also seems to apply to Varietas Delectat. Is that a mod that also should not come with FPKs in your opinion?


As for the actual question, I compiled an (example) Powershell script to automate the mod packaging process as much as possible, with the idea of turning your development folder into an uploadable ZIP/RAR file with one click. This includes FPKing your art, although not being aware of all the caveats Xyth just listed it simply paks everything excluding movies, which now turns out to be foolish. Still, you can still make use of it (or maybe TC01's Autopak utility it relies on itself, depending on your needs) until I manage to upload an updated version.

 

[Sword_Of_Geddon]

Wow Xyth and Leoreth thankyou for the helpful responses.

SO what I understand is that Leaderhead packing doesnt help but actually harms the mod, and doesnt reduce loading times, but all other kinds of graphical files being packed will reduce loading times?

 

[Xyth]

What you're saying also seems to apply to Varietas Delectat. Is that a mod that also should not come with FPKs in your opinion?

Definitely not, unless it's been massively optimized since I last looked at it a few years ago. I expect a lot of unit sets in it were replaced with Bakuel's, so that would make a big difference. Bakuel's units are not only accurate and gorgeous, they're efficient.

Like I say, HR's Unit art FPK is around ~230MB and that is enough to trigger a few MAFs when playing big/busy maps on a 32bit system. Any mod with more than 200MB of unit art is probably better off leaving it unpacked and suffering slow loading.

SO what I understand is that Leaderhead packing doesnt help but actually harms the mod, and doesnt reduce loading times, but all other kinds of graphical files being packed will reduce loading times?

Yes, but at cost of more MAFs if your mod has a lot of unit art.

 

[Nightinggale]

As for the actual question, I compiled an (example) Powershell script to automate the mod packaging process as much as possible, with the idea of turning your development folder into an uploadable ZIP/RAR file with one click. This includes FPKing your art, although not being aware of all the caveats Xyth just listed it simply paks everything excluding movies, which now turns out to be foolish. Still, you can still make use of it (or maybe TC01's Autopak utility it relies on itself, depending on your needs) until I manage to upload an updated version.

How do you script FPK generation? Can the exe be controlled from CLI with arguments or something? Or do you recreate some filesystem generation? The latter could actually be doable if the format is documented.

I would love to see/have a working script for such a task and more importantly, I would like to see the source code.

 

[Leoreth]

I have already shared the script here (sorry I meant to include a link in my original post).

TC01's Autopak (which again wraps Pakbuild) is an EXE that can be called using Powershell. I don't think I have passed further parameters to it, and created FPKs mostly by copying the desired files to a temporary location and then packaging an entire directory. But it's all completely automated, and at the end you get a nice RAR in which the mod art is in FPK format.

 

[AbsintheRed]

Using FPKs will make your mod launch considerably faster, as it's quicker to load a few big files to memory than it is to load thousands of small ones. They will not make your mod smaller, as Civ4 doesn't seem to cope with PakBuild's compression option.

Using FPKs is not slower or faster in-game than not using them. However, using FPKs does seem to restrict Civ4's ability to manage memory, making MAFs more likely on larger maps. So there is a significant trade-off.

My advice (and what I do for my mod), is to have 2 or 3 FPKs: one for Art/Terrain, one containing both Art/Shared and Art/Structures, and (maybe) one for Art/Units. Leave everything else unpacked. I've done a lot of experimenting and this gives me the best balance of launch time and memory usage. I say maybe for the Unit art, because it depends how much your mod has. For most, it'll be the single biggest contributor to slow launch times and MAFs alike. I pack it when I release a new version, but tell those that prefer playing on large maps or with 18+ civs to unpack it (and expect much slower launch time as a result).

For reference, my mod has ~230MB of unit art. It used to be nearly 700MB but I did a massive clean up of my units folder a while back – deleting unused textures and models, baking a lot of textures together (why use 5 textures when it could all fit on 1 or 2, grr), even deleting unused alpha channels (presence of an alpha channel doubles the size of a .dds file regardless of whether it's used). Took weeks, but worth the half a GB of memory saved.

Important: Never put leaderhead art into an FPK. Leader art is loading from disk on demand, not loaded into memory at game launch. But an FPK containing leader art does get loaded into memory, meaning that you reduce available memory (increasing the likelihood of MAFs) for no benefit whatsoever.

Just one addition (question?) to this post:
I vaguely remember that movie files have various issues if they are PAK-ed.
At least I got that advice years ago that those .bik files should never be in FPK files.
Can you confirm that?

 

[Leoreth]

I can confirm, movies in FPKs will not load.

 

[MatteM]

As I gather there were other drawbacks as well with fpk-ing, such as unstable late games in large mods? The mod I'm currently working on has 28 000+ files but I'm not going to
FPK anything before I have the complete picture

 

[Leoreth]

Yes, that is a direct consequence of the drawbacks Xyth pointed out. FPKs mean that more art will be in your memory during a game than would normally be the case. If you have a large mod, in the late game there will probably a lot of units and cities in the game, all with diverse custom art, so the mod already requires a lot of memory. As soon as you get the limit of what the 32 bit BtS exe can handle, you run into instability and MAFs.

The effect is definitely real, reports about MAFs etc. went down noticeably after I followed Xyth's advice and unpacked some of my FPKs again.

 

[AbsintheRed]

I have a couple related questions, maybe someone already has more experience in them:

Is there any difference between having a huge FPK file (e.g., for all unit art in Realism Invictus, with 650+ megabytes), or having more smaller ones (say 100-150 megabytes each)?
In loading time there shouldn't be any real difference, but maybe smaller files can help in reducing MAFs or memory usage?

I also read in a couple places that too many FPK files in a mod might cause various issues.
Some modders in C2C claimed that they have experienced more memory problems when each modder had a separate FPK file, and it should be kept at maximum 7-8.
Can anyone else confirm this? Can it possibly affect the game that way?

I might be on the wrong track, but now I'm thinking maybe the best possible method would be to separate unit art:
With adding only the more commonly used units to FPKs, so they are loaded into the memory with it right on start.
At the same time having the rarely used and lategame units unpaked, as they won't be that much calls for them.
Would have the benefit that loading time is still significantly reduced, while exactly those units are not added to the memory when starting the mod which are rarely called.
Or this doesn't matter at all for units? Only leader art is loaded from disk on demand?

 

[Dancing Hoskuld]

Some things I have heard but do not remember any actual links to evidence

• all art for units and buildings and interface is loaded into memory at the start
  
  
• 7 is the limit on the number of FPK files in a mod. I think this is a suggestion as I have seen more work.
  
  
• smaller is better but not too small, I vaguely remember 100Mb being mentioned as a maximum size

Caveman2Cosmos has 9 FPK files ranging in size from 3Mb to 300Mb