Utilizing PakBuild for faster mod load

[codiac]

In my testing it takes about 38 seconds to get to the main menu when starting WTP. A big chunk of this time the UI has no feedback. I think that big pause in the UI makes it seem a lot longer than it actually is.

Using PakBuild to package the Art folder reduced this time to ~8 seconds for me. The load progress UI displays immediately, so there is no big pause and it feels a lot better.

The most difficult thing about using PakBuild is that all the instructions I found involved copying the Art folder around to isolate it. I am too lazy for that and worked out how to use a link.

I did this in the git checkout, it can probably be done in the installed mod as well, but my instructions are for the git checkout.

Here is how to do it.

1. Download PakBuild https://forums.civfanatics.com/threads/civ4-pakbuild.136023/
2. Make directory for PakBuild in mod source directory

   1. cd WTP-mod
      mkdir PakBuild\Assets

3. Unzip PakBuild.exe in to the new PakBuild directory, so you have PakBuild\PakBuild.exe
4. Create a link from the real Art directory to use with PakBuild. You need to do this step in a command line run as administrator. You also need to `cd` in to the sub-directory or the link won't work properly.
   

   cd PakBuild\Assets
   mklink /d Art ..\..\Assets\Art

5. From the mod's root directory you can now run PakBuild PakBuild\PakBuild.exe /F /I=PakBuild\Assets /O=PakBuild /R=Art this will generate PakBuild\Art0.FPK
6. Delete Art folder from the mod's install directory and copy FPK file there

If there are updates to the Assests\Art directory in git then you need to git pull and rerun PakBuild to generate an updated FPK file.

You can run the command from anywhere but make sure to adjust the paths for the /I and /O arguments.

If you want to use the PakBuild UI, for testing or such, it's pretty easy.

1. Run PakBuild.exe
2. Select menu File->New
3. Select the new Assets directory [I].\PakBuild\Assets[/I] (Not the Art link inside it)
4. Click Store full paths
5. Select menu PAK->Build
6. Wait until the right hand list shows all files
7. Close PakBuild, do not save
8. Replace `Art` folder in install directory with Pak0.FPK file

Enjoy!

 

[Schmiddie]

The tool is well known. Thank you. However, as long as the mod is "in progress" I would not like to offer a packed version - there are still so many changes that it would be a waste of time to unpack the files every time.

If the mod should ever reach a "final" status it is definitely worth doing it.

 

[raystuttgart]

@codiac
Thanks for the input.
We appreciate when people try to help.

Some comments to PakBuild:

• Ingame it does not make any difference if the graphics are packed using PakBuild, since they will be held separately in the cache of the GPU.
• Considering the loading times it really just makes a bigger difference if you do not have the mod on an SSD drive or your RAM or GPU is slow.
• Using PakBuild is extremely annoying for modders when the mod still in the middle of development - you want to have easy access to files then.

As @Schmiddie said, we know this tool since the first days we started modding - it is common knowledge in modding.
And also as he said, maybe once we finished the development we may pack the graphic files, but until then probably not.

Yeah, for me if I load it the first time a day, it also requires about 30s to load, because the graphics are not in the cache.
But when I leave the game and then load it a second time it only takes about 16s, because the cache still has the data.

So it is kind of a difference of around 14 seconds on my machine, which is not worth the hassle to me.
Not if I would take minutes to constantly pack and unpack stuff during modding - since we also work on graphics regularly.

Summary:
We know the tool but purposely do not use it right now.

 

[jooe]

Just a small additional comment: As you noted, the high loading times are apparently due to many small files having to be read. Windows is quite slow accessing many small files (you can see this when copying the mod folder, it might work at just a few MB/s).
If the option exists for you, running the game under Linux/Proton improves the situation a lot. I have set up the compilation in a Windows VM (which is a little slower than native, but still quite bearable) but run the game in latest GE-Proton, allowing startup within about 15 seconds on an old laptop with cached data and a little longer if not cached. GE-Proton has all the protontricks included, so setup is as easy as installing the game, installing GE-Proton and done.

 

[codiac]

Hi, thanks for the responses.

1. You don't have to unpack anything, you regenerate the pak if you update an Art asset.
2. If you add files to the install directory they override what is in the pak, so changes could get added to the file system if you where changing art assets and updated in the git repo/pak later. For certain types of people, perhaps like me, this might make it easier to track what you have changed in the install directory
   
3. "Final" state never happens
4. I don't get the cached improvement Ray speaks of e.g. if I change an XML setting, in the install directory, and restart the game to test it I have to wait around
   
5. GE-Proton sounds interesting, might look in to that as an alternative to "upgrading" to Windows 11
6. There are lots of sounds, maybe they should go in the pak too?
7. How are the clouds on the Plains branch so damn pretty?
   

 

[Nightinggale]

I have a few comments about pak files. I'm not strictly against them, but I'm not a huge fan either. My biggest concern is that multiple people (me included) have experienced crashes when loading a big mod using pak files, but unpacked files made the mods load just fine. This however seems to only have happened on 32 bit systems so it may or may not be a problem today.

Using PakBuild is extremely annoying for modders when the mod still in the middle of development - you want to have easy access to files then.

The nature of git makes it incompatible with the pak design. Git stores changes between each revision and you have every single revision stored in a hidden folder in your working copy. You download all revisions ever made each time you clone. Not an issue with text files and a minor issue with small binary files like audio or images. This is a huge issue with pak files as replacing a pak file can add hundreds of MB to a revision and we will in no time exceed the size/bandwidth limit of our free git host. We can't use pak files on git even if we wanted to do so (which we don't for many reasons).

If we are going to use pak files, then they should be created by the build script, which creates the release zip files. Given that this thread has the command line instructions to do so that should in theory be possible. Since we want people to have the option to mod files in releases and there is the risk of stability issues, then it sounds like we would need to make two releases. Doable, but I fear it will increase the release build time to 10-15 minutes. Not a deal breaker, but compare that to the 22 seconds it takes to compile a debug build for internal testing.

My main issue with this is mainly the time spend on updating the build script and if pak files will cause crashes.

Considering the loading times it really just makes a bigger difference if you do not have the mod on an SSD drive or your RAM or GPU is slow.

I have benchmarked the slow startup. An SSD won't fix the problem because it's not a disk bottleneck. I tried moving RaR to a RAM disk once and it didn't improve loading time. I have tried benchmarking the startup and the disk load is near nothing as in it can literally read absolutely nothing for bursts of multiple seconds.
My theory is that the exe is caching the files using an unordered map (just like vanilla does with getDefineINT) and the implementation of those in our ancient compiler is awful when it comes to performance. This also matches the exponentially slowdown when adding more files, like twice as many files will not just take twice as long.

I don't get the cached improvement Ray speaks of e.g. if I change an XML setting, in the install directory, and restart the game to test it I have to wait around

Ray seems to be in luck that the caching has always worked for him. It is currently working for me, but it didn't on my old computer. Other people report hit and miss in this regard too. There is no pattern and no indication of hardware performance being related to how long you need to wait. The slowdown (which is civ4 engine specific, not just Colonization) is still mostly a mystery, which affects all mods.
38 seconds isn't awful. There have been reports counting in minutes. If I recall correctly my old computer could use up to 5 minutes.

There are lots of sounds, maybe they should go in the pak too?

Mods using pak files usually has one for art and one for audio. Sometimes art is split across multiple files. This is a detail to be determined if we go down the pak route.

 

[codiac]

Hi @Nightinggale I agree with most of your points. For me checking generated content in to a SCM is just wrong


Having a PAK doesn't stop people modifying things, it does require them to have a copy in the right place on the disk. I actually find it much easier to mod with the PAK because only the files I have modified are outside the PAK. YMMV on documenting that versus having multiple releases.


I don't think many people will be modifying the art, so unless you also want to put the XML in to a PAK it's probably not a big issue either way.


It takes me less than a minute to build the PAK so I don't think it would blow the time out to much.


FWIW I couldn't figure out (in the requires 33 seconds) how to get the git way of building the release to exclude all the art files so I wrote a Perl script to zip it up. It will use the art folder if it can't find the PAK. YMMV but I thought I'd attach it in case it's useful to anyone.

 

[codiac]

Apparently I won't be attaching the Perl file, let me know if you want a peek and I'll paste it in a comment or something.

 

[Nightinggale]

FWIW I couldn't figure out (in the requires 33 seconds) how to get the git way of building the release to exclude all the art files so I wrote a Perl script to zip it up. It will use the art folder if it can't find the PAK. YMMV but I thought I'd attach it in case it's useful to anyone.

You can exclude files (and folders?) by editing .gitattributes in the root of the git repository (note that it might be invisible depending on your windows settings).

# export ignores
# The following files will not make it into git exports (as in a zip of the git repository)

.gitattributes export-ignore  
.gitignore     export-ignore
.ini.bak       export-ignore

That's fairly straight forward. The makefile then has a line for building the zip file, which is

@$(GIT) archive --format=zip --prefix=Assets/ --add-file=Assets/CvGameCoreDLL.dll --prefix= -9 --output "Project Files"/WeThePeople-temp.zip head

We can add multiple files by repeating --add-file multiple times.

Apparently I won't be attaching the Perl file, let me know if you want a peek and I'll paste it in a comment or something.

You can always attach a source file by adding .txt to the end of the file. Alternatively you can copy paste the code into the forum itself using [ code ] [ /code ] or [ php ] [ /php ]. (without the spaces inside []). This will preserve the formatting and php will attempt with color highlight of commonly used programming keywords.

 

[codiac]

I saw the git attributes file, I didn't want to hard code the switch and it looked difficult to try an make it optional.

Spoiler zip_release.pl :

#!/usr/bin/perl

use strict;
use warnings;
use 5.10.1;

use Carp;
use Archive::Zip qw( :ERROR_CODES :CONSTANTS );

## TODO: add other files to FPK?
# https://forums.civfanatics.com/threads/civ4-pakbuild.136023/page-4

my $project = 'WeThePeople';
my $version = '3.0.2';         # where should this actually come from?

my $branch = `git branch --show-current`;
chomp($branch);
$version = $branch unless $branch eq 'master';
my $rootdir = "$project-$version";

my @root_files = (
    'changelog.txt',          'install instructions.txt',
    'Readme_WTP_deutsch.txt', 'Readme_WTP_english.txt',
    'WtP_desktop_icon.ico',   'WtP_desktop_icon2.ico',
);

my @asset_dirs = qw(
  Python
  Res
  Sounds
  XML
);

my $zip = Archive::Zip->new();

say "Adding files";
foreach my $file (@root_files) {
    $zip->addFile( "../$file", "$rootdir/$file", 9 );
}

say "Adding directories";
foreach my $dir (@asset_dirs) {
    $zip->addTree(
        {
            root             => "../Assets/$dir",
            zipName          => "$rootdir/Assets/$dir",
            compressionLevel => 9,
        }
    );
}

say "Adding maps";
$zip->addTree(
    {
        root             => '../PrivateMaps',
        zipName          => "$rootdir/PrivateMaps",
        compressionLevel => 9,
    }
);

say "Adding dll";
$zip->addFile( '../Assets/CvGameCoreDLL.dll',
    "$rootdir/Assets/CvGameCoreDLL.dll", 9 );

if ( -f '../../PakBuild/Art0.FPK' ) {
    say "Adding PAK file";
    $zip->addFile( '../../PakBuild/Art0.FPK', "$rootdir/Assets/Art0.FPK", 0 );
}
else {
    say "Adding Art directory";
    $zip->addTree(
        {
            root             => '../Assets/Art',
            zipName          => "$rootdir/Assets/Art",
            compressionLevel => 9,
        }
    );
}

say "Writing $rootdir.zip";
unless ( $zip->writeToFileNamed("$rootdir.zip") == AZ_OK ) {
    croak "write error: $!";
}

And while I am handing out half baked stuff, in case anyone is interested here is a NSIS file for building an installer.

It's based on steam install so it probably needs a tweak to support other installs.

The English is bad and the German comes from Google

Spoiler WTP.nsi :

; Includes ---------------------------------
!include MUI2.nsh

# Comment this out to use unpacked Art
!define USEPAK

; Settings ---------------------------------
!define _MODNAME "WeThePeople"
!define _VERSION "New-Hope-2"
!define _FULLNAME "${_MODNAME}-${_VERSION}"
!define MUI_ICON ".\..\WtP_desktop_icon2.ico"
ICON ".\..\WtP_desktop_icon2.ico"

Name "${_FULLNAME}"
OutFile "${_FULLNAME}.exe"
RequestExecutionLevel user
Unicode True

; Pages ------------------------------------
!insertmacro MUI_PAGE_COMPONENTS
!insertmacro MUI_PAGE_DIRECTORY
!insertmacro MUI_PAGE_INSTFILES

!insertmacro MUI_UNPAGE_CONFIRM
!insertmacro MUI_UNPAGE_INSTFILES

; Languages -------------------------------
!insertmacro MUI_LANGUAGE "English"
!insertmacro MUI_LANGUAGE "German"

; Registry --------------------------------
InstallDirRegKey HKCU "Software\${_FULLNAME}" ""

; Install -----------------------

Function .onInit

  !insertmacro MUI_RESERVEFILE_LANGDLL

  # This is where steam puts stuff, not sure about other installs
  ReadRegStr $R0 HKCU "SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders" "personal"
  ExpandEnvStrings $0 $R0
  StrCpy $INSTDIR "$0\My Games\Civilization IV Colonization\MODS\${_FULLNAME}"

FunctionEnd

Section "The Mod" section_mod

  SectionIn RO


  # If you get bored waiting ...
  #SetCompress off

  SetOutPath "$INSTDIR"

  File "..\Readme_WTP_deutsch.txt"
  File "..\Readme_WTP_english.txt"
  File "..\WtP_desktop_icon.ico"
  File "..\WtP_desktop_icon2.ico"
  File "..\changelog.txt"
  File "..\install instructions.txt"
  File /r /x .gitattributes "..\PrivateMaps"

  !ifndef USEPAK
    File /r /x .gitattributes "..\Assets"
  !else
    SetOutPath "$INSTDIR\Assets"

    File /r /x .gitattributes "..\Assets\Python"
    File /r /x .gitattributes "..\Assets\Res"
    File /r /x .gitattributes "..\Assets\Sounds"
    File /r /x .gitattributes "..\Assets\XML"
    File "..\..\PakBuild\Art0.FPK"
  !endif

  SetOutPath "$INSTDIR\Assets"
  File "..\Assets\CvGameCoreDLL.dll"

  WriteRegStr HKCU "Software\${_FULLNAME}" "" $INSTDIR
  WriteUninstaller "$INSTDIR\Uninstall.exe"

SectionEnd

Section /o "Steam Shortcut" section_steam_shortcut

  ReadRegStr $R0 HKLM "Software\Valve\Steam" InstallPath

  CreateShortCut "$DESKTOP\${_FULLNAME}.lnk" \
    "$R0\steam.exe" "-applaunch 16810 -mod=\${_FULLNAME}" \
    "$INSTDIR\WtP_desktop_icon.ico" 0 \
    SW_SHOWNORMAL "" "Launch ${_FULLNAME}"

SectionEnd

; Descriptions -----------------------------
LangString DESC_section_mod ${LANG_English} "The mod itself"
LangString DESC_section_steam_shortcut ${LANG_English} "A desktop shortcut to launch the mod"

LangString DESC_section_mod ${LANG_German} "Der Mod selbst"
LangString DESC_section_steam_shortcut ${LANG_German} "Eine Desktop-Verknüpfung zum Starten der Mod"

!insertmacro MUI_FUNCTION_DESCRIPTION_BEGIN
  !insertmacro MUI_DESCRIPTION_TEXT ${section_mod} $(DESC_section_mod)
  !insertmacro MUI_DESCRIPTION_TEXT ${section_steam_shortcut} $(DESC_section_steam_shortcut)
!insertmacro MUI_FUNCTION_DESCRIPTION_END

; Uninstall --------------------------------
Section "Uninstall"

  Delete "$DESKTOP\${_FULLNAME}.lnk"

  Delete "$INSTDIR\Uninstall.exe"

  RMDir /r "$INSTDIR"

  DeleteRegKey /ifempty HKCU "Software\${_FULLNAME}" "" $INSTDIR

SectionEnd

Function un.onInit

  !insertmacro MUI_UNGETLANGUAGE
 
FunctionEnd

 

[codiac]

Testing on New_Hope (gigantic maps, 2560 x 1440 resolution) I have been seeing some issues where during normal game play the game complains it can't allocate video memory and crashes. Reloading a save will allow it to run again for some time.

Crash is in : \main\Civ4\SDKs\Gamebryo2_0\CoreLibs\NiDX9Renderer\NiDX9SourceTextureData.cpp, Line:321

This only seems to happen when using a Pak, which is why I posted here and not the bug thread

I have a 16Gb video card, GPU-Z says that just under 2GB are in use. My, uneducated, guess is that when you load a Pak it sets up a cache, or counter, and that is filled up. If you don't use a Pak the it just tries to malloc as it goes and that works.

I've searched a bit and found some suggestions but nothing that seems like a supportable solution, it's probably better to suffer with the start up lag than the crash.

 

[Nightinggale]

I think by now it's pretty much settled that we won't be packing the files. It's a shame it's not working reliable, but we have to do the best we can with the tools available to us. Since the game is a 32 bit application, we have to comply with 32 bit limitations.

 

[jooe]

I think by now it's pretty much settled that we won't be packing the files. It's a shame it's not working reliable, but we have to do the best we can with the tools available to us. Since the game is a 32 bit application, we have to comply with 32 bit limitations.

Out of curiosity and also because I do hit the 4GB barrier quite often recently (can't use Wine/dxvk any more because of that) - did you ever discuss asking Firaxis for a 64bit recompile of the exe? With decent code quality that should not be too much of an effort - and seeing the game (including Civ4) has still a decent player base and a price quite above "abandoned", it might even be interesting for them to do a re-release.

Of course I would prefer them to open source the game engine so we could do it ourselves and even improve some stuff (like a Linux port , but that's probably wishful thinking.

 

[Mr. ZorG]

did you ever discuss asking Firaxis for a 64bit recompile of the exe?

they'd rather release Colonization 2 than do it...

 

[Nightinggale]

did you ever discuss asking Firaxis for a 64bit recompile of the exe? With decent code quality that should not be too much of an effort - and seeing the game (including Civ4) has still a decent player base and a price quite above "abandoned", it might even be interesting for them to do a re-release.

Firaxis released colonization in 2008 and then abandoned it in 2008. They then made a surprise patch in 2009 and then officially abandoned it again. Somehow I don't think they really want to make a 64 bit version for us more than a decade after they officially stopped support. They did make an update to support the steam lobby at some point, but that backfired in the sense that the lobby is in the steam version we tell people not to use because it hurts mod support. A steam lobby is a selling point on steam, hence a way to increase sales. Releasing a 64 bit exe is not something, which checks a box on steam meaning it doesn't carry a return in investment in the same way.

Of course I would prefer them to open source the game engine so we could do it ourselves and even improve some stuff (like a Linux port , but that's probably wishful thinking.

You mean like we won't be stuck with a (nearly) 20 year old compiler... wow now you are really dreaming.
Releasing source code is something which happens, but I think it takes more than some random guy asking Firaxis nicely.

they'd rather release Colonization 2 than do it...

Honestly I would rather have the source code for civ4colo than colonization 2. Firaxis' new releases aren't as great when it comes to what can be modded. In fact WTP wouldn't be possible in any of the civ engines newer than civ4.

 

[jooe]

Firaxis released colonization in 2008 and then abandoned it in 2008. They then made a surprise patch in 2009 and then officially abandoned it again. Somehow I don't think they really want to make a 64 bit version for us more than a decade after they officially stopped support. They did make an update to support the steam lobby at some point, but that backfired in the sense that the lobby is in the steam version we tell people not to use because it hurts mod support. A steam lobby is a selling point on steam, hence a way to increase sales. Releasing a 64 bit exe is not something, which checks a box on steam meaning it doesn't carry a return in investment in the same way.

I was actually thinking of something like a Civ4Col remastered edition which would also include a recompiled exe. That seems to me the most realistic perspective, even if we might have to pay for that new exe by purchasing that edition ...

You mean like we won't be stuck with a (nearly) 20 year old compiler... wow now you are really dreaming.
Releasing source code is something which happens, but I think it takes more than some random guy asking Firaxis nicely.

Honestly I would rather have the source code for civ4colo than colonization 2. Firaxis' new releases aren't as great when it comes to what can be modded. In fact WTP wouldn't be possible in any of the civ engines newer than civ4.

Of course that is when I started dreaming.
But do you know if anyone from these forums actually tried? I mean, the Civ4 engine is really great in its modding capabilities and the community seems quite active even though the game is almost 15 years old.

 

[Mr. ZorG]

Honestly I would rather have the source code for civ4colo than colonization 2. Firaxis' new releases aren't as great when it comes to what can be modded. In fact WTP wouldn't be possible in any of the civ engines newer than civ4.

that was the irony) they will never do either the first or the second ...

 

[Nightinggale]

But do you know if anyone from these forums actually tried? I mean, the Civ4 engine is really great in its modding capabilities and the community seems quite active even though the game is almost 15 years old.

Not sure if anybody ever tried. There was a thread about it at some point, but it lead nowhere and honestly if Firaxis was contacted based on that thread, odds are that they wouldn't even bother to respond. At least some of the posts were like writing to the bus company saying "I want free rides with the bus because that would be cool. I don't care about your finances". If we are ever going to get the source code, it should be in a way of mutual interest, not "I want" or even "I'm entitled to because I have played for years" or whatever else of that nature people can come up with.

 

[codiac]

Testing on New_Hope (gigantic maps, 2560 x 1440 resolution) I have been seeing some issues where during normal game play the game complains it can't allocate video memory and crashes. Reloading a save will allow it to run again for some time.

Crash is in : \main\Civ4\SDKs\Gamebryo2_0\CoreLibs\NiDX9Renderer\NiDX9SourceTextureData.cpp, Line:321

This only seems to happen when using a Pak, which is why I posted here and not the bug thread

I have a 16Gb video card, GPU-Z says that just under 2GB are in use. My, uneducated, guess is that when you load a Pak it sets up a cache, or counter, and that is filled up. If you don't use a Pak the it just tries to malloc as it goes and that works.

I've searched a bit and found some suggestions but nothing that seems like a supportable solution, it's probably better to suffer with the start up lag than the crash.

Sadly this crash is not limited to using PakBuild. I still have this happen from time to time when not using PakBuild. It certainly happens a lot less often, so still better to avoid PakBuild.

I'm guessing it's some kind of memory leak as loading a save seems to give a similar long playtime before it crops up again.