Dynamic FPKs and encapsulated dependencies.

[billw2015]

These are a couple of things I have implemented for my experimental git repository that might be of interest to push to SVN as well.
Dynamic FPKs means that instead of having manually packed FPKs stored in the repository, there is a directory with the unpacked art and a script to automatically pack it into a set of FPKs. This uses the PakBuild.exe FKP tools command line to generate the FPKs from a single directory, and uses the handy option to limit the size of the FPKs that are generated. Currently I chunk them into 256MB files and we get 4 of them, C2C0.FPK to C2C3.FPK.
This has a few benefits:
1. The FPKs are maximally optimal in number, fewer = faster to load (I hope).
2. The art can be stored as loose files making editing it trivial.
3. You don't need to worry about making mistakes in what you put in the FPKs, it is automatic, everything in the unpacked directory goes into the FPKs.
4. Source control (SVN and git) handle smaller files better, and a separate history will be maintained for each file, rather than a change to an entire FPK when a single dds was modified.
5. Is actually not slow. It takes 1m35s to fully pack the ~30000 files on an SSD and same on a mechanical (according to someone elses tests).

Encapsulated dependencies make development of the DLL easier to setup. The compiler, tools and all SDKs (including boost and python) are all put in a single package that can be extracted automatically inside the repository when you want to build the DLL. The whole package (thanks to Anq/alberts2 for the parts I am using) comes in at just over 10MB, 100MB once unpacked.

I also made a set of scripts for:
- Unpacking all existing FPKs: during transition people can use it to update the unpacked folder
- Packing the unpacked directory to the dynamic FPKs
- Installing the dependency package (this is done lazily by the build scripts anyway)
- Build the DLL in each config
- Clean the DLL builds

If you want one or more of these things in SVN how shall I proceed with getting it there?
I have write access but I don't want to break peoples workflows or leave them confused about why the FPKs are gone etc.

 

[Toffer90]

Dynamic FPKs means that instead of having manually packed FPKs stored in the repository, there is a directory with the unpacked art and a script to automatically pack it into a set of FPKs. This uses the PakBuild.exe FKP tools command line to generate the FPKs from a single directory, and uses the handy option to limit the size of the FPKs that are generated. Currently I chunk them into 256MB files and we get 4 of them, C2C0.FPK to C2C3.FPK.

I have some questions.

So it will look something like this (humour me):

\Caveman2Cosmos\Assets\FPK-Workshop\

In here we have some scripts and the PakBuild.exe
When we want to pack new loose files into FPK's then we
1. Start by running the FPK unpack script so that the old FPK's gets unpack to "\Caveman2Cosmos\Assets\FPK-Workshop\Art"

The script would look one folder up ..\ to find the FPK's I guess; or one have to move the FPK's into this FPK-Workshop folder first.​

2. Then we move the entire art folder from "\Caveman2Cosmos\Assets\Art" to "\Caveman2Cosmos\Assets\FPK-Workshop\Art" and merge with what's there.
3. Finish up by running the FPK build script which makes the new FPK's and place them in the FPK-Workshop folder so we can move them up one folder manually and overwrite what's there.

Or, the script automatically place the new FPK's one folder up and replace the files that's there for us.
The art folder may, or may not, be deleted by the script after packing is finished. Curious about what needs be done manually here.​

Do I see the gist of what you presented here?

 

[billw2015]

No, it is simpler than that! The art that is intended to be in FPKs is kept in the repository in a folder called unpacked\art (so it stays separate from the art that is NOT meant to be packed). The FPKs are no longer in the repository at all, they are only built locally. So when you sync or modify the unpacked\art directory you can run PackFPKs and it will re-create the FPK files in the Assets directory from the unpacked directory contents.
https://imgur.com/a/rGUX0Zz

/edit Actually I may not have understood what you are asking.
Are you asking about the day to day process of using the FPKs, or what is required during the transition period from having the FPKs in the repository to having the loose files instead?

 

[Toffer90]

No, it is simpler than that! The art that is intended to be in FPKs is kept in the repository in a folder called unpacked\art (so it stays separate from the art that is NOT meant to be packed). The FPKs are no longer in the repository at all, they are only built locally. So when you sync or modify the unpacked\art directory you can run PackFPKs and it will re-create the FPK files in the Assets directory from the unpacked directory contents.
https://imgur.com/a/rGUX0Zz

/edit Actually I may not have understood what you are asking.
Are you asking about the day to day process of using the FPKs, or what is required during the transition period from having the FPKs in the repository to having the loose files instead?

If there are no old FPK's on the SVN the steps about unpacking them first would disappear from the outline in my last post.

I would probably never bother to make the FPK's as my game loads pretty fast on my PCI-express SSD, so having only loose files isn't that big of an issue for me.
So I would rather just copy the art folder from unpacked\art into \Caveman2Cosmos\Assets every time I update my SVN instead as I won't keep an eye on whether new loose files have been added since I last updated the SVN.

I was really outlining the process needed to be done every time an official release is to be made, the FPK's would have to be pre-packed in that release as players who gets the official version of the mod wants, and some needs, it to be as simple as possible. But I also assumed the FPK's from last release would be in the SVN with the new loose files growing in numbers until the next release where we repack the FPK's.

I only see a need to pack FPK's at official releases, which means something like 3 times every 5 years. The script would indeed be nice to have for occurrences like those.
If I had to pack FPK's frequently, even with the ease of your script, or if I had to juggle FPK's or art folders around all the time, well, I would get quite annoyed.

I get that it may be beneficial do it that way with git, so it won't contain large files like FPK's,
However on the SVN the script makes most sense for me to be only used for packing up FPK's at new official releases.

 

[Thunderbrd]

I'm going to have to get in and poke and prod at this a bit with the git setup but I'm thinking it's all actually just brilliant. One concern - unpacked art files need to be in the file path for the XML to read them. It would be nice to be able to have those there and just auto-pack the Art folder itself. However, this would mean that some file types should NOT go in the Art file (or it means that the automatic FPK generator must ignore them - but if that's to be the case then they should be in another location entirely because the trick at release would be to simply delete the art folder entirely after packing the FPKs for release.

Am I making any sense here? I find that if you have XML that references any art from anywhere but the Art folder, once packed into the FPK, it doesn't work right, and it would be nice to, as Toffer mentions, have the option whether to pack the FPKs or not at all and just leave them to be referenced by the game where the game expects them either way.

@Toffer90 having just run it, I find that this is actually seriously easy... if you change any art files, just run the packer program before playing. There's an update exe that makes it even faster to just adapt to the latest changes. I'm... totally cool with the way this is setup but trying to run this with a ton of loose files without packing will have trouble with the xml reference path issue since he keeps the art files in an 'unpacked' folder in the Assets folder. Therefore, they really aren't there for the game to read them if they aren't packed.

I'm not sure what all is included with the automatic setup you're talking about... you DO need to independantly install VS (or whatever you're working with) outside of all that right? Just making sure... Otherwise it sounds like you just run the script and it puts everything in place. Does your script modify the makefile paths file accordingly in the process?
I'll stop asking stupid questions now.

Well this is pretty damned awesome and shall make joining us a LOT more inviting! I just ran it all and it works like a charm. Still learning how to work with Git in full - the one frustration so far is this massive reliance on branching, but it does help to understand Alberts2 and others wanting to use branches since this forces you to become very comfortable with that being a integral and required part of the design process. Kinda still trying to figure out how you update the master but I'm thinking I'm not getting the options you talk about for that because nothing has been changed since I cloned the repository.

I'll discuss that more elsewhere I suppose. We need a thread for that subject here so I urge that we start one and link folks to your Git thread so we can get discussions going on that here. I was going to give the link on the forum here but I don't want to jump the gun on any plans you have with that yet.

Unfortunately, I think your main availability time is squarely while I'm working. Urgh! So discord is a little sub-optimal for non-realtime discussion.

 

[Thunderbrd]

On a slightly different note, apparently this would be the same as Anq got me up to but is there any way you are aware of that allows us to clear these errors up?:

@alberts2?

Or is this just a necessity to have these here. I know they don't seem to be causing any real problems.

 

[Anq]

On a slightly different note, apparently this would be the same as Anq got me up to but is there any way you are aware of that allows us to clear these errors up?:
View attachment 530349
@alberts2?

Or is this just a necessity to have these here. I know they don't seem to be causing any real problems.

That was intentionally left there because if you include the boost headers for Visual Studio to analyze, it will output thousands of errors and make it completely unworkable.

 

[Thunderbrd]

That was intentionally left there because if you include the boost headers for Visual Studio to analyze, it will output thousands of errors and make it completely unworkable.

Well... here's what I don't get... we used to not get these errors. How is that?

Perhaps it was just how it was on VS 2017 huh? I might not be quite realizing that was the real underlying catalyst.

Also - don't take this as a chastising... I'm just concerned is all.

 

[billw2015]

I get that it may be beneficial do it that way with git, so it won't contain large files like FPK's,
However on the SVN the script makes most sense for me to be only used for packing up FPK's at new official releases.

I was told it was a LOT slower to load the loose files. I imagine it would be on a mechanical drive, but I can't test that easily.

I'm... totally cool with the way this is setup but trying to run this with a ton of loose files without packing will have trouble with the xml reference path issue since he keeps the art files in an 'unpacked' folder in the Assets folder.

So this can also work if we put the unpacked stuff in the normal art folder instead, the packing tool can exclude stuff from FPKs based on extension so we can just pack all but *.bik files to get what we have now.
So this is another approach, I just thought it would be more clear if they are entirely separated, and I don't know what the games does if there are the same files in FPK AND art folder.

 

[raxo2222]

So this is another approach, I just thought it would be more clear if they are entirely separated, and I don't know what the games does if there are the same files in FPK AND art folder.

If same file is in same path then game prefers file in art folder.

 

[olduser]

I was told it was a LOT slower to load the loose files. I imagine it would be on a mechanical drive, but I can't test that easily.

It is and it doesn't only depend on the drive speed a single CPU core will be maxed out as well if the drive is fast enough.
Before our dll is even load the bts.exe scans through all files in the mods directory and does some kind of hash calculation.

 

[Dancing Hoskuld]

I was told it was a LOT slower to load the loose files. I imagine it would be on a mechanical drive, but I can't test that easily.

I think that the only reason we moved to FPKs was because it reduced the size of the download. Any speed up in load was not noticed.

 

[billw2015]

Still learning how to work with Git in full - the one frustration so far is this massive reliance on branching

Although I phrased it this way in the wiki article, you can make branching a minimal part of your workflow. For instance you can just have a general main-dev branch off master that most people work in, then when you want to make releases or pre-releases you can just merge to master and make one there. So main-dev just behaves like your current trunk in SVN. Then people can make branches off of it for longer term work.

Kinda still trying to figure out how you update the master but I'm thinking I'm not getting the options you talk about for that because nothing has been changed since I cloned the repository.

I think you made this more difficult by using a fork instead of using the main repository directly. Now that you are collaborator you can do that, and updating master would just mean clicking the pull button. I intend to separate the setup guides into collaborator (basically everyone here with SVN falls under that category), and contributor (random people who want to experiment without getting added officially). The collaborator guide will be a lot simplified, I will just layout a set of guidelines specific to how you can work on this project, rather than try and be general about git.

 

[raxo2222]

I think that the only reason we moved to FPKs was because it reduced the size of the download. Any speed up in load was not noticed.

Well they were regularly packaged, so there never was too much loose art at least outside modules.
When I unpacked all this stuff to art folder as experiment game couldn't even start loading - was too busy with loose art files.

 

[billw2015]

When I unpacked all this stuff to art folder game couldn't even start loading - was too busy with loose art files.

It definitely makes sense it would be a lot slower, but I definitely wouldn't suggest it is used like that regardless.
Currently the SVN guide recommends strongly that people DON'T use the Mods directory directly for their working copy, instead recommending export to that directory from the working copy that is stored somewhere else.
I could make a system that mimics that closely: you do an initial setup where you select your mods folder, it records it into your working copy directory, and then there is a script that will sync from one to the other, including both building the DLL and packing the FPKs *if they have changed*. What about that? Of course the DLL build could still push directly to the mod directory if enable it, but it can use the same saved directory you set up before. I can also auto detect the civ directories in most cases as 95% of the time they are in steam or program files.

 

[raxo2222]

It definitely makes sense it would be a lot slower, but I definitely wouldn't suggest it is used like that regardless.
Currently the SVN guide recommends strongly that people DON'T use the Mods directory directly for their working copy, instead recommending export to that directory from the working copy that is stored somewhere else.

Well I was telling DH, that regularly packaging any new loose art in art folder prevented slowdown.
I use working copy directly in Mods folder - I guess that part is obsolete, that is fine to do so now as long as you know what you are doing.

 

[olduser]

I think that the only reason we moved to FPKs was because it reduced the size of the download. Any speed up in load was not noticed.

HOW TO: Get your mod to load faster

 

[billw2015]

I use working copy directly in Mods folder - I guess that part is obsolete, that is fine to do so now as long as you know what you are doing.

This could still work okay actually (if art is still in a folder that isn't called art, then repository can be in Mods folder without it ignoring the FPKs): I noticed there are batch files for running the game, these could integrate the same build/pack *if required* behaviour as the deploy scripts I described. i.e. you double click "Run Caveman2Cosmos.bat" file, and if the unpacked folder no longer matches the FPKs (asking SVN or git to tell me, it should be fast) it will repack them, then run the game+mod.

Basically this would boil down to one script that you can use to run the game+mod from anywhere:
- if you never ran it before AND it can't automatically detect where your mods folder is (99% of the time it probably can) it will throw up a folder browser for you to tell it
- it will detect if unpacked files have changed, if so it will repack them
(optionally it will compile the DLL if required)
- if it isn't already in the mods folder it will deploy the required files to the mods folder
- finally it will run the game with the mod on command line.

My feeling is that the benefits of this outweigh the costs, but I didn't work on this project really, I can only go by experience from other projects, what I can see, and my experience getting setup in this one:

Benefits:
- Simple to explain: Run this batch file to run the game
- Simple to add new art: Just put it in the unpacked folder where you normally would, no manual FPK stuff required at all
- Better for svn database? I think the loose files are better than FPKs for svn itself but not sure.
- Better for readability in svn - you can see *what* art files changed in a change list, not just an FPK change.
- Better for debugging - you can verify the pathing directly without opening an FPK (once you find the right one).
- More robust - no possibility of duplicate files in different FPKs

Cons:
- You can't just svn and play - However I think with a script to do everything you can kind of svn and play, but play=run this batch file, instead of just run the game. It actually removes the step of having to enable the mod, so it could be considered easier.
- Packing FPKs takes a bit of time (30 seconds maybe?) - with the script automatically detecting if this is actually required it should be minimal impact.
- It's a change - yep can't get around that one!
- Other cons here?

 

[Toffer90]

and I don't know what the games does if there are the same files in FPK AND art folder.

It loads all the files in the FPK first then it loads all the loose files afterwards, loose files always overwrite files from FPK's.

I was told it was a LOT slower to load the loose files. I imagine it would be on a mechanical drive, but I can't test that easily.

I'll time the difference to FPK's and loose, just so I know the difference myself, been a couple of years since I last tried only having loose art files.

 

[Thunderbrd]

I don't know what the games does if there are the same files in FPK AND art folder.

This has probably been answered already but since I haven't read the whole thread and am getting caught up: What is NOT in the FPK takes priority over that which is IN the FPK. The hierarchy goes to the loose file and it's somewhat expected as a short term stage of development.

If my coffee-less brain is following the conversation regarding running the game from the modding copy of the mod folder, keeping the files in an alternative folder actually is better because then you won't have that conflict of having all the loose files sitting there being detected and used by the game over the FPK files - just requires that before you RUN the game, if you've recently updated (pulled? That's the word for this in Git?) or altered the unpackaged art folder contents, then you need to update the FPKs before running the game. A batch file for starting the game, rather than an EXE would be a helpful alternative solution to that... but did you mention compiling the dll as part of that step?

I think you made this more difficult by using a fork instead of using the main repository directly. Now that you are collaborator you can do that, and updating master would just mean clicking the pull button. I intend to separate the setup guides into collaborator (basically everyone here with SVN falls under that category), and contributor (random people who want to experiment without getting added officially). The collaborator guide will be a lot simplified, I will just layout a set of guidelines specific to how you can work on this project, rather than try and be general about git.

Ok, yeah the guide wasn't specific about that key difference. That's ok... I don't feel critically lost yet, just unsure like a cat in an abandoned building it might make a home.