FPK live (tm)

[billw2015]

@Toffer90 came up with ingenious suggestion to build the FPKs IN the dll, and I confirmed that this should be in theory possible.
What it would mean is:

• No FPKs in SVN, loose files instead: Faster update, faster commit, easier and faster merge, better history, easier art changes etc. All the benefits.
• Optionally you can run a batch file to build the FPKs whenever you want.
• When the DLL starts it will check that the FPKs are up to date with the loose files, and if they aren't it will build them before the game starts. If they ARE then there is no delay and the game just starts and loads the FPKs like normal.

Does anyone object to this? I can't see any downsides really. Art changes could incur a first time startup cost, but they are rare, and the time is probably made up for in time spent syncing, updating, editing, repacking etc., the FPKs.

Note: the art for FPKs doesn't go in the art folder itself still, or the game would just load it from there instead (slowly), so new art can be kept out of the packs until we want to put it into them (as it is now), at which point you just move it to the fpk folder for automatic packing.

 

[Toffer90]

I see no downsides, only improvements. ^^

You stole my trademark!!
lol

 

[Dancing Hoskuld]

But increased download time. The main reason we went to FPKs to start with was because it was taking too long to down load as individual files.

 

[KaTiON_PT]

But increased download time. The main reason we went to FPKs to start with was because it was taking too long to down load as individual files.

For the first update, after you get the unpacked files they stay there and don't need to be re-downloaded next time.

 

[Toffer90]

But increased download time. The main reason we went to FPKs to start with was because it was taking too long to down load as individual files.

For the first update, after you get the unpacked files they stay there and don't need to be re-downloaded next time.

We also get revision history for each art file, something we currently loose every time we pack the FPK's and delete those loose art files from the SVN.
I'm not sure if the checkout/fresh download time will increase that much either. And the update time will be so much faster when one doesn't have to download all the FPK's every time they would change, as they wouldn't exist on the SVN.

The main reason we went for FPK's to start with was to make the game load faster upon launch.
We still get the FPK's for that reason. It is faster to make the FPK files and then load them than it is to load the loose files. And the FPK's don't have to be packed everytime the game is launched.

 

[billw2015]

Also download time is a good trade off for dev time. It costs nothing, just make tea or watch youtube. Dev time dealing with FPKs, merging them, unpacking to find stuff etc., is more valuable.

 

[Thunderbrd]

But increased download time. The main reason we went to FPKs to start with was because it was taking too long to down load as individual files.

Firaxis put them in FPKs to begin with.

The download time is a problem for the big ones but not individual files. Might take longer but if you get interrupted on an FPK update, you have to start from the beginning every time. On less stable internet connections (a problem I'm still at least sympathetic to considering how much I've struggled with it) many small files are MUCH better than one big one cuz you can always start over where you left off after a cleanup.



One thing I'd urge, @billw2015 , is to make that coding that automatically updates the FPKs also detect if the initial ones are even present and if not, run the full packing system rather than the update. I'm sure you were going to do that but this way we don't have a problem when people dive in without following the instructions

 

[billw2015]

One thing I'd urge, @billw2015 , is to make that coding that automatically updates the FPKs also detect if the initial ones are even present and if not, run the full packing system rather than the update. I'm sure you were going to do that but this way we don't have a problem when people dive in without following the instructions

Yeah definitely it will not require any instructions, the intention would be it just works.

 

[Dancing Hoskuld]

For the first update, after you get the unpacked files they stay there and don't need to be re-downloaded next time.

I am thinking m ore of releases where people have to download the whole thing again...

 

[Thunderbrd]

I am thinking m ore of releases where people have to download the whole thing again...

Still better that way than trying to make sure the internet can stay solid during the entire FPK download...

 

[JosEPh_II]

If you can make work for an to use then I got no problem.

 

[alberts8]

Actually turns out maybe there is an entry point we can use: DllMain. It is called before any loading from what I can tell.

Yep confirmed:
https://i.imgur.com/3jOgf9r.gifv

@alberts2 , what you think?

Go for it.

Note: the art for FPKs doesn't go in the art folder itself still, or the game would just load it from there instead (slowly), so new art can be kept out of the packs until we want to put it into them (as it is now), at which point you just move it to the fpk folder for automatic packing.

The loose art files can't be inside the mods directories or must be hidden for this to work.
It might also be necessary for the fpk's 'at least empty fpk's without files in them' to exist at the time the exe performs it's file indexing.

 

[Dancing Hoskuld]

Still better that way than trying to make sure the internet can stay solid during the entire FPK download...

Except the FPKs are inside a compressed file so if the internet is not stable enough for one FPK it is not going to be stable enough for the full mod. Not only that the compressed mod with FPKs was smaller than the compressed mod with without them.

 

[Toffer90]

Except the FPKs are inside a compressed file so if the internet is not stable enough for one FPK it is not going to be stable enough for the full mod. Not only that the compressed mod with FPKs was smaller than the compressed mod with without them.

Official releases will not have any loose art files, only FPK's, like its always been.
The SVN on the other hand will only have loose files, and if you loose connection while downloading files from the SVN then you won't have to redownload all the files the SVN managed to get before loosing connection, it can continue where it left off.
With big FPK's it is annoying when you loose connection after downloading 233 MB of the 235 MB sized FPK and the whole FPK is corrupted and discarded and you need to download it from scracth when you retry the connection.

 

[Thunderbrd]

Except the FPKs are inside a compressed file so if the internet is not stable enough for one FPK it is not going to be stable enough for the full mod. Not only that the compressed mod with FPKs was smaller than the compressed mod with without them.

But with an SVN checkout, if it gets interrupted you can simply clean the folder and hit update and it will pick up starting to download the last file that it wasn't able to download and will continue the string of them from there. If that happens in the middle of some thousand image files, fine, you start at 354 where you stalled out. But if it happens during an FPK, the whole thing fails and you have to download it all over again. When I started with the SVN, I sometimes spent days trying to get a checkout because the FPKs were so large I failed in the middle of them about 90% of the time and would have to start all over again with them. Had they been in individual files, it may have been a much larger overall download, but where I stopped I wouldn't have had to backtrack so far and take such a great risk on an all or nothing 5-15 minute prayer that I'd actually get the full FPK in before the internet flickered again. That would've made uploading take MUCH less time in total because it would always work for at least a little while and as it did it would've been getting somewhere instead of losing all progress every time.

Plus, I agree with this:

Official releases will not have any loose art files, only FPK's, like its always been.
The SVN on the other hand will only have loose files, and if you loose connection while downloading files from the SVN then you won't have to redownload all the files the SVN managed to get before loosing connection, it can continue where it left off.
With big FPK's it is annoying when you loose connection after downloading 233 MB of the 235 MB sized FPK and the whole FPK is corrupted and discarded and you need to download it from scracth when you retry the connection.

 

[JosEPh_II]

Sooo, are we gonna do a New Release to Public now?

 

[Toffer90]

Sooo, are we gonna do a New Release to Public now?

Not a chance. We just did one and nothing indicates that the SVN is currently on a stable streak for any patch to be released either.

 

[Thunderbrd]

Sooo, are we gonna do a New Release to Public now?

I think so. Another .something once we get to the point of breaking compatibility. We might have to back up a rev and grab that for an official sub-release.

To comment on what Toffer just said, we'd only do that if we had a seriously stable platform for a release at that point. Things have been... challenging in that regard.

 

[JosEPh_II]

I sense some disagreement here.

 

[Toffer90]

I sense some disagreement here.

TB's an optimist, I'm a pessimist. ^^