Mac Extractor for Assets .fpk files

[AlanH]

Please give version 1.2.1 a try.

 

[Xyth]

It worked! You're a legend Alan, thanks.

 

[Xyth]

Looks like I was wrong about the effect of FPKs on loading time. I've been doing some testing on a mod similar to mine, trying launching it with FPKs and without and it looks like there is quite a massive difference:

- With units not packed (574.4MB), time for loading splash to appear was 3 mins 28 secs.
- With units packed (537.5MB), time for loading splash to appear was 7 secs.

Is it possible to give Macapaka packing functionality as well?

 

[AlanH]

OK, I'll have another look at it, but it won't be until May earliest, as we have a UK General Election in progress here, and I'm heavily involved in getting a great MP re-elected!

 

[Xyth]

OK, I'll have another look at it, but it won't be until May earliest, as we have a UK General Election in progress here, and I'm heavily involved in getting a great MP re-elected!

No hurry, good luck with the election!

 

[Xyth]

Hope your MP got back in Alan (unless you support the BNP or something lol). Welcome to the fun world of coalition governments, we're pretty used to them down here.

Any chance you could have a look at adding packing functionality to Macapaka sometime in the next couple weeks? I got hold of an old PC laptop that I'm going to try use to to pack the next version of HR, but thought I'd check in with you here to see if I need to or not

 

[AlanH]

Yes, thanks. Our LibDem got back in by a whisker.

I'll have another look at it over the next week or so. I seem to recall I had a partially working setup, but was hung up on a couple of issues:

(a) There are two unknown values per file in the .fpk file format. I am not sure whether I can get away with putting dummy values in these locations. If I need to give them "correct" values, this will be a problem unless I can persuade someone at Firaxis to tell me how to calculate them. I need to create a set of files containing dummy values and see if it can be read by Civ4 and/or pakbuild.

(b) User interface. I'm not very impressed with the pakbuild Windows UI, it seems rather clunky and not very intuitive. Do you need a full .fpk file editor where you can add and remove assets files individually, or is a simple "Pack this folder to create a set of .fpk files" interface all that's required?

 

[Xyth]

Yes, thanks. Our LibDem got back in by a whisker.

Good to hear. LibDems seem like a good bunch, hopefully they can get some form of PR happening for Britain. Best thing that ever happened in NZ politics.

(b) User interface. I'm not very impressed with the pakbuild Windows UI, it seems rather clunky and not very intuitive. Do you need a full .fpk file editor where you can add and remove assets files individually, or is a simple "Pack this folder to create a set of .fpk files" interface all that's required?

I haven't tried pakbuild before but I've been setting my mod up so that I can hopefully pack each type of art into a separate FPK. Assets/Art/Leaderheads -> Leaderheads.fpk, Assets/Art/Units -> Units.fpk, etc. So a fairly simple "pack this folder and all subfolders and contents into a single FPK" would be fine.

 

[AlanH]

Please give Macapaka version 1.3.1 a try - link in the first post. I've added a File menu option - "New FPK File..." that lets you select a folder full of files. You can then select any subset of that folder as content to be Packed into a FPK file.

I have tested it by unpacking the Assets0.fpk file, then Packing the resulting files. I then used the new file to replace the standard Assets0.fpk file, and I was able to start up Civ4 with the new file. PakBuild unpacks the new file OK as well. However, the acid test is whether it will do what you need to build mods.

Please let me know whether it works for you.

 

[Xyth]

Please give Macapaka version 1.3.1 a try. I've added a File menu option - "New FPK File..." that lets you select a folder full of files. You can then select any subset of that folder as content to be Packed into a FPK file.

I have tested it by unpacking the Assets0.fpk file, then Packing the resulting files. I then used the new file to replace the standard Assets0.fpk file, and I was able to start up Civ4 with the new file. PakBuild unpacks the new file OK as well. However, the acid test is whether it will do what you need to build mods.

Please let me know whether it works for you.

Unfortunately I can't get it to work. The main issue seems related to subfolders. Firstly, when the window comes up that lets you choose which contents you want to include, selecting a folder in that window doesn't automatically select all the contents of that folder as well. You need to rotate the little arrow and select them all manually. I noticed the following messages in the Console while browsing in this window:

01/06/10 1:35:32 PM	Macapaka[3739]	*** -[NSCFArray objectAtIndex:]: index (4) beyond bounds (0)

The index number would change depending on what I was selecting. I also noticed that the file and byte count in the New FPK window would not always update properly. The count would increase if I selected a folder or file at the 'root' but the byte count would not increase appropriate to content size of any selected folder. Manually selecting a file in a folder would not increase the file or byte count.

Actually I just noticed that no matter what you select in the root directory the byte count only ever increases by exactly 6148 per item.

I then tried to make an FPK that contained no folders. This worked but the resulting FPK was too small (528 KB) compared to a folder of 26 items (3.7 MB). Trying to open this FPK in Macapaka failed with the console message:

01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x200960
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x200960
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x16d000
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x16d000

Hope this helps :/

EDIT: Just tried to unpack and repack Assets0.fpk like you did. The new FPK was only 8.6 MB compared to the original 104.9 MB. What could be different on my system from yours? I'm using OSX 10.6.3.

 

[AlanH]

Unfortunately I can't get it to work. The main issue seems related to subfolders. Firstly, when the window comes up that lets you choose which contents you want to include, selecting a folder in that window doesn't automatically select all the contents of that folder as well.
You need to rotate the little arrow and select them all manually.

That is only a cosmetic problem. The subfolders are all selected, they are just not highlighted. You should not have to select the subfolders explicitly. The window behaves the same as when it is displaying the contents of a FPK file (it's running the same code).

I noticed the following messages in the Console while browsing in this window:

01/06/10 1:35:32 PM	Macapaka[3739]	*** -[NSCFArray objectAtIndex:]: index (4) beyond bounds (0)

The index number would change depending on what I was selecting. I also noticed that the file and byte count in the New FPK window would not always update properly. The count would increase if I selected a folder or file at the 'root' but the byte count would not increase appropriate to content size of any selected folder. Manually selecting a file in a folder would not increase the file or byte count.

Actually I just noticed that no matter what you select in the root directory the byte count only ever increases by exactly 6148 per item.

I then tried to make an FPK that contained no folders. This worked but the resulting FPK was too small (528 KB) compared to a folder of 26 items (3.7 MB). Trying to open this FPK in Macapaka failed with the console message:

01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x200960
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x200960
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x16d000
01/06/10 1:51:52 PM	Macapaka[3808]	-[AHNode setParameters:location:param1:param2:]: unrecognized selector sent to instance 0x16d000

Hope this helps :/

EDIT: Just tried to unpack and repack Assets0.fpk like you did. The new FPK was only 8.6 MB compared to the original 104.9 MB. What could be different on my system from yours? I'm using OSX 10.6.3.

Were you running Macapaka from the disk image, or did you copy it to your hard drive? I can get it to crash and do some other strange things if I run it from the disk image, though I'm not sure why. I''ll investigate that further this evening.

 

[Xyth]

That is only a cosmetic problem. The subfolders are all selected, they are just not highlighted. You should not have to select the subfolders explicitly. The window behaves the same as when it is displaying the contents of a FPK file (it's running the same code). Were you running Macapaka from the disk image, or did you copy it to your hard drive? I can get it to crash and do some other strange things if I run it from the disk image, though I'm not sure why. I''ll investigate that further this evening.

I was running it from my hard drive. I'll keep experimenting and see if I can figure any thing else out.

 

[AlanH]

I have found one issue with subsets of files/folders. The packed file contains the wrong file count. I've fixed that, and issued version 1.3.2, linked in the first post.

 

[Xyth]

I have found one issue with subsets of files/folders. The packed file contains the wrong file count. I've fixed that, and issued version 1.3.2, linked in the first post.

I've had some success now. It looks like it will only pack properly if there is nothing else in the chosen root directory. Once I copied the folders I wanted to pack into directories with nothing else in them it started to work. Interestingly this seems to affect Pakbuild on Windows too.

One side effect I'm getting is that the presence of FPKs seems to partially break modular loading of art. I imagine this is nothing to do with Macapaka though. I have a busy day today but I'll do more experimenting tonight and see what more I can figure out, but I suspect the issues I'm still having with it are simply limitations/bugs in the original Pakbuild. I'll report back when I know.

Thanks very much for this though, having this functionality is going to make things so much easier. Copying my mod continually back and forth between my Mac and a PC really sucks.

 

[AlanH]

Thanks, that's interesting. I can't imagine how the presence of unselected files/folders in the root directory would affect the generated FPK file. And there's no fundamental reason why PakBuild and Macapaka would have similar symptoms in relation to this, as I have no idea how the PakBuild code works. I'll try some experiments with extra directories and files in the root folder.

What does "modular loading of art" look like, and how does it misbehave when it breaks?

 

[Xyth]

What does "modular loading of art" look like, and how does it misbehave when it breaks?

Well for my mod all my unique units are not working when Art/Units is packed. The button appears pink in the pedia and the game crashes when trying to view them - meaning their art can't be found. The art for these is inside the .fpk but their definitions and info xml files are modular (somewhere in Assets/Modules). Units defined in Assets/XML are fine, as are units with both art and xml entirely in Assets/Modules. I guess it needs to be all one or the other.

EDIT: Actually it looks like most packed unit art can't be found, I must have done something wrong. I'll try again tonight.

 

[Xyth]

Okay I've done a fair bit of testing now. Unfortunately most FPKs generated by Macapaka are not working at all. In all tests I packed the Art folder with pre-selected contents from a temporary directory with no other contents.

I first tried packing the entire Art folder of History Rewritten. The result was 804.9 MB compared to 798.2 MB from Pakbuild. Both could be unpacked fine by Macapaka but BTS couldn't find any of the art in the Macapaka one.

I had various tries with separate FPKs (Art/Leaderheads, Art/Units, etc). None of the ones from Macapaka worked (and all were slightly larger than their Pakbuild counterparts), but some of the ones from Pakbuild didn't work either. This implies to me that it's more an issue with how BTS loads these files than anything else. I'll do some research on this, but for now we're probably best to focus only on packing an Art folder in its entirety.

I then tried unpacking and repacking Assets0.fpk. This worked fine this time, sizes were the same and both Civ4 and BTS launched without issue.

Finally I tried unpacking and repacking some FPK from other mods (Diplomacy II, Varietas Delectat). These didn't work either. Both of those had multiple FPKs though so that may not be a fair comparison. I'll see if I can find a smaller mod with a single FPK to test tomorrow.

 

[AlanH]

I've had a look at the file/folder selection logic in Macapaka, and I think I might have resolved a couple of bugs that may have been causing problems.

Please can you have a look at version 1.3.3 and see if it is any better. I can't promise that it has solved all the issues, but I think some of them may have gone.

Apologies for subjecting you to this trial and error sequence, but I don't have any good test cases that I can use here. If there are still problems, and you have a test case I can use to investigate, please give me the details. You can email sample files to alanh <at> civfanatics <dot> net.

BTW, it's probably obvious, but I recommend that you ensure that the top level directory that you select from the "New FPK File" menu is the Assets folder or its equivalent, even if you plan to pack a file subset from lower in the folder hierarchy. This ensures that the .fpk archive records its file paths relative to the Assets folder. I don't know whether FPK files will work in Civ4 if they are created and saved at lower levels of the hierarchy.

 

[Xyth]

I've had a look at the file/folder selection logic in Macapaka, and I think I might have resolved a couple of bugs that may have been causing problems.

Please can you have a look at version 1.3.3 and see if it is any better. I can't promise that it has solved all the issues, but I think some of them may have gone.

It's my son's 1st birthday this weekend so I won't be able to do any more decent testing until next week. I'll let you know how it goes then.

Apologies for subjecting you to this trial and error sequence, but I don't have any good test cases that I can use here. If there are still problems, and you have a test case I can use to investigate, please give me the details. You can email sample files to alanh <at> civfanatics <dot> net.

No need to apologize, you are going out of your way to add this functionality for us modders and I'm hugely appreciative. The least I can do is help you test it.

If you want a test case, I just released History Rewritten 0.5. It's got 5 FPKs in it, all packed in Pakbuild, thanks to a crappy old Windows laptop I was able to borrow. Perhaps unpacking and repacking those in Macapaka could help to see what's going on?

 

[AlanH]

Thanks. Downloading History Rewritten now.

[UPDATE] OK, I see a problem with FPK created using multiple selections, and I'll investigate it. Don't bother trying version 1.3.3