Decoder and encoder for FTXTInterface UI textures

[LRN]

For anyone interested, here's a free FTXTInterface encoder/decoder for Civilization: Beyond Earth (and V, maybe).

Repository: https://gitorious.org/lrn-tools/ftxt
Download: https://gitorious.org/lrn-tools/ftxt/archive/0.2.zip

Requires Python3 (confirmed to work with CPython 3.4.2) and PIL or Pillow (confirmed to with Pillow 2.6.1)

 

[LRN]

One thing that is kind of bothering me is that currently automatic block size detection is extremely dumb: it just encodes at all plausible block sizes (ones where worst-case number of blocks is < 0xFFFF, because that's the limit of a 16-bit block map; all the files that i've seen use 8- or 16-bit block maps) and picks the one that results in the smallest file.

This can take a long time on images that are large enough to be slow to encode, but small enough to be plausibly encodeable with small blocksizes.

My gut feeling is that Firaxis has some heuristic for this (say, images larger than 4096 are encoded with blocksize 8, images larger than 2048 - with blocksize 4, and so on), but i was unable to figure it out due to the limited data set. Do you, guys, have any ideas? I haven't looked at CivV resources, maybe the logic is more apparent there.

 

[Lord Tirian]

Wow, this was totally overlooked, I guess... does this tool essentially does what this one does (i.e. decode the RLE DDS files), but can do so in reverse?

If so, what's the impact on the file size? At the moment, I have the dilemma that full RGBA DDS files are huge and that compression is sometimes very noticeable on the Civ5-style icons (especially seeing how Firaxis' files never seemed to have that issue).

In any case, I'll have to dig into this, just need to get a Python3 install on my Windows box.

 

[LRN]

Wow, this was totally overlooked, I guess... does this tool essentially does what this one does (i.e. decode the RLE DDS files), but can do so in reverse?

Yes

If so, what's the impact on the file size?

Depends on the input. Input that is highly regular (significant amount of areas of constant color or same features repeated often) compresses farily well (given that it's lossless). You can just uncompress existing FTXTInterface DDS files and compare the size - that should give you a good idea. Obviously, this works well only on artificial images (UI elements), not on real-life images (textures for game models). Hence the 'Interface' in 'FTXTInterface'.

At the moment, I have the dilemma that full RGBA DDS files are huge and that compression is sometimes very noticeable on the Civ5-style icons (especially seeing how Firaxis' files never seemed to have that issue).

Yes, this is exactly the case for this kind of compression. Mods like Colour-filtered Tech Web" should use it (although, obviously, adding color makes image somewhat less compressible; original Firaxis icons often use shades of a single color).

 

[Lord Tirian]

I quickly tested it on the golden age yields I made here (showing the PNG).

Result:
Uncompressed: 321 KB
DXT5: 81 KB
RLE: 94 KB (53 KB texture + 41 KB index)

Of course, that's not a super-stringent test but considering the full quality despite the use of gradients there for the actual icon, I suspect that a lot is gained from the heavy use of transparency in all UI atlases (e.g. every icon is a 172px circle in a 256px square) and similar UI elements.

In any case, good job, you should post it in the corresponding Civ 5 section as well! This could improve the file sizes for almost every mod (as almost every Civ5 mod has some sort of icons).

 

[LRN]

Does it actually work for CivV? That is, have you tried putting the compressed image into a mod and loading it, does CivV grok the image?

 

[Lord Tirian]

I was overly enthusiastic, it seems. I assumed if the tool works for Civ:BE, it would work for Civ5... but it doesn't.

What is the expected format of the input DDS? I'm feeding it 8.8.8.8 ARGB files and get files out that are at least readable in an editor... but I don't know what you intended it with, so maybe there's an issue.

 

[LRN]

I was overly enthusiastic, it seems. I assumed if the tool works for Civ:BE, it would work for Civ5... but it doesn't.

Doesn't how exactly? The game claims that it has failed to read the texture (in BE, if you give the game a bad texture file, it will show a message box with an error when it tries to load it)?

What is the expected format of the input DDS? I'm feeding it 8.8.8.8 ARGB files and get files out that are at least readable in an editor... but I don't know what you intended it with, so maybe there's an issue.

Expected input is uncompressed DDS. The decompressor preserves the format of the original: copies over the header, rewrites the width/height fields, then copies pixel blocks from the compressed image into appropriate places of the decompressed image. It never tries to interpret the pixels. Compressor does the same thing in reverse.

I've tested it in Civ:BE by decompressing and then compressing again various interface textures (technology atlas most prominently, at various resolutions, as it's the biggest interface texture in the game). Try something similar with CivV - pick a few interface textures, recompress them, see if the game is still able to load them. If it is, then there's some kind of incompatibility with the input images you provided. If it isn't, then V uses a different format than BE.

I don't have CivV available at the moment, so i can't check this.

I've skimmed Dragon Unpacker source code after publishing my FPK unpacker, and from the comments there i've gathered that V used a bit different FPK format than BE (with dumb filename scrambling, for example). It's possible that FTXTInterface format also differs.

 

[LRN]

I've mentioned FPK differences earlier. Turns out, i misremembered this: filename scrambling was used in CivIV, not in CivV.

Since there seems to be a demand for this, i've managed to get a copy of CivV (BNW) and tested the program on it by doing the following:

• Unpacked UITextures.fpk with my FPK unpacker
• Moved the fpk file away (so that the game won't be able to load it)
• Moved the unpacked files into assets subdirectory (so that the game will be able to load them)
• Found a file to test (buildingatlas.dds and buildingatlas-index.dds)
• Decompressed buildingatlas.dds and buildingatlas-index.dds into buildingatlas.decompressed.dds
• Moved the buildingatlas.dds and buildingatlas-index.dds files away from the assets subdirectory
• Ran the game, looked at a building (Monument) in the civopaedia. The game said that it couldn't load buildingatlas.dds
• Closed the game
• Compressed buildingatlas.decompressed.dds into buildingatlas.dds and buildingatlas-index.dds
• Ran the game, looked at a building (Monument) in the civopaedia. The image displayed correctly, no errors.

Therefore i would conclude that FTXTInterface (de)compressor is CivV-compatible.