[Col] Colonization picture viewer

[angrylion]

Hello, gentlemen! A New Year present from Russia for all Colonization fans!
I've written a picture viewer for Colonization (DOS version only). It seems I'm the first one to have deciphered the compression algorithm. I hope it's not against the rules of this forum and I won't get punished .
It's just a picture viewer, it can't edit pictures or save them in different formats (if you want a screenshot, run it in DosBox). In addition, the game is so old. So I hope it won't irritate anybody. I also don't provide source code so nobody will be able to decipher the game further.
Anyway, this viewer is a DOS program. You can also run it under DosBox or 32-bit Windows.
Colonization (DOS) stores graphics in files of two types - with .SS extension and with .PIK extension.
So download my program and unpack it into the directory where Colonization is installed. Then execute it and enter "1" and then Enter if you'd like to view a .SS file, or enter "2" and then Enter if you'd like to view a .PIK file. Then input the name of the desired file without the extension. It will be shown. Press Escape to quit.
Some .SS files contain not just one picture, but several pictures. For example, icons.ss, building.ss, terrain.ss, bdark.ss, phys0.ss, mpslogo.ss, mpsname.ss and clos-fwk.ss contain several pictures. To view the next picture in these .SS files press -> (cursor right), to view the previous one press <-.
Also there's a peculiar file named opening.pik. It's too large to fit 320*200 VGA screen. You can, however, scroll it with two aforementioned cursor keys.
On the attached screenshot you can see what's hidden inside the file called CLOS-BKG.PIK.
Have fun and comment!

 

[angrylioneater]

I will punish you!

 

[angrylioneater]

you could at least have been honest and tell why you don't have any sourcecode... you just took the disassembled source of the original colonization (either viceroy.exe or mapview.exe). I really hoped somebody found out more about this compression (pFAB to be exact).

 

[angrylion]

Yes, I have basic decompression routines disassembled from mapedit.exe.
And no, I do have the source code. It is in ASM. You have to severely rewrite what a disassembler outputs to compile the listing again. So I disassembled mapedit.exe, then modified the listing so that it compiles under MASM 6.11, then introduced some kind of user interaction and some additional checks into the source, and here it is. I also dropped from the source things unimportant to the viewer (e.g. EMS, XMS support, editing functions of mapedit.exe, and many other things). Some pictures didn't want to open with standard mapedit.exe routines, so I had to tweak them a bit.
So I have the source code and it compiles under MASM 6.11 for DOS. I couldn't have done the viewer without the source code, its constant modification and debugging. I just won't disclose the source code, because it contains some ripped parts.
As for MADSPACK decompression (or pFab, as you like), I do know what particular routines execute it. MADSPACK is Microprose's internal name for the whole algorithm, while pFab stands in the beginning of symbolic names of the key decompressing routines. I actually tried to RE them, to rewrite them into C, but I found it difficult. These routines were written in pure asm, and include a lot of opcodes (RCR, SAR, SHR) I don't know how to reproduce in C.
Also you will never find out anything about MADSPACK compression. All of Microprose executables include decompressing only.

 

[thomasd]

Hi, time to move in as well...

First of all, thanks for the program and the explanation! I did disassemble the mapedit.exe as well (using IDA) and made it compile with MASM...
It did work, although I didn't change anything, because I was just curious about the whole MADSPACK algorithm.

I think I am at the same point as you... found the routines responsible for decompression and tried to rev-eng them. But it's just to difficult...
Concerning the FAB - I think its just a marker in the compressed files, i.e. look at terrain.ss, there are several FAB sections here, one for each stored image.
The bad thing is, while I do know where they are decompressed and I can even look at the asm-source, I can't make much sense of it, it's just to difficult to read.
God, I wished I could read assembler like a high-level language ;-)

 

[angrylion]

It's really amazing that you were able to compile it and get it working without modifying the IDA listing. Or maybe my version of IDA is outdated...
Anyway, it's amazing, because there's a lot of absolute adressing against an offset of a segment (I mean, pointers to code). And there's a lot of instructions like "sub sp, 0h". This instruction can be compiled as 81EC0000h or, alternatively, as 83EC00h. And it appears that MASM 6.11 compiles it not in the same manner as the original developer's tools. So, in my case, all segment addressing got screwed until I replaced absolute offsets with labels. It's just an example. I had to rewrite a lot of stuff to compile IDA listings in MASM 6.11.
Or I probably have misunderstood you? You probably meant that you modified IDA listings but you didn't modify the program' behaviour. Yes, this interpretation of yours seems more likely.

As for FAB, I do know that these symbols are kind of markers separating different sections within compressed files. I also know where in the code they check for the presence of this symbols.
But I talked about pFab in symbolic names. You should know that mapedit.exe contains symbolic (aka debugging) info (i.e. names of functions and variables) in a very specific format called NB02. This NB02 can be viewed only in CodeView 3.11/3.14. So get this old CodeView debugger, load mapedit.exe into it, then go to Options>>Source Window and there check "Source" and check "Show symbolic name". You'll see this debugging info but not entirely. Press F8 once and you'll see all of it. Then you can save the CodeView listing with debugging info. Symbolic info will help you if you continue your efforts.

EDIT: You're plain wrong about the role of "FAB" strings in compressed files. They separate sections of the file, not every compressed image. There's only 4 instances of "FAB" in terrain.ss, while with my viewer you'll see that there's more than twenty pictures in terrain.ss.
EDIT2: CodeView is so stupid that it only can save up to 64 Kb of a disassembled code starting from the segment currently being debugged. It's not enough for mapedit.exe. So you'll have to trace mapedit.exe until you're in another segment and save your listing once again being there. Then you'll have to concatenate two CodeView listings to get the full disassembly of mapedit.exe with symbolic info. I've also written a program to concatenate CodeView listings automatically and can share it if you like (it doesn't contain anything proprietory).

 

[thomasd]

Hmm, I have IDA 5.2 here and it did work. But my MASM seems to be older, something like 5.x...

Anyway, thanks for your information - I'm trying to find the old CodeView (so far I only got my hands on version 4.0 - which doesn't load this NB02 Format.
But I am going to find an older one and start hacking away. I would really like to understand the compression algorithm.

cheers.

PS: sorry for the FAB / pFab mixup - I know that pFab is the prefix of some of the function names in mapedit.exe. FAB seemed to me as a "marker" in the first place.
But how are the single pictures in .SS files encoded? I thought each image on its own - with the FAB as marker... Still researching...

EDIT: If you could send me your concatenator via PM I would be really grateful!
EDIT2: You understood me correct - I didn't modify the program's behaviour - the asm listing still had to be modified to compile correctly.

 

[angrylion]

But how are the single pictures in .SS files encoded? I thought each image on its own - with the FAB as marker... Still researching...

No, all compressed images within .SS are placed in one FAB section. Another FAB section contains palette (don't know which one). The section from the beginning up to the 1st FAB is a kind of header. As for other FAB sections, I don't know.
I sent you a PM.

 

[Zharramadar]

HI ppl,

I was researching about these file formats too, even talked with Brian Reynolds about it, but he said he don't remember it, and that probably any original Colonization file is destroyed by now.

Angrylion, check your PM, just sent you.

I was doing this parse as I'm doing a multiplayer remake of Colonization using SDL, and I want to use the original files.
I will research further for FF files, which use MADSPACK as well.

 

[angrylion]

Check your email. Too bad if Colonization's source code was lost.
Colonization is not the only game using MADSPACK compression (and MADS game engine, in general), though. There are also:
Rex Nebular (1992)
Return of the Phantom (1993)
Dragonsphere (1993)
Bloodnet (1994) - maybe?
Darklands (1992) - maybe?
EDIT: I suppose all these games were programmed by Reynolds. Maybe their source code wasn't lost?

 

[Zharramadar]

E-mail checked, read, and replied

Yeah, probably he made those too, they are from Microprose as well... or just used his engine....

He told me everything should be lost by now....

Let me quote him:

I wish I could help you with the file formats, but I&#8217;m afraid I don&#8217;t have any of the materials for the game. Not only have I changed companies twice since those days, but also the original company I was with (Microprose) has changed hands many times, being sold to Spectrum Holobyte, to Hasbro Interactive, to Infogrames, and now I think the parent company is Atari. The original studio/office was closed several years ago too, so I&#8217;m not sure if the source material even exists any more, alas. I remember that &#8220;SS&#8221; files were &#8220;sprite series&#8221; files, and might contain one sprite or an animation. We used a variable run-length-encoding scheme, but boy I don&#8217;t remember a thing other than that. I&#8217;m afraid I can&#8217;t even remember what an FF was at all &#8230; maybe it was instructions for an animation. Possibly PIX represents some generic picture file format. I hate to say it but your only conceivable avenue forward is likely to be through Atari, and after 13 years I&#8217;m not sure they could find the stuff even if they want to.

A pity indeed, but I also contacted Atari, as he suggested, but I got no reply (as I expected lol)...

 

[angrylion]

Reynolds appears to be a helpful guy.
I also sent an email to Firaxis about a year ago and also received no reply.
FF files contain fonts, it's pretty obvious, if you'd look into symbolic info. "SS" (indeed stands for sprite series) files contain either one or several small images. PIK files contain one big picture.

 

[Zharramadar]

Reynolds appears to be a helpful guy.
I also sent an email to Firaxis about a year ago and also received no reply.
FF files contain fonts, it's pretty obvious, if you'd look into symbolic info. "SS" (indeed stands for sprite series) files contain either one or several small images. PIK files contain one big picture.

Indeed, he seems helpful, I didn't thought I would be replied.
Yeah, I though FF was font files in the moment I saw them, this e-mail from Reynolds I got is quite old now, when I started to try to gather some info about the files.

 

[jfren484]

I have written a program that opens colonization files (.SS and .PIX so far) and will eventually show the images in them. Coloview.exe works, but I want to do something that will work in regular windows. So far I've written (completely in C# with no disassembling and reassembling) the code that can unpack a file into it's various sections. For .SS files, I see that the first section is a header of some sort (I'm only getting a couple meaningful pieces of information from it), the second section is a list of all of the images in the .SS file (16 bytes per image, most of which I understand, but a couple of UInt16 values I'm not sure about), the third is a palette of some sort (but is something other than simple 24-bit RGB values), and the fourth is the image data itself. Does anyone have any other information about the sections? I'd be happy to give out the code when I'm finished.

 

[thomasd]

Wow, that's further than I have gotten!

The palette data is not 24bit but simple 8-bit RGB - remember, this game is from the 90's . I made a simple "palette viewer" once (in java) which tries to load and show the palette embedded in the .SS and some .PIK files.

How did you manage to identify the contents of the sections?

 

[Fankman]

hi guys,
thanks angrylion for this great tool

i have a problem with the black background-color.
i will cut out the pictures. but the founding fathers have partially the same clothes-color like the background and it's impossible to cut out those shapes.
now, how i can change the background-color (for example in 'pink' or 'red')?
it is possible only to change a hex-value with a hex-editor?

thanks for your ideas

 

[angrylion]

Hey Fankman,
I've added this feature for you. It's very basic but should work. Tell me if you need a broader selection of background colors.

 

[Fankman]

Hey, thanks very much for your fast answer, i'm impressed
The updated tool works great, you are amazing, it is just what i need
Now i can edit the pics in Gimp and select the pink color with the color-picker. I think this one is the right choice, the col-graphics will not use this color.

Spoiler :

Well i can arrange my next pic with the founding fathers, just like this:

Spoiler :

 

[Roscado]

Hi,
I'm trying to translate the game Rex Nebular but some texts are compressed in the file "GLOBAL.HAG".
I read that the compressor used is MADSPACK 2.0, MADSCONCAT 1.0 or pFab.

I need these programs (I can not find them)
to extract, edit text, images ... and later compress the file "GLOBAL.HAG" again.

Also if possible a small guide to using the programs.

A greeting and thanks for your time.

 

[m4D]

hi.

a bit offtopic, but did anyone of you guys tried to mess with colowin? looks like sprites aren't compressed there, but i can't find a palette for now. can anuone give me a hint on this?

b.r.,
m4D.