As already stated in this thread, the Earth Map of Civilization I (for DOS) is stored as an image file named "MAP.PIC".
The image data is compressed, but I've found a way to encode any custom map so that it can be used instead of the default Earth Map.
I've written a program that will do the encoding for you. All you need to do is to write your own custom map to a simple text file (see below for the required format).
[edit]
I've updated the program and simplified the code at one point, which also results in a slightly reduced output file size.
Also, I've added an example text file which, if left unchanged, will result in the standard earth map of civ1. You can use this file to make small changes to the standard earth map without having to write the whole map to a text file beforehand
[/edit]
I've also included the C++ source code, so if anybody is interested in the encoding algorithm, have a look at it.
Below you will find a part of the readme file for my encoding program, and further below I'll discuss the image file format for "MAP.PIC".
Have fun
MAP.PIC FILE FORMAT
(you don't have to read this unless you're interested in the encoding method; scroll down to the end of this posting to find the attached program)
The file is divided into 2 parts.
The first part (palette information) begins with a small header (4 bytes). It contains a "magic number" (2 bytes, always "0x4D, 0x30") and a size specification (2-byte integer, default value: 0x302).
After that comes the palette data. Its length is equal to the size value specified in the header, and it's divided into 16 color entries (3 bytes each: red, green, blue), followed by 720 empty bytes (theoretically providing enough space for 240 more color entries, but this image has only 16 colors).
The second part (pixel information) also begins with a small header (4 bytes), which contains a "magic number" (2 bytes, always "0x58, 0x31") and a size specification (2-byte integer, default value: 0x42C), whose value is equal to the total length of the rest of the second part.
The header is followed by the heigth and width of the image in pixels (2 bytes each, default heigth: 0x140, default width: 0xC8). After that, there's a single byte with the default value "0x0B", which seems to specify the encoding method, It's best left as it is.
Following that, we have the pixel data (320 x 200 = 64,000 pixels), which is compressed in three ways:
1. Each pair of two pixels (represented by their resp. palette indices between 0x0 and 0xF) is merged into one byte, with the first pixel taking up the less significant bits.
For example, the pair "ocean, then forest", represented by "0x1 (ocean ID), 0x2 (forest ID)" would be merged into "0x21". This works because there are only 16 colors, so each color can be specified by 4 bits, i.e. "half a byte".
The result of step 1 is a stream of 32,000 bytes.
2. If several identical bytes follow each other, they are now run-length encoded, forming a block of 3 bytes with this format
"0xPP, 0x90, 0xTT",
where "0xPP" is the value of the recurring byte, "0x90" is the RLE (run-length encoding) indicator, and "0xTT" is the repeat counter. The byte "0xPP" will be repeated "0xTT" times.
For example, the block
"0x21, 0x90, 0xFF"
would result in the aforementioned ocean/forest pair being written 0xFF = 255(dec) times.
It seems that RLE blocks can go across image row borders, but I would not recommend that.
Since 0x9 is the "river" ID and 0x0 is the background color ID, the value 0x90 is never used except as the RLE indicator.
3. After the RLE, a dictionary-based algorithm is used for further compression. Bytes can be raw pixel data (maybe RLE compressed) or dictionary pointers. To distinguish one from the other, each byte is followed by one or more indicator bits(!).
The first 256 bytes are each followed by a single indicator bit.
The next 512 bytes are each followed by two indicator bits.
The next 1024 bytes are each followed by three indicator bits.
The next 256 bytes are each followed by a single indicator bit.
etc.
Since a single byte can only assume 256 different values, it makes sense that after 256 bytes, more than one indicator bit is needed to specify dictionary pointers. After 256+512+1024 bytes, the dictionary table seems to be reset.
Now for the important thing: If all indicator bits are set to zero, all bytes are treated as data (with RLE compression if applicable).
This gives us a nice workaround without having to worry about the (complicated) dictionary-based encoding algorithm.
What we DO have to worry about is how the indicator bits are stored. Imagine the bytestream after the RLE, sorted from left to right. Each bytes is represented by 8 bits, starting with the least(!) significant bit. Example:
"0x21, 0x90, 0xFF" would be
"10000100 00001001 11111111".
Now the indicator bits are inserted in-between the bytes. Assuming that there have been less than 256 bytes, one indicator bit is inserted after each byte:
"100001000000010010111111110"
Now the whole bitstream is again converted to bytes (again, least significant bit first). Note that these new bytes can be each made up of parts of two of the "old" bytes, as well as indicator bits. So the new bytestream does not look very similar to the actual data.
Side note:
After finding out how the RLE is implemented, I googled for "RLE" and "0x90" and found this interesting page:
http://www.darklands.net/files/PicFileFormat.txt
It seems that the image formats of Civ1 and Darklands are quite similar to each other. The Darkland image viewer can even open Civ1 savegame maps!
The image data is compressed, but I've found a way to encode any custom map so that it can be used instead of the default Earth Map.
I've written a program that will do the encoding for you. All you need to do is to write your own custom map to a simple text file (see below for the required format).
[edit]
I've updated the program and simplified the code at one point, which also results in a slightly reduced output file size.
Also, I've added an example text file which, if left unchanged, will result in the standard earth map of civ1. You can use this file to make small changes to the standard earth map without having to write the whole map to a text file beforehand
[/edit]
I've also included the C++ source code, so if anybody is interested in the encoding algorithm, have a look at it.
Below you will find a part of the readme file for my encoding program, and further below I'll discuss the image file format for "MAP.PIC".
Have fun
MAP.PIC FILE FORMAT
(you don't have to read this unless you're interested in the encoding method; scroll down to the end of this posting to find the attached program)
The file is divided into 2 parts.
The first part (palette information) begins with a small header (4 bytes). It contains a "magic number" (2 bytes, always "0x4D, 0x30") and a size specification (2-byte integer, default value: 0x302).
After that comes the palette data. Its length is equal to the size value specified in the header, and it's divided into 16 color entries (3 bytes each: red, green, blue), followed by 720 empty bytes (theoretically providing enough space for 240 more color entries, but this image has only 16 colors).
The second part (pixel information) also begins with a small header (4 bytes), which contains a "magic number" (2 bytes, always "0x58, 0x31") and a size specification (2-byte integer, default value: 0x42C), whose value is equal to the total length of the rest of the second part.
The header is followed by the heigth and width of the image in pixels (2 bytes each, default heigth: 0x140, default width: 0xC8). After that, there's a single byte with the default value "0x0B", which seems to specify the encoding method, It's best left as it is.
Following that, we have the pixel data (320 x 200 = 64,000 pixels), which is compressed in three ways:
1. Each pair of two pixels (represented by their resp. palette indices between 0x0 and 0xF) is merged into one byte, with the first pixel taking up the less significant bits.
For example, the pair "ocean, then forest", represented by "0x1 (ocean ID), 0x2 (forest ID)" would be merged into "0x21". This works because there are only 16 colors, so each color can be specified by 4 bits, i.e. "half a byte".
The result of step 1 is a stream of 32,000 bytes.
2. If several identical bytes follow each other, they are now run-length encoded, forming a block of 3 bytes with this format
"0xPP, 0x90, 0xTT",
where "0xPP" is the value of the recurring byte, "0x90" is the RLE (run-length encoding) indicator, and "0xTT" is the repeat counter. The byte "0xPP" will be repeated "0xTT" times.
For example, the block
"0x21, 0x90, 0xFF"
would result in the aforementioned ocean/forest pair being written 0xFF = 255(dec) times.
It seems that RLE blocks can go across image row borders, but I would not recommend that.
Since 0x9 is the "river" ID and 0x0 is the background color ID, the value 0x90 is never used except as the RLE indicator.
3. After the RLE, a dictionary-based algorithm is used for further compression. Bytes can be raw pixel data (maybe RLE compressed) or dictionary pointers. To distinguish one from the other, each byte is followed by one or more indicator bits(!).
The first 256 bytes are each followed by a single indicator bit.
The next 512 bytes are each followed by two indicator bits.
The next 1024 bytes are each followed by three indicator bits.
The next 256 bytes are each followed by a single indicator bit.
etc.
Since a single byte can only assume 256 different values, it makes sense that after 256 bytes, more than one indicator bit is needed to specify dictionary pointers. After 256+512+1024 bytes, the dictionary table seems to be reset.
Now for the important thing: If all indicator bits are set to zero, all bytes are treated as data (with RLE compression if applicable).
This gives us a nice workaround without having to worry about the (complicated) dictionary-based encoding algorithm.
What we DO have to worry about is how the indicator bits are stored. Imagine the bytestream after the RLE, sorted from left to right. Each bytes is represented by 8 bits, starting with the least(!) significant bit. Example:
"0x21, 0x90, 0xFF" would be
"10000100 00001001 11111111".
Now the indicator bits are inserted in-between the bytes. Assuming that there have been less than 256 bytes, one indicator bit is inserted after each byte:
"100001000000010010111111110"
Now the whole bitstream is again converted to bytes (again, least significant bit first). Note that these new bytes can be each made up of parts of two of the "old" bytes, as well as indicator bits. So the new bytestream does not look very similar to the actual data.
Side note:
After finding out how the RLE is implemented, I googled for "RLE" and "0x90" and found this interesting page:
http://www.darklands.net/files/PicFileFormat.txt
It seems that the image formats of Civ1 and Darklands are quite similar to each other. The Darkland image viewer can even open Civ1 savegame maps!
