MAP, PIC and PAL formats figured out ! (almost)

[axx]

Great! Keep up the good work!

 

[Dack]

...Attached, a screenshot of the current version:...

Your rendering of the map is very impressive. The shoreline and rivers mouths are excellent. You took the time to intrepid the adjacent hills and mountains. Staring at my own feeble interpitation of these features, I just realized the hills and mountains like the rivers only connect at north-south and east-west axis. Again very nice indeed.

 

[darkpanda]

Your rendering of the map is very impressive. The shoreline and rivers mouths are excellent. You took the time to intrepid the adjacent hills and mountains. Staring at my own feeble interpitation of these features, I just realized the hills and mountains like the rivers only connect at north-south and east-west axis. Again very nice indeed.

Thanks a lot, Dack !

As a matter of fact, after being able to export the PIC image data, it gives a lot of insight about the map graphics generation. Here are SP257.PIC and TER257.PIC, exported with CIVED 0.0.4a (don't know if there's any copyright issue putting them here ? If yes, let me know and I'll remove them):

SP257.PIC TER257.PIC

You see in TER257.PIC that for each terrain type, there is a row of 16 different terrain 16x16 tiles... Actually, we can identify a relationship between the tile index in the row with the type of adjacent terrains:
0 = 0b0000 = N, E, S and W are all different types
1 = 0b0001 = Only N has the same type
2 = 0b0010 = Only E has the same type
3 = 0b0011 = N and E have the same type
4 = 0b0100 = Only S has the same type
5 = 0b0101 = S and N have the same type
etc...

So if you set 4 flags based on adjacent terrain types, it directly gives you the proper tile index in the terrain tiles' row... Pretty neat to implement.

But there are several exceptions:
- for some reason, Civ does not use this mechanism for all terrain types; for example arctic and jungle always use the single tile graphics, even when there are adjacent arctic/jungle tiles:



- for the shoreline, the algorithm is more complex: Civ not only looks at NWSE neighbours, but also at corner neighbours NW, NE, SW, and SE... depending on whether those are also shore, land, or sea, Civ uses mini-8x8 tiles overlaid on the default shoreline tile - I didn't finish figuring out this algorithm yet:



- for river mouths, there are also additional tiles to be overlaid on the shoreline tiles that have a river tile adjacent to them

Other than that I am pretty happy about the rendering indeed...

I have now started to look into the formulas to define the location of special resources, huts and grasslands specials. Although I have found a number of posts about it, does anyone have them already figured out ?

Dack, I read you identified a TerrainMasterWord (which I believe would be the seed used by the random routine for generating the map), where is it in the SVE ?

 

[jarvisc]

Darkpanda, you are uncovering a fascinating detective story. Thanks for keeping us updated.

 

[darkpanda]

Just a heads-up to announce the imminent release of the Java-based game save editor I have been working on...

It will come bundled with the PIC editor in a kind of Civ 1 "tool box", in a matter of days.

Cheers

 

[Mize]

Damn, makes me want to see the 'real' jungle ingame. This is great stuff.

 

[darkpanda]

Just a heads-up to announce the imminent release of the Java-based game save editor I have been working on...

It will come bundled with the PIC editor in a kind of Civ 1 "tool box", in a matter of days.

Cheers

Just for the sake of notifying: CIVED has been refurbished and is now released as bundle in the JCivED toolbox: see this thread.

 

[Dark_Nephi]

This looks great guys. Im looking forward to map editing my civ 1 game. keep up the good work and keep everyone posted. On my other computer I was playing win civ and changed all the .wav files to other ones.. go so used to it that when I accidently changed them back I was sad. #editingcivFTW

 

[darkpanda]

Damn, makes me want to see the 'real' jungle ingame. This is great stuff.

Now you can See over there -> Modding Civ I - Bug-fixing: terrain layout

 

[Quintillus]

A quick update: I figured out the logic of 16-color EGA palette, so this should pave the way to customizing/modding PIC files

At some point - when I have enough time - I will post the details of those file formats, as well as the tools to manipulate them (Java API).

Is the Micro Prose PIC format documentation available somewhere, or have I missed it? I was wondering since I saw one of Theov's posts, and thought, "Wouldn't it be cool to be able to use Civ1 graphics in Civ3 and have a retro look, or vice versa?" So I started looking into the PIC format, but found that it appears MicroProse's variant is different than the common PIC format. Which doesn't surprise me too much - their PCX for Civ3 was also not quite standard, though only the transparency differed there.

At any rate I've been looking at a PIC file in the hex editor, and reading the Darklands documentation you linked to. I also took a look at the code on SourceForge (hope that's okay), and between the two I think I have an idea of where to start. The palette looks pretty straightforward, and similar to other image formats. The image itself is a little trickier. As I understand it, to read it, LZW decoding is needed first, followed by RLE encoding. It's late so I haven't looked too much into the details of the compression, such as the specifics of RLE encoding (I know from working with bitmaps that there are a few variants, but once the variant it known it's straightforward). I wonder how much space they saved by going with LZW + RLE versus just one or the other?

I'm also curious if you already have code that converts PIC files into a standard Java Image object (java.awt.Image), or something easily convertible into an Image. From the poking around I've done so far, it looks like you convert them into a custom ImageData object, and then use that to create a GIF, but I'm not sure when that might be easily convertible to a java.awt.Image. Since JCivED can display PIC files, I reckon there's probably a way to do something like that... just not sure if there's a good API-like way to do so.

At any rate, I think it would be cool to be able to do a MicroProse PIC to a Firaxis PCX conversion without an intermediary GIF stop. I can probably piece it together given the Darklands documentation and general LZW documentation, but figured I'd comment first in case there were better documentation or an easy way to do it that I was missing.

 

[SWY]

Quintillus, I've used darkpanda's code and converted it to C#... it's available in the CivOne source code. You can use the source code of CivOne for any purpose you like (Creative Commons licence), feel free to have a look.

My code translates the PIC files to a .NET Bitmap object. You should be able to go anywhere from there...

 

[Quintillus]

SWY, thanks. That's helpful. Although I usually use Java, your C# code is more similar in style to what I usually write than the JCivED code, so it's easier to grasp quickly. The extra comments are also appreciated.

 

[darkpanda]

Is the Micro Prose PIC format documentation available somewhere, or have I missed it?

Like several other things, my wish to document has remained at the "wish" state so far...

I'm also curious if you already have code that converts PIC files into a standard Java Image object (java.awt.Image), or something easily convertible into an Image.
[...]
From the poking around I've done so far, it looks like you convert them into a custom ImageData object, and then use that to create a GIF, but I'm not sure when that might be easily convertible to a java.awt.Image. Since JCivED can display PIC files, I reckon there's probably a way to do something like that... just not sure if there's a good API-like way to do so.

JCivED is almost only relying on java.awt.BufferedImage (a subclass of java.awt.Image), so there is definitely code in JCivED that will "convert" a PIC image file into a BufferedImage... But then, just trying to locate the code that you ask on SourceForge (my local copy has been updated a lot since the last sourceforge commit), I realized it is not very obvious...

I apologize for this it is true that I never focused too much on making JCivED source code readable by others, and I keep re-organizing the code a lot.

Anyway, here are the 2 classes/methods you are interested in:

• dd.commons.io.ImageFileReader.readPIC(): creates a PicImage object from a .PIC file:
  

  PicImage picImage = ImageFileReader.readPIC(new FileInputStream("XXX.PIC"));

• dd.commons.graphics.pic.PicImage.toBufferedImage256(): returns a 256-color java.awt.BufferedImage (a sub-class of java.awt.Image) corresponding to the 256-color version of the PIC:
  

  Image image = picImage.toBufferedImage256();

 

[Quintillus]

Ah, that toBufferedImage256 method in PicImage is the one I hadn't found! I'd located the readPIC one, but that was only half the solution. So instead, I'd tried extracting the LZW/RLE methods and combining it with my own code for reading little-endian files and extracting the palette, using the CivOne C# code as a model for how to glue it together. Unfortunately, something didn't quite work out (maybe due to using a different little-endian file reader? Or perhaps I made a mistake when gluing together the code), and I was getting different errors depending on the PIC file I tried. I never did completely understand the LZW code, either - particularly how the variable bit lengths for codes were handled - which made debugging difficult.

I'll try it again with the toBufferedImage256 method, and I'm optimistic that will work. Then it's just a matter of adding the ability to export to PCX, as I realized I've written a reader, but my current writer only modifies the palette, not the actual bitmap. That should be fairly easy, though, since I already have a pretty thorough understanding of PCXs.

 

[illepokok]

Ah, that toBufferedImage256 method in PicImage is the one I hadn't found! I'd located the readPIC one, but that was only half the solution. So instead, I'd tried extracting the LZW/RLE methods and combining it with my own code for reading little-endian files and extracting the palette, using the CivOne C# code as a model for how to glue it together. Unfortunately, something didn't quite work out (maybe due to using a different little-endian file reader? Or perhaps I made a mistake when gluing together the code), and I was getting different errors depending on the PIC file I tried. I never did completely understand the LZW code, either - particularly how the variable bit lengths for codes were handled - which made debugging difficult.

I'll try it again with the toBufferedImage256 method, and I'm optimistic that will work. Then it's just a matter of adding the ability to export to PCX, as I realized I've written a reader, but my current writer only modifies the palette, not the actual bitmap. That should be fairly easy, though, since I already have a pretty thorough understanding of PCXs.

I just built a version in Javascript after going through the JCivEd source and reading all of the posts on this forum. Took a long time to wrap my head around how it works. You have to take three passes through the data (once to pull out integers of increasing size) followed by the LZW decompression followed by the run-length encoding. And of course, if any of those are wrong, your picture will look like TV fuzz. But once you see that first picture.... pure elation!

The key files that helped me figure it out from JCivEd were:
• LZWDecoder.java
• LZWDictionary.java
•ImageData.java
• ImageDataX0.java

Hopefully that gets you started. I'm not sure a loosely typed language like Javascript will help with C#, but happy to post that code if it will help.

Huge props to DarkPanda for your work figuring this stuff out!

 

[Quintillus]

Thanks, will keep that in mind when I get back around to it. I started a new job the day after my last post, which led to being distracted from my own coding projects, and I wound up doing other work when I got back to them. I'm actually coding in Java, and was just using the C# code as a model for integrating the Java part with my own work, since the styles of Java were fairly different but the C# coding style was similar to my Java coding style. Wouldn't be opposed to seeing the JavaScript version though. And not even so much for my own benefit, as for the benefit of anyone who may come next and find it useful. I think on the whole a lot of working could have been saved in the Civ modding community over the years by more code being available, and if someone comes along wanting to work with these formats and JavaScript is their language of choice, it'll be all the better if an implementation in their language of choice is available.

That said, like Weird Al I'm fluent in JavaScript (though not Klingon), so even though Java is my primary language, I may be able to benefit from it as well.

 

[darkpanda]

I just built a version in Javascript after going through the JCivEd source and reading all of the posts on this forum. Took a long time to wrap my head around how it works. You have to take three passes through the data (once to pull out integers of increasing size) followed by the LZW decompression followed by the run-length encoding.
[...]

Huge props to DarkPanda for your work figuring this stuff out!

Special thanks for the props, because this truly was the starting point of the whole JCivED experience for me, 3 years ago...

By the way: I am pretty sure the whole co/dec can be programmed in a streaming way, each layer (input PIC, lzw, variable bit length integers, rle, X0) consuming the bytes from the previous one and feeding the next one. In the end it would be just 1-pass, although the complexity remains O(N) in both cases.

 

[darkpanda]

Also: I would be interested in the JS port if you care to share

 

[illepokok]

As I have been going through decoding / rendering the maps, I had to translate the "pixel" level values into the appropriate tiles. Figured I would post here to continue to the documentation development effort

Layer 0: Terrain
0x1 - Water
0x2 - Forrest
0x3 - Swamp
0x4 - Unknown (have yet to see it used)
0x5 - Unknown (have yet to see it used)
0x6 - Plains
0x7 - Tundra
0x8 - Unknown (have yet to see it used)
0x9 - River
0xA - Grass Lands
0xB - Jungle
0xC - Hills
0xD - Mountains
0xE - Desert
0xF - Arctic

Layer 4/5 - Terrain Improvements
0x2 - Irrigation
0x4 - Mine
0x8 - Road/Rail
0xA - Road/Rail + Irrigation
0xB - City
0xC - Road/Rail + Mine

Each "pixel" in Layer 4 is really a bit-flag where bit 0 indicates City, bit 1 indicates Irrigation, bit 2 indicates Mine and bit 4 indicates Rail. The only confusing part is that Cities have irrigation and roads automatically, thus you will never see "0x1", only "0xB" (1 + 2 + 8).

There doesn't seem to be any distinction between roads and rails in this bitflag

 

[illepokok]

Also: I would be interested in the JS port if you care to share

I just threw a work-in-progress version online:

http://cived.tigermonkey.org

If you choose a local .map file, it will render the map to a canvas element. The functionality is in there to render the VGA/EGA graphics as well, but in moving on to map rendering, it's no longer exposed in the UI.

The code also leverages the Quintus JS Game Engine to make plotting TileSheets a bit simpler.

Still lots to do re: terrain improvements, plotting cities, units and stuff, but hey, it's a start!