MAP, PIC and PAL formats figured out ! (almost)

[darkpanda]

I am remastering this post to publish releases of CIVED, latest being 0.0.4a, a freshly built PIC viewer that can read Civilization I PIC and MAP files, and convert them to GIFs.


CIVED has been refurbished and is now released as bundle in the JCivED toolbox: see this thread.

The original thread was kind of development log, and started as follows:

---------------------------------------

Hi there,

For my first post here, a little story:

I've been wanting to hack into Civ savefiles (SVE and MAP) for so many years, I finally took it upon me to progress on it...

I have been hacking SVE for a long time, and only parts of it remain mysterious (e.g. REPLAY data). The real challenge was to hack MAP files.

Through this journey, I read a couple of weeks ago this thread that indicated that a small fan soft for Microprose Darklands (PicViewer, see here) could actually open .MAP files as pictures.

I was intrigued, and read the doc accompanying PicViewer, that provided some insight into the Microprose PIC file formats, including Civ's MAP.

As the C source of PicViewer is available, I decided to (try to) port it to Java, but didn't succeed... The main issue was that neither PicViewer's author (his C code is partly reversed-engineered from CIV.EXE assembly) nor anyone I found so far could figure out exactly the details of the dictionary-based compression algorithm used on top of RLE for pictures.

So I searched, and took a guess at one of the most basic and famous one: LZW.

After a lot of research for various open source implementations of LZW (copyright-free since a couple years ago), trial and error, I could finally code a full, exact, parser of Civ's MAP files in Java!
Turns out, with a little more coding to parse color palette data, I can also extract all Civ's PIC files! At least in VGA, I am still working on EGA 16-colors palette format...

Saving back to MAP seems to work (Civ can load the savegame) but so far I failed to achieve identical byte-to-byte LZW compression when compressing back... Somehow I can't handle the LZW dictionary reset like Civ does when it gets full.

I didn't try for PIC files yet, just been working today.

Just for the sake, I am attaching a screenshot of my CIV folder, full of extracted GIFs!

TTYL

 

[darkpanda]

A quick update: I figured out the logic of 16-color EGA palette, so this should pave the way to customizing/modding PIC files

At some point - when I have enough time - I will post the details of those file formats, as well as the tools to manipulate them (Java API).

 

[jarvisc]

WOW, congratulations!! This sounds like a major development!

I registered for this forum just to tell you congrats. Have recently, myself, gotten back into civ1. Nice work, again! (By the way, how many facial expressions does Abe L. have?)

 

[darkpanda]

WOW, congratulations!! This sounds like a major development!

I registered for this forum just to tell you congrats. Have recently, myself, gotten back into civ1. Nice work, again! (By the way, how many facial expressions does Abe L. have?)

Thanks for the congrats!

For the details of Civ's inner workings, I really have no idea, so just by the look of it in the previous' post attachments, it seems every leader has 14 possible facial expressions, +1 the full bust frame in the lower right corner. For some leaders, not all 14 slots are used, but for Abe Lincoln, it seems like all 14 have content.

I take advantage of this reply to confirm that my piece of Java code is now able to code back virtually any image file (provided it's 320x200) into a Civ-compliant PIC file that looks good in VGA-256 colors (some more work needed to make it look nice in 16 colors)... So if some artists/modders out there are eager to test new leaders, unit or terrain looks, feel free to send me sample for test!
The code needs some cleanup and proper I/O before release.

For the sake, attaching the FR version of Civ's menu screen with a custom PIC based on Civ's box art.

 

[jarvisc]

Hey, thanks for the reply (and for answering my question

Nice box-art title screen. Keep up the good work!

 

[darkpanda]

A quick heads-up: after several attempts to put back arbitrary GIFs into arbitrary PICs, I had to come back to figuring out exactly why my re-encoding loop could not generate byte--to-byte perfect PIC files when re-using original CIV art...

Eventually, hours of debugging highlighted a couple of tiny - yet deadly - bugs and mishaps, but now I can proudly announce I did figure it out, byte-to-byte !

After I'll get some sleep, that means I will finally be able to try out actual PIC modding, and most likely release a piece of JAR for the amateurs out there...

Cheers

 

[darkpanda]

Of course I could not go to sleep without a quick try!

So I resorted to modify BACK0A.PIC, which is the background picture when negotiating with another Civ in ancient despotism or anarchy (see first attched picture)

I took a Mars landscape picture from the web, at this address: http://www.wolaver.org/Space/MarsLandscape.html:

[IMG width="640"]http://www.wolaver.org/Space/MarsLandscape.jpg[/IMG]

Then cropped it, added some white overlay at the bottom, and converted it to a 320x200 pixels 255-color indexed PNG (see second attached picture)

I then built the PIC (can't attach, format no recognized by the forum, pm if you want me to send it) from this input PNG, re-using the original BACK0A.PIC color palette (the not-so-beautiful result is seen in 3rd attachement).

Finally, launched a new game just to see how it would behave in-game, and you see the result in 4th and 5th attachments, meeting Shaka the Zulu and Queen Elisabeth...

It's ugly, sure, I am no artist, but it works!

 

[jarvisc]

I like the Mars landscape. That is hilarious. Well done!

 

[darkpanda]

This thread is kind of becoming my development blog

Anyway, back to my initial goal, being able to understand and modify my saved MAP files. In my initial post I stated that:

Through this journey, I read a couple of weeks ago this thread that indicated that a small fan soft for Microprose Darklands (PicViewer, see here) could actually open .MAP files as pictures.

What really prompted me to start this mini-project on my own was that I actually *could* open my MAP savegames with PicViewer, and was intrigued at its contents... Today I can open MAP files with my own code, and generate a GIF with their content (see first attachment).

MAP files actually contain 10 separate layers that are the size of the game map, plus some unused data which seems an arbitrary copy of what's in memory at the time of saving (namely icons for city improvements, and government advisors).

My rough guesses at what the 10 layers are:
- layer 0: terrain data, with 12 distinct values for the 12 valid terrain types
- layer 1: Per-Civ land occupation, mixed with land appeal for city-building (an overlay of layer 3)
- layer 2: area segmentation, with identifiers for separate land masses and inner seas; for some reason, the land where the player starts is not identified... I don't know why yet
- layer 3: terrain-based land appeal for the computer to build cities
- layer 4: same as layer 5 below, but only what's visible to the player
- layer 5: terrain improvements (irrigation, mining, roads)
- layer 6: same as layer 7 below, but only what's visible to the player
- layer 7: railroads, rods, rivers, fortresses
- layer 8: Per-Civ land exploration and active units
- layer 9: Mini-map render (attached to next post because of attachments limit)

I attached pictures for all those layers, if anyone is interested.

I am currently in the process of understanding them all, and possibly create a first basic map editor, maybe text-based command line, before aiming at something in the like of TerraForm...

 

[darkpanda]

Just to post the 9th layer (minimap render) and the corresponding Civ savefiles (SVE and MAP) if anyone is interested.

The game is Prince-level Babylonians, with heavy use of fast-settler cheat and save-load to win battles.

 

[jarvisc]

Personally I think what you are doing is fascinating. Just curious, in the map layers which can be viewed as images, is it one pixel per game tile? One byte per game tile?

 

[darkpanda]

Personally I think what you are doing is fascinating. Just curious, in the map layers which can be viewed as images, is it one pixel per game tile? One byte per game tile?

It is both: 1 game tile = 1 byte = 1 pixel

I did not elaborate on the PIC format yet, but basically they are byte-indexed images, i.e. each pixel is represented by 1 byte, representing the index of a color within a 256-color palette, which is the color of the pixel.
In some alternate PIC files (e.g. MAP.PIC, SPRITES.PIC, TORCH.PIC) each pixel is represented by half a byte, each half-byte representing the index of a color within a 16-color palette.

Anyway, I am thinking of a good way to share in more details about those file formats (dedicated website, project-hosting wiki + source code, etc.), but in the meantime, I am attaching my first releases of CIVED to this thread (see first post).

Enjoy!

 

[darkpanda]

Seems that editing posts doesn't mark the thread as being recently updated, so I am adding this post to notify of the release of CIVED 0.0.3a (see top post).

 

[Quintillus]

lurker's comment:

Interesting work. I've only played newer versions of Civ, but have done some work with scenario editing there. It's nice to see Civ1 getting some modding love, too.

I've suggest keeping the documentation about the file formats at a Civ fansite (either here or another one; I don't know if there's another site with more Civ1 activity than CFC). Mainly, because of visibility. For example, Gramphos did a nice job of this with the Civ3 scenario format at Apolyton (the links to sections of the format worked before Apolyton updated their forum software a year or so ago, which was nice). That way, any updates are more likely to be seen. The Modiki might also be an option. I'm not sure how much it's still maintained, and it's primarily Civ4, but it is quite civ-focused. Of course, a link to an external site from this thread might work too... but you'd probably still have to make posts in the thread about updates so that people were aware of them. I know I pretty much never go to random Civ sites without going through a forum first (whether by seeing an update, or more rarely, searching for something on the forum [not Google]).

 

[darkpanda]

Thanks a lot for the directions ! I will look into those...

 

[jarvisc]

Hi darkpanda, Thanks for the information and all your work. By the way, I took a stab at running cived 0.0.3a but ran into an exception. Oh well, I look forward to more of your progress!

 

[darkpanda]

Thanks for the heads-up jarvisc, I just published 0.0.4a which fixes the bug you mentioned... I should definitely put in place some kind of test framework, this was really a stupid bug !!!

 

[jarvisc]

Cool. Here is the "map" from my recent game (I magnified the attached image 2X).

I think what you've done is really neat.

 

[darkpanda]

Cool. Here is the "map" from my recent game (I magnified the attached image 2X).

I think what you've done is really neat.

Thanks a lot for your encouragements

I've not been active in the forum recently, but just to let you know I am spending a lot of time building an embryo of comprehensive "Civ savegame editor" that would include both SVE and MAP, possibly using native Civ graphics.

At the moment I am working on SVE a lot, but attached you will find a full MAP (terrain only) generated using native graphics and some guessed logic... I didn't achieve Civ-like look for coasts or special resources yet, though...

Cheers

 

[darkpanda]

A head-up after the holiday season !

I have spent a lot of time trying to design a modular GUI system to edit SVE data, and while I reached something usable, I put this work on pause in order to finish the one functionality I had aimed from the start: terrain editing.

So now, the CIVED-fork work-in-progress version I am working on (0.0.5a) can be used to edit geography without any (known) problems... But I am not satisfied with so little functionality ! So unless someone dies to have it under his hands, I am will postpone my next release to a point when I have implemented everything I can as far as SVE editing is concerned.

Attached, a screenshot of the current version:



Happy new year!