1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

MAP, PIC and PAL formats figured out ! (almost)

Discussion in 'Civ1 - General Discussions' started by darkpanda, Oct 18, 2012.

  1. axx

    axx Chieftain

    Joined:
    Feb 19, 2011
    Messages:
    93
    Great! Keep up the good work!:band:
     
  2. Dack

    Dack Terra Form Moderator

    Joined:
    Nov 26, 2003
    Messages:
    532
    Location:
    civ dos 474.05
    Your rendering of the map is very impressive. The shoreline and rivers mouths are excellent. You took the time to intrepid the adjacent hills and mountains. Staring at my own feeble interpitation of these features, I just realized the hills and mountains like the rivers only connect at north-south and east-west axis. Again very nice indeed.
     
  3. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
    Thanks a lot, Dack !

    As a matter of fact, after being able to export the PIC image data, it gives a lot of insight about the map graphics generation. Here are SP257.PIC and TER257.PIC, exported with CIVED 0.0.4a (don't know if there's any copyright issue putting them here ? If yes, let me know and I'll remove them):

    SP257.PIC SP257.PIC.X0.gif TER257.PIC.X0.gif TER257.PIC

    You see in TER257.PIC that for each terrain type, there is a row of 16 different terrain 16x16 tiles... Actually, we can identify a relationship between the tile index in the row with the type of adjacent terrains:
    0 = 0b0000 = N, E, S and W are all different types
    1 = 0b0001 = Only N has the same type
    2 = 0b0010 = Only E has the same type
    3 = 0b0011 = N and E have the same type
    4 = 0b0100 = Only S has the same type
    5 = 0b0101 = S and N have the same type
    etc...

    So if you set 4 flags based on adjacent terrain types, it directly gives you the proper tile index in the terrain tiles' row... Pretty neat to implement.

    But there are several exceptions:
    - for some reason, Civ does not use this mechanism for all terrain types; for example arctic and jungle always use the single tile graphics, even when there are adjacent arctic/jungle tiles:

    CIVED_jungle_tiles.png

    - for the shoreline, the algorithm is more complex: Civ not only looks at NWSE neighbours, but also at corner neighbours NW, NE, SW, and SE... depending on whether those are also shore, land, or sea, Civ uses mini-8x8 tiles overlaid on the default shoreline tile - I didn't finish figuring out this algorithm yet:

    CIVED_shore_tiles.png

    - for river mouths, there are also additional tiles to be overlaid on the shoreline tiles that have a river tile adjacent to them

    Other than that I am pretty happy about the rendering indeed...

    I have now started to look into the formulas to define the location of special resources, huts and grasslands specials. Although I have found a number of posts about it, does anyone have them already figured out ?

    Dack, I read you identified a TerrainMasterWord (which I believe would be the seed used by the random routine for generating the map), where is it in the SVE ?
     
  4. jarvisc

    jarvisc Chieftain

    Joined:
    Oct 24, 2012
    Messages:
    50
    Darkpanda, you are uncovering a fascinating detective story. Thanks for keeping us updated.
     
  5. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
    Just a heads-up to announce the imminent release of the Java-based game save editor I have been working on...

    It will come bundled with the PIC editor in a kind of Civ 1 "tool box", in a matter of days.

    Cheers
     
  6. Mize

    Mize Warlord

    Joined:
    Jun 17, 2011
    Messages:
    292
    Damn, makes me want to see the 'real' jungle ingame. This is great stuff.
     
  7. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
    Just for the sake of notifying: CIVED has been refurbished and is now released as bundle in the JCivED toolbox: see this thread.
     
  8. Dark_Nephi

    Dark_Nephi Chieftain

    Joined:
    Jan 25, 2013
    Messages:
    22
    Location:
    Utah
    This looks great guys. Im looking forward to map editing my civ 1 game. keep up the good work and keep everyone posted. On my other computer I was playing win civ and changed all the .wav files to other ones.. go so used to it that when I accidently changed them back I was sad. #editingcivFTW
     
  9. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
  10. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,354
    Location:
    Columbus
    Is the Micro Prose PIC format documentation available somewhere, or have I missed it? I was wondering since I saw one of Theov's posts, and thought, "Wouldn't it be cool to be able to use Civ1 graphics in Civ3 and have a retro look, or vice versa?" So I started looking into the PIC format, but found that it appears MicroProse's variant is different than the common PIC format. Which doesn't surprise me too much - their PCX for Civ3 was also not quite standard, though only the transparency differed there.

    At any rate I've been looking at a PIC file in the hex editor, and reading the Darklands documentation you linked to. I also took a look at the code on SourceForge (hope that's okay), and between the two I think I have an idea of where to start. The palette looks pretty straightforward, and similar to other image formats. The image itself is a little trickier. As I understand it, to read it, LZW decoding is needed first, followed by RLE encoding. It's late so I haven't looked too much into the details of the compression, such as the specifics of RLE encoding (I know from working with bitmaps that there are a few variants, but once the variant it known it's straightforward). I wonder how much space they saved by going with LZW + RLE versus just one or the other?

    I'm also curious if you already have code that converts PIC files into a standard Java Image object (java.awt.Image), or something easily convertible into an Image. From the poking around I've done so far, it looks like you convert them into a custom ImageData object, and then use that to create a GIF, but I'm not sure when that might be easily convertible to a java.awt.Image. Since JCivED can display PIC files, I reckon there's probably a way to do something like that... just not sure if there's a good API-like way to do so.

    At any rate, I think it would be cool to be able to do a MicroProse PIC to a Firaxis PCX conversion without an intermediary GIF stop. I can probably piece it together given the Darklands documentation and general LZW documentation, but figured I'd comment first in case there were better documentation or an easy way to do it that I was missing.
     
  11. SWY

    SWY Prince

    Joined:
    Oct 3, 2014
    Messages:
    330
    Gender:
    Male
    Location:
    The Netherlands
    Quintillus, I've used darkpanda's code and converted it to C#... it's available in the CivOne source code. You can use the source code of CivOne for any purpose you like (Creative Commons licence), feel free to have a look.

    My code translates the PIC files to a .NET Bitmap object. You should be able to go anywhere from there...
     
  12. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,354
    Location:
    Columbus
    SWY, thanks. That's helpful. Although I usually use Java, your C# code is more similar in style to what I usually write than the JCivED code, so it's easier to grasp quickly. The extra comments are also appreciated.
     
  13. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
    Like several other things, my wish to document has remained at the "wish" state so far...

    JCivED is almost only relying on java.awt.BufferedImage (a subclass of java.awt.Image), so there is definitely code in JCivED that will "convert" a PIC image file into a BufferedImage... But then, just trying to locate the code that you ask on SourceForge (my local copy has been updated a lot since the last sourceforge commit), I realized it is not very obvious...

    I apologize for this :( it is true that I never focused too much on making JCivED source code readable by others, and I keep re-organizing the code a lot.

    Anyway, here are the 2 classes/methods you are interested in:
     
  14. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,354
    Location:
    Columbus
    Ah, that toBufferedImage256 method in PicImage is the one I hadn't found! I'd located the readPIC one, but that was only half the solution. So instead, I'd tried extracting the LZW/RLE methods and combining it with my own code for reading little-endian files and extracting the palette, using the CivOne C# code as a model for how to glue it together. Unfortunately, something didn't quite work out (maybe due to using a different little-endian file reader? Or perhaps I made a mistake when gluing together the code), and I was getting different errors depending on the PIC file I tried. I never did completely understand the LZW code, either - particularly how the variable bit lengths for codes were handled - which made debugging difficult.

    I'll try it again with the toBufferedImage256 method, and I'm optimistic that will work. Then it's just a matter of adding the ability to export to PCX, as I realized I've written a reader, but my current writer only modifies the palette, not the actual bitmap. That should be fairly easy, though, since I already have a pretty thorough understanding of PCXs.
     
  15. illepokok

    illepokok Chieftain

    Joined:
    Sep 13, 2015
    Messages:
    6
    I just built a version in Javascript after going through the JCivEd source and reading all of the posts on this forum. Took a long time to wrap my head around how it works. You have to take three passes through the data (once to pull out integers of increasing size) followed by the LZW decompression followed by the run-length encoding. And of course, if any of those are wrong, your picture will look like TV fuzz. But once you see that first picture.... pure elation!

    The key files that helped me figure it out from JCivEd were:
    • LZWDecoder.java
    • LZWDictionary.java
    •ImageData.java
    • ImageDataX0.java

    Hopefully that gets you started. I'm not sure a loosely typed language like Javascript will help with C#, but happy to post that code if it will help.

    Huge props to DarkPanda for your work figuring this stuff out!
     
  16. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,354
    Location:
    Columbus
    Thanks, will keep that in mind when I get back around to it. I started a new job the day after my last post, which led to being distracted from my own coding projects, and I wound up doing other work when I got back to them. I'm actually coding in Java, and was just using the C# code as a model for integrating the Java part with my own work, since the styles of Java were fairly different but the C# coding style was similar to my Java coding style. Wouldn't be opposed to seeing the JavaScript version though. And not even so much for my own benefit, as for the benefit of anyone who may come next and find it useful. I think on the whole a lot of working could have been saved in the Civ modding community over the years by more code being available, and if someone comes along wanting to work with these formats and JavaScript is their language of choice, it'll be all the better if an implementation in their language of choice is available.

    That said, like Weird Al I'm fluent in JavaScript (though not Klingon), so even though Java is my primary language, I may be able to benefit from it as well.
     
  17. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603

    Special thanks for the props, because this truly was the starting point of the whole JCivED experience for me, 3 years ago...

    By the way: I am pretty sure the whole co/dec can be programmed in a streaming way, each layer (input PIC, lzw, variable bit length integers, rle, X0) consuming the bytes from the previous one and feeding the next one. In the end it would be just 1-pass, although the complexity remains O(N) in both cases.
     
  18. darkpanda

    darkpanda Dark Prince

    Joined:
    Oct 28, 2007
    Messages:
    603
    Also: I would be interested in the JS port if you care to share
     
  19. illepokok

    illepokok Chieftain

    Joined:
    Sep 13, 2015
    Messages:
    6
    As I have been going through decoding / rendering the maps, I had to translate the "pixel" level values into the appropriate tiles. Figured I would post here to continue to the documentation development effort :)

    Layer 0: Terrain
    0x1 - Water
    0x2 - Forrest
    0x3 - Swamp
    0x4 - Unknown (have yet to see it used)
    0x5 - Unknown (have yet to see it used)
    0x6 - Plains
    0x7 - Tundra
    0x8 - Unknown (have yet to see it used)
    0x9 - River
    0xA - Grass Lands
    0xB - Jungle
    0xC - Hills
    0xD - Mountains
    0xE - Desert
    0xF - Arctic

    Layer 4/5 - Terrain Improvements
    0x2 - Irrigation
    0x4 - Mine
    0x8 - Road/Rail
    0xA - Road/Rail + Irrigation
    0xB - City
    0xC - Road/Rail + Mine

    Each "pixel" in Layer 4 is really a bit-flag where bit 0 indicates City, bit 1 indicates Irrigation, bit 2 indicates Mine and bit 4 indicates Rail. The only confusing part is that Cities have irrigation and roads automatically, thus you will never see "0x1", only "0xB" (1 + 2 + 8).

    There doesn't seem to be any distinction between roads and rails in this bitflag
     
  20. illepokok

    illepokok Chieftain

    Joined:
    Sep 13, 2015
    Messages:
    6
    I just threw a work-in-progress version online:

    http://cived.tigermonkey.org

    If you choose a local .map file, it will render the map to a canvas element. The functionality is in there to render the VGA/EGA graphics as well, but in moving on to map rendering, it's no longer exposed in the UI.

    The code also leverages the Quintus JS Game Engine to make plotting TileSheets a bit simpler.

    Still lots to do re: terrain improvements, plotting cities, units and stuff, but hey, it's a start! :)
     

Share This Page