Civ3 Show-And-Tell

I tried rendering the map in html and had several issues. Here is today's best-looking result, although it isn't isometric. Black tiles are fogged, green tiles are unfogged, and the numbers are the index number of the tile in the flat list. I had thought that I read tile order was top-bottom, left-right and thought I coded for that, but the results look correct and clearly the tiles are left-right first and then top-bottom based on the index numbers in the table. So I managed to make two wrongs that added up to a right.

http://lib.bigmoneyjim.com/civfan/c3sat/20130421/map.html

I need to transform the table to isometric logically and then visually, and of course I need to extract and display more useful than fog of war. But progressing results are encouraging.

For the moment I'm having my program generate all the div tags, but ultimately I'll probably have an intermediate format and generate the map in the browser with javascript to keep from having to push all those div tags over the network.
 
Nice! You've got the tile-order-in-the-flat-list-to-isomorphic-ordering conversion right as it is now. Good choice of a starter goal. There does have to be something for the "explored, but not visible now" tiles in the .SAV, but that's beyond my BIQ area of expertise.
 
I have two short-term itches: show map terrain differentiation and pull maps from arbitrary saves. That's when it becomes almost generally useful.

I've been having trouble matching what's in my save with the scant TILE documentation I've found, but two things are promising to help with the search:

  • This hexdump function helps me directly compare peer TILE sections which is difficult to do in a simple hexdump of the whole file
  • I just realized once I isolate a section offset in question I can print it on the HTML map in place of the index and better see if the numbers correspond to differences on tiles.

Sample of the hexdumps showing the 36-length TILE sections one-per-line. The columns with repeating 06's and some 08's are my current focus:
Spoiler :
Code:
0000  00 00 ff ff ff ff ff ff ff ff ff ff 4e 06 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 ff ff ff 00 00 00 00   ............N.......................
0000  00 00 8d 8d ff ff ff ff ff ff ff ff 4e 06 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 8d ff ff 00 00 00 00   ............N.......................
0000  00 00 8c 8d ff ff ff ff ff ff ff ff 4e 06 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 8d ff ff 00 00 00 00   ............N.......................
0000  00 00 ff ff ff ff ff ff ff ff ff ff 24 08 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 ff ff ff 00 00 00 00   ............$.......................
0000  00 00 87 87 ff ff ff ff ff ff ff ff 32 08 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 83 ff ff 00 00 00 00   ............2.......................
0000  00 00 88 88 ff ff ff ff ff ff ff ff 40 08 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 88 ff ff 00 00 00 00   ............@.......................
0000  00 00 ff ff ff ff ff ff ff ff ff ff 4e 08 00 00 00 00 00 00 ff ff ff ff ff ff 06 00 06 ff ff ff 00 00 00 00   ............N.......................

Edit: Interesting note: I'm highlighting tiles flagged as "visible now" and noticed some discrepancies with the game. Apparently "visible now" flags only apply to unit visibility and not culture or city visibility. It took me a while to figure it out; I thought I had my map formatted incorrectly at first.

Edit 2: Oh, I wonder if the "visible now" flag is for barb spawning and not for visible fog-of-war? I would have to try to figure out if non-combat units trigger the flag, but it's just a curiosity and I won't effort on it anytime soon. Edit 3: No, that doesn't make sense because barbs can't spawn in culture line-of-sight. Maybe the superset of these tags and culture views is what to reveal with line-of-sight?
 
I'm using this post as a scratchpad for trying to figure out where the terrain info is.

There are four TILE sections for each game tile with data lengths of 36, 12, 4 and 128 which I have taken to referring to as Tile36, Tile12, Tile4 and Tile128.

Tile128 appears to conform at least in the early part to TILE "type B" described here.
  • 0x00 - 4-byte bitmap, mask 0x02 indicates player-visible tile
  • 0x04 - 4-byte bitmap, mask 0x02 indicates player-visible-"now" tile, in my experience applies only to tiles units can see

I am intuitively looking to Tile36 to have basic terrain info.
  • 0x0c - This byte seems to have nothing to do with terrain. It may be the LSB in a 4-byte integer for purposes unknown
  • 0x0d - 1-byte seems to be highly correlated to terrain. 0x08 may be ocean, 0x06 may be sea or sometimes coast. 0x00 is also sometimes associated with coast. Hmm, there is correlation, but it doesn't seem to identify the terrain type. Update: While 0x08 and 0x06 are highly correlated with ocean and sea, 0x00 occurs in places that don't seem to correlate with each other. I am not seeing any bitmask patterns or any terrain or other indicators of meaning. (I am putting the value in the HTML map and using jQuery to highlight by value to get a good visual on value-to-map.)
  • 0x0e-0x13 seem to always be zero
  • 0x14-0x19 seem to be always 0xff
  • 0x1a Continent ID. It's possible this is a 2-byte short int. looks somewhat similar to 0x0d. UPDATE: BINGO, 0x06 in this field appears to be perfectly correlated to saltwater, although this byte alone doesn't seem to differentiate saltwater types. Nevermind, this is the continent. In this map, there is one "continent" of water and six continents of land, and each tile's continent is consistent with the rest of its continent-mates. Apparently fresh water tiles are part of thier landmates' continent.
  • 0x1b seems to be zeroes
  • 0x1c All 0x06 on my test map. looks somewhat similar to 0x0d
  • 0x1d varies a lot
  • The last 6 bytes seem to always be ff ff 00 00 00 00

After finding the continent ID I peeked back at the BIC info to try to match them up. I am now thinking C3C terrain info spilled over into Tile12 and/or Tile4. That kind of makes sense.

Here is the HTML map that let me identify the continents ID location. I colored all the 0x6 values blue and saw clearly the continents for the first time!

Tile12:
  • 0x00 - Worker improvements bitmap. The question marks are from other sources and not verified by me so far.
    • 0000 0001 = Road
    • 0000 0010 = Rail (?)
    • 0000 0100 = Mine
    • 0000 1000 = Irrigation
    • 0001 0000 = Fortress (?)
    • 0010 0000 = Goody Hut (?)
    • 0100 0000 = Pollution (?)
    • 1000 0000 = Barbarian (?)
  • 0x01-0x04 - all zeroes in my map
  • 0x05 - Terrain! Interesting, mostly repeated digits between the nybbles with some exceptions. Suspect this is two flag/bitmap nybbles with similar meanings, perhaps like what I mentioned earlier with what the tile currently is versus what it looked like last time it was in line-of-sight. Come to think of it I think only worker actions (and overlays like forest, pollution, jungle, marsh) are hidden from visible fog-of-war, so maybe this is a terraform flag byte. This is clearly terrain, and the mismatched nybbles are due to overlays like forests, cities, jungles, etc.. It seems that in this coding mountains and hills are overlays, too...interesting. Now that I've found another key value I can probably use other sources as a reference for the meaning and surrounding info.
  • 0x06-0x09 - all zeroes
  • 0x0a - some nybble-nybble repetition like 0x05 but not as much; think it's a bitmask possibly including the next byte
  • 0x0b - mostly 0x00's with a few 0x10's
 
Heh, I found a compromise between isometric and visually-contiguous. I call it isometbrick.

It's certainly not the target format for this utility, but it is a dev-time compromise between easy-to-make, visually-contiguous and having the tiles in proper position with respect to each other. The biggest visual issue with this (besides being fugly) is that North/South is visually disconnected.

I kept all methods (text table, html square table, faux-isometric and now isometbrick) so I can switch betwen them if this one gives me trouble. The faux-isometric is hard to look at, though.
 
I put way too much work in on this over the weekend, and I'm not sure I have enough to show for it, but here is the weekend's pièce de résistance:

http://lib.bigmoneyjim.com/civfan/c3sat/20130421/map-with-terrain.html

The Python program created the isometbrick html layout with the terrain value in each "tile", and as a quick hack I used jQuery to recolor the tiles based on the terrain value. It is a beautiful thing to me, and hopefully it is recognizable as a Civ3 map to you.

There is much work to do, but this is a decent proof-of-concept.

Here is a screenshot of the HTML map and the in-game minimap for comparison:





Python code:
Spoiler :
Code:
#!/usr/bin/env python

# 2013-04-20 Another attempt at reading a save file; this time I'm going
#   to try extracting just the map data

import struct   # For parsing binary data

class GenericSection:
    """Base class for reading SAV sections."""
    def __init__(self, saveStream):
        buffer = saveStream.read(8)
        (self.name, self.length,) = struct.unpack_from('4si', buffer)
        self.offset = saveStream.tell()
        self.buffer = saveStream.read(self.length)

class Tile:
    """Class for each logical tile."""
    def __init__(self, saveStream):

        self.Tile36 = GenericSection(saveStream)
        self.continent = get_short(self.Tile36.buffer, 0x1a)
        del self.Tile36.buffer

        self.Tile12 = GenericSection(saveStream)
        #self.whatsthis = get_byte(self.Tile12.buffer, 0xa)
        self.whatsthis = get_byte(self.Tile12.buffer, 0x5)
        self.whatsthis2 = get_byte(self.Tile12.buffer, 0x5)
        self.whatsthis3 = get_byte(self.Tile12.buffer, 0xa)
        self.whatsthis4 = get_byte(self.Tile12.buffer, 0xb)
        self.whatsthis5 = get_int(self.Tile12.buffer, 0x0)
        #del self.Tile12.buffer

        self.Tile4 = GenericSection(saveStream)
        del self.Tile4.buffer

        self.Tile128 = GenericSection(saveStream)
        self.is_visible_to = get_int(self.Tile128.buffer, 0)
        self.is_visible_now_to = get_int(self.Tile128.buffer, 4)
        self.is_visible = self.is_visible_to & 0x02
        #self.is_visible = self.is_visible_to & 0x10
        self.is_visible_now = self.is_visible_now_to & 0x02
        del self.Tile128.buffer

class Tiles:
    """Class to read all tiles"""
    def __init__(self, saveStream, width, height):
        self.width = width      # These may eventually be redundant to a parent class
        self.height = height
        self.tile = []          # List of individual tiles
        logical_tiles = width / 2 * height
        while logical_tiles > 0:
            self.tile.append(Tile(saveStream))
            logical_tiles -= 1

    def table_out(self):
        """Return a string of a simple text table of visible tiles."""
        table_string = ''
        for y in range(self.height):
            if y % 2 == 1:
                table_string += '  '
            for x in range(self.width / 2):
                if self.tile[x + y * self.width / 2].is_visible:
                    table_string += '#'
                else:
                    table_string += '.'
                table_string += '   '
            table_string += '\n'
        return table_string

    def html_out(self):
        """Return a string of a html table of visible tiles."""
        table_string = '<div class="map">'
        for y in range(self.height):
            table_string += '<div class="maprow">'
            for x in range(self.width / 2):
                i = x + y * self.width /2
                #info = hex(self.tile[i].whatsthis)
                info = str(i)
                if 0 <= i < len(self.tile):
                    if self.tile[i].is_visible_now:
                        table_string += '<div class="tile visible visiblenow">' + info + '</div>'
                    elif self.tile[i].is_visible:
                        table_string += '<div class="tile visible">' + info + '</div>'
                    else:
                        table_string += '<div class="tile fog">' + info + '</div>'
                        pass
                else:
                    table_string += '<div class="tile notile">' + info + '</div>'
            table_string += '</div>'
            table_string += '\n'
        table_string += '</div>'
        return table_string

    def html_fake_iso(self):
        """Return a string of a html table of visible tiles. Filling with blank table entries to position isometric tiles."""
        table_string = '<div class="map">'
        for y in range(self.height):
            table_string += '<div class="maprow">'
            if y % 2 == 1:
                table_string += '<div class="tile notile"></div>'
            for x in range(self.width / 2):
                i = x  + y * self.width /2
                info = hex(self.tile[i].whatsthis)
                #info = str(i)
                if 0 <= i < len(self.tile):
                    if self.tile[i].is_visible_now:
                        table_string += '<div class="tile visible visiblenow">' + info + '</div>'
                    elif self.tile[i].is_visible:
                        table_string += '<div class="tile visible">' + info + '</div>'
                    else:
                        table_string += '<div class="tile fog">' + info + '</div>'
                        pass
                else:
                    table_string += '<div class="tile notile">' + info + '</div>'
                table_string += '<div class="tile notile"></div>'
            table_string += '</div>'
            table_string += '\n'
        table_string += '</div>'
        return table_string

    def table_out(self):
        """Return a string of a html table of visible tiles. Trying fake-isometric layout with column spanning"""
        table_string = '<table>'
        for y in range(self.height):
            table_string += '<tr>'
            if y % 2 == 1:
                table_string += '<td class="tile notile">.</td>'
            for x in range(self.width / 2):
                i = x  + y * self.width /2
                info = hex(self.tile[i].whatsthis)
                #info = str(i)
                cssclass = 'tile '
                if 0 <= i < len(self.tile):
                    if self.tile[i].is_visible:
                        cssclass += 'visible '
                        if self.tile[i].continent == 6:     # HACK! Hard-coding continent number for ocean on my test save; need to link this to CONT sections in the future
                            cssclass += 'bigblue '
                        table_string += '<td colspan="2" class="' + cssclass + '">' + info + '</td>'
                    else:
                        table_string += '<td colspan="2" class="tile fog">' + info + '</td>'
                #else:
                #    table_string += '<td class="tile notile">' + info + '</td>'
                #table_string += '<td class="tile notile"></td>'
            if y % 2 == 0:
                table_string += '<td class="tile notile">.</td>'
            table_string += '</tr>'
            table_string += '\n'
        table_string += '</table>'
        return table_string


def get_byte(buffer, offset):
    """Unpack an byte from a buffer at the given offest."""
    (the_byte,) = struct.unpack('B', buffer[offset:offset+1])
    return the_byte

def get_short(buffer, offset):
    """Unpack an int from a buffer at the given offest."""
    (the_short,) = struct.unpack('H', buffer[offset:offset+2])
    return the_short

def get_int(buffer, offset):
    """Unpack an int from a buffer at the given offest."""
    (the_int,) = struct.unpack('I', buffer[offset:offset+4])
    return the_int

def parse_save():
    saveFilePath = "unc-test.sav"
    saveFile = open(saveFilePath, 'rb')
    print 'HACK: Skipping to first TILE in my test SAV.'
    saveFile.seek(0x34a4, 0)
    print 'HACK: Instantiating the class that reads TILEs with width x height hard-coded to my test SAV.'
    game = Tiles(saveFile, 60, 60)
    return game

def hexdump(src, length=16):
    """Totally yoinked from https://gist.github.com/sbz/1080258"""
    FILTER = ''.join([(len(repr(chr(x))) == 3) and chr(x) or '.' for x in range(256)])
    lines = []
    for c in xrange(0, len(src), length):
        chars = src[c:c+length]
        hex = ' '.join(["%02x" % ord(x) for x in chars])
        printable = ''.join(["%s" % ((ord(x) <= 127 and FILTER[ord(x)]) or '.') for x in chars])
        #lines.append("%04x  %-*s  %s\n" % (c, length*3, hex, printable))
        lines.append("%04x  %-*s  %s" % (c, length*3, hex, printable))
    return ''.join(lines)

def main():
    game = parse_save()
    print 'Printing something(s) from the class to ensure I have what I intended'
    #print game.name, game.length
    #print game.tile.pop()[1].length
    #print game.width, game.height
    #print game.tile[0].Tile128.length
    #print game.tile[1000].Tile128.length
    #print game.tile[0].is_visible_to
    i = 0
    max = len(game.tile)
    while i < max:
        #print i, hex(game.tile[i].Tile36.offset), hex(game.tile[i].whatsthis), hex(game.tile[i].whatsthis2), hex(game.tile[i].whatsthis3), hex(game.tile[i].whatsthis4), hex(game.tile[i].whatsthis5)
        #print i, hex(game.tile[i].Tile36.offset), hex(game.tile[i].whatsthis2)
        #print hex(game.tile[i].whatsthis)
        print hexdump(game.tile[i].Tile12.buffer)
        i += 1
    #print game.html_out()

if __name__=="__main__":
    main()

The second Python source I use to call the first block (named tileonly.py) to generate the html file:
Spoiler :
Code:
#!/usr/bin/env python

import tileonly
import datetime


def main():
    outputhtmlpath = '/usr/share/nginx/www/map.html'
    htmlhead = 'html/head'
    htmltail = 'html/tail'

    game = tileonly.parse_save()

    write = open(outputhtmlpath, 'w')
    head = open(htmlhead, 'r')
    tail = open(htmltail, 'r')

    write.write(head.read())
    write.write(str(datetime.datetime.now()))
    #write.write(game.html_out())
    #write.write(game.html_fake_iso())
    write.write(game.table_out())
    write.write(tail.read())

main()

The HTML and jQuery code can be snarfed from the linked html map.
 
Lurking on this thread and have nothing to add,

but congratulations on getting that far. It's cool to see the results of your work.
 
Thanks! It's encouraging to know there is interest.

After being away from it for the morning some things occurred to me:

  • I should read the tiles into a 2-D array instead of a flat list. That should have been obvious, but I went directly from decoding the file to figuring out how to display and skipped over the question of how it should be stored. The 2-D array will simplify creating any type of map.
  • I thought my goal was to skew and rotate the HTML table until each cell was the familiar isometric tile, but I now realize that messes up the coordinates. The checkerboard-style map I called faux or fake isometric is actually logically correct as far as x+2 is one move to the East and x+1,y+1 is one move SE. My expectation of the map coordinates of objects is that they will match this layout. It is very hard to look at as-is, but I am now thinking I should keep this structure and try to arrange isometric-shaped cell backgrounds (preferably in CSS) that are large enough to close the gaps.
  • I briefly thought I may want to store the 2D tile array with pad elements to keep the proper coordinates...in fact I am now thinking that again. Perhaps I can have a class that represents it in either fashion. The reason not to do it is to minimize the data transferred to the browser, and then javascript can fill in the spacing because the spacing is known. But I now realize I may have a need to refer to tiles by map coordinates in the Python code, and storing it with padding would simplify that. Or perhaps I could have a class method to handle any translation needed.
  • The program is starting to become more complex. I started having trouble finding functions immediately, and I started a separate file dedicated to regenerating the HTML map. I also probably need to add debug and spoiler flags to globally turn each on or off as needed. I was going to attempt to code with never showing spoiler info, but once I figured out I could used the HTML map to analyse what save file data meant it was useful to show fogged tile values to help determine correlation.
  • Generating maps is a lot more fun and rewarding than decoding binary data. Refocusing in that direction made Sunday much more fun. Maybe. I probably would have done and done something else if it weren't for that.

It seemed to be a choice between working on the map display code and figuring out more of the file contents, but now I think I can work on both simultaneously.

Well, after some refactoring, perhaps. I do want to add flags/switches for debug and spoiler info and start thinking about separating modules and request code.

I'm pretty sure one of my next steps will be to write a method to spit out tile info as JSON and then use jQuery or similar to turn the JSON into the map. And once I do that I can pass more data in JSON and selectively display it with javascript, so instead of regenerating a map for each field I want to display I could load several into the JSON data and swap it out in the browser as desired to compare known data to unknown data.

I'm naturally drawn to wanting to make the map look better, so I'll probably spend time on that first. But I would also like for the parser to start working on any C3C save files and perhaps even start looking at the differences between reading C3C, PTW and vanilla. Now might be a good time for that to help guide the structure of the data and code, but it doesn't seem fun, does it?

Edit: Some additional random things falling out of my head: While it is cool to have the terrain colored, I also liked looking at the world as simply land and water. I think that will be one of the final viewing options.

Edit 2: Idea! I had been wondering if it would be possible--given this is "show and tell"--to annotate the map. The idea is to allow annotation (notes, proposed city sites) by providing a JSON data file to attach to the forum post and a link to use it. The file would tell the browser where to get the save game data and provide the notes created by the user. In other words the browser will pull from more than one data source. Alternately I could store data server-side, but I'm not sure I want to get into that.
 
Nice work with the isometbrick display. I agree that given the current limitations in the checkboard-style map, the isometbrick is a lot easier to look at.

Interestingly, I've actually stuck with a 1-D list of tiles all this time. But I did add in a method to convert an (X, Y) coordinate into the 1-D index. That method is below:

Code:
    //This lives in IO.java (which represents the BIQ as a whole)
    //I refer to worldMap.get(0) because a BIQ can have more than one world, although Civ3 can't make use of any beyond the first one
    public int calculateTileIndex(int xPos, int yPos)
    {
        //TODO: Look at the world wrap settings, don't assume X wrapping
        //Through version 0.82, we simply said if it's off the map, we can't do anything
        //However, we should wrap east/west, since the world is not flat (most of the time, anyway)
        //if (xPos >= worldMap.get(0).width || yPos >= worldMap.get(0).height || xPos < 0 || yPos < 0)
        if (yPos >= worldMap.get(0).height || yPos < 0)
            return -1;
        if (xPos >= worldMap.get(0).width)  //wrap it back to the other side of the world
            xPos = xPos - worldMap.get(0).width;
        if (xPos < 0)   //went past the date line going west
            xPos = xPos + worldMap.get(0).width;
        int index = 0;
        //always add in a width-worth * yPos/2 (truncated)
        index+=((yPos/2)*(worldMap.get(0).width));
        if (yPos % 2 == 1)  //add in half a width worth of tiles
            index+=((worldMap.get(0).width)/2);
        index+=xPos/2;
        return index;
    }

The reason it accepts values outside the boundaries of the width is so that if you have a method that requests the tile to the east of the easternmost tile, it will still give you the tile that you would get if you went east in-game.

Whether it's better to go 1-D or 2-D, I don't know. The proper coordinates with pad elements vs. not-really-proper without question is a good one. If you don't have pad elements you might need a translate-to-coordinates method like I'm using anyway, to help avoid errors when referring to tiles and keep sanity if nothing else. How much overhead would it be to have an essentially null pad element?

Complexity will be a problem. Including comments, whitespace, and boilerplate code, wc reports that I have a bit over 63,000 lines of code in my editor. You might not get that many, and it looks like Python might be more condensed than Java, but there will be a lot of code once it's all said and done.

I think both Steph and I ran into issues with this in our editors. I'd initially coded the entire front-end all in one file, and have an old copy from December 2009 (before it was available on CFC) where that file has 14,618 lines. Responsiveness from a development environment standpoint was becoming quite slow, and eventually I ran into a limitation where a method was too big for Java to compile it. So at that point I had to refactor the code, and made each tab a separate file, with some new code added to tie them together. I think the responsiveness in the IDE is probably essentially what Steph ran into that caused him to make the tabs separate windows in version 0.80 of his editor.

I'd recommend waiting on differentiating between C3C, PTW, and Vanilla. That will add a fair amount of decoding binary data work. It's not a bad idea to add PTW in particular if GOTM is a possible use, but I wouldn't put it on the front burner.
 
Thanks! It's encouraging to know there is interest.
The technical details are way beyond me. So I'm following the thread. Glad to see the ongoing conversation between you & Quintillus. seems like there's some synergy happening. The image of the iso-brick map grabbed me. Keep up the good work.
:coffee:
 
Thanks all!

In the past 22 hours I haven't coded anything new, but I created a GitHub repo to publish the code and gain some version control so I can better keep track of revisions which will be extra important in the next steps of refactoring/reorganizing what I have so far. I am releasing the code under the GPL.

Whether it's better to go 1-D or 2-D, I don't know. The proper coordinates with pad elements vs. not-really-proper without question is a good one. If you don't have pad elements you might need a translate-to-coordinates method like I'm using anyway, to help avoid errors when referring to tiles and keep sanity if nothing else. How much overhead would it be to have an essentially null pad element?

I don't think there is any significant overhead, and I also believe that Python lists are pointers to objects, so I think I can have flat-list, 2D and fake-iso-padded-2D representations of the data without duplicating the actual tile objects in memory and offer all three paradigms simultaneously with negligible overhead. On the other hand I'm not sure that code outside of the class has any business helping itself to the tile matrix, so maybe I will just have method interfaces to handle any map tile queries and thusly could use whichever format(s) are useful for the class methods.

Complexity will be a problem. Including comments, whitespace, and boilerplate code, wc reports that I have a bit over 63,000 lines of code in my editor. You might not get that many, and it looks like Python might be more condensed than Java, but there will be a lot of code once it's all said and done.

Well, your code not only displays the BIQ info but allows comprehensive editing and saving back in native-readable format. I have the advantage so far of cherry picking the data I might display and then handing it off to a web browser to handle the display and ultimately any user interaction.

Still, if I implement some of my more complex ideas the code could still bloat a lot. Or could it? I haven't thought about the more complex abstract ideas lately; if I find myself calculating corruption, am I doing that in the browser or in the parser? A question for the future. Well, javascript is still code, and it would bloat.

I'd recommend waiting on differentiating between C3C, PTW, and Vanilla. That will add a fair amount of decoding binary data work. It's not a bad idea to add PTW in particular if GOTM is a possible use, but I wouldn't put it on the front burner.

I would agree, but my current approach of having the class load itself might need some polymorphism which might be easier to implement if I start looking at the differences now. In other words the differences in save files may cause me to change some of the fundamental structure of the data storage. But as usual, it's more rewarding to do work that produces visible results, and decoding the file mostly isn't a visible accomplishment.

Random thought: Looking at the isometbrick map it occurs to me I should be able to provide instructions on how to build the map out of Lego or similar blocks. That would be amusing.

Something in my head I haven't said yet: I plan to make it so that terrain modders could alter the look of the HTML map by editing style sheets so the HTML map resembles the modified terrain. But it's too early to determine exactly how that would work in practice.
 
Idea! I had been wondering if it would be possible--given this is "show and tell"--to annotate the map. The idea is to allow annotation (notes, proposed city sites) by providing a JSON data file to attach to the forum post and a link to use it. The file would tell the browser where to get the save game data and provide the notes created by the user. In other words the browser will pull from more than one data source. Alternately I could store data server-side, but I'm not sure I want to get into that.

Upon further thought, this won't work. The point of the app from the end-user's perspective is easy point-and-click peeking into a game's status. There are too many things that could go wrong with having the user place his own data file and get the links lined up, even with app help. Any annotation will have to be stored server-side or somehow squeezed into the url, but I don't think there's enough room in the URL to make that feasible.

I don't wan't to implement a user database, but perhaps (if I pursue this feature) I could authenticate OpenID or Twitter or Facebook APIs to allow annotation creation within user scopes, or maybe it would be better for annotation to be anonymously creatable?

If you're not quite following along in my head, any annotation would be a separate store of data, probably in a JSON file, that would be stored in the web app server, and I'm hesitant to allow such data creation on the server without some sort of user monitoring/auditing.

Edit: Another thing just popped into my head: I had been thinking of this as a Civ-3 specific thing, but once I have the intermediate JSON data and the browser reading from that, what I really have is a (potentially) general-purpose map browser. If I can keep the javascript modular I could pave the way for the browser portion to create different types of maps based upon a variety of JSON matrices. The parser which is the hard part is game-specific, but the browser doesn't have to be at all. I'll try to keep this in mind and separate generalized map-generating javascript code in one folder and civ3-specific code in another. CSS files can also be divided as such.
 
Edit: Another thing just popped into my head: I had been thinking of this as a Civ-3 specific thing, but once I have the intermediate JSON data and the browser reading from that, what I really have is a (potentially) general-purpose map browser. If I can keep the javascript modular I could pave the way for the browser portion to create different types of maps based upon a variety of JSON matrices. The parser which is the hard part is game-specific, but the browser doesn't have to be at all. I'll try to keep this in mind and separate generalized map-generating javascript code in one folder and civ3-specific code in another. CSS files can also be divided as such.
Have you checked in the other Civ forums for such a thing (the browser portion)? I'd hate to see you invest a lot of work into that if it has already been done.
 
I seem to have some time to hack on this tonight. Tiles-to-JSON are on the list, as is toying around with how to display the HTML map as isometric tiles. Here are some ideas I've come up with:

  • Some sort of rotated and skewed <div>. I would have thought this might be easy, but I'm not seeing any examples of it so far.
  • Oversized unicode character ( &#9830; ) as tile. Anticipated issues: getting it the right size and aspect ratio, difficulty in placing other items on top of it. Just realized: duh, turn it 90 degrees and I'm probably nearly at the correct angles.
  • SVG shapes per tile
  • Canvas items per tile

Have you checked in the other Civ forums for such a thing (the browser portion)? I'd hate to see you invest a lot of work into that if it has already been done.

I haven't gone looking for it. Has anyone heard of such a thing? A non-native tiled-map browser?
 
Version control is definitely a good idea! I've been using a local Mercurial repo, and it's come in handy more than once.

If you do end up directly using the code I e-mailed for BIQ stuff, ask me at that point about licenses. I'm OK with that code being open source, but am not a fan of the "spider plant" nature of the GPL (as Stallman refers to it; Craig Mundie prefers "viral"). I'm more of a fan of Apache/BSD/MIT (though I haven't chosen anything in particular for any of my projects - but I have ruled out GPL). Of course if you're just using it as a reference for writing BIQ support in Python/JavaScript/etc., that's great no matter what.
 
Well, the JSON part was harder than I expected. I wound up storing the data (continent ID and terrain ID so far) as hashes and then creating a less complex 2-d array of "dictionary" hashes to facilitate the JSON conversion.

I found making the table from the JSON data in javascript/jQuery difficult but finally got it done.

JSON is somewhat more compact than a prebuilt table but bigger than I thought. In most cases it will be compressed when downloading, so I'll compare compressed sizes as both JSON and HTML are very repetitious. I'll have to reconsider where I want to code the table because I've found Python to be more or less intuitive but javascript to be difficult for procedural programming even though jQuery has been awesome for altering CSS. However I think with some of the things I want to do JSON will make more sense for storing data that is displayed on demand and not automatically (like unit stacks when highlighting a tile, perhaps).

I didn't have much time to play with the map layout, but the unicode diamond idea doesn't seem promising.

Reportedly adding an object to multiple lists does not duplicate it, so now when I read the tiles in I put them in three different formats: a flat list, a 2d list and a padded 2d fake-isometric list. I may not need all three, but during development I'll leave it like that.

Version control is definitely a good idea! I've been using a local Mercurial repo, and it's come in handy more than once.

At first I was just copying the file to the "old" folder every now and then, but especially with multiple file updates the versioning system will be handy. Putting it on a public repo will at least offer some lasting value if I flake out and quit updating it.

If you do end up directly using the code I e-mailed for BIQ stuff, ask me at that point about licenses. I'm OK with that code being open source, but am not a fan of the "spider plant" nature of the GPL (as Stallman refers to it; Craig Mundie prefers "viral"). I'm more of a fan of Apache/BSD/MIT (though I haven't chosen anything in particular for any of my projects - but I have ruled out GPL). Of course if you're just using it as a reference for writing BIQ support in Python/JavaScript/etc., that's great no matter what.

Sure. I appreciate the code, but so far my stumbles have been trying to read the non-BIQ part of the save, although I need to look again and see if you interpret the map data; I know you copy it, but at first glance I didn't see that you parse it for meaning.

I gave a little thought to the license and figured since my environment Python and Linux are GPL and Linux has been good to me for...wow almost 20 years now...even though I mostly use Windows for desktop computing Linux has always been there for free stable servers and easy tinkering/learning. So it's my small way of giving back.

I opted against the Affero GPL, and while I'm not altering the licensing with this statement it is my understanding that if an organization wanted to use my "vanilla" GPL'ed code as an app server they would not be compelled to share any code changes so long as they didn't redistribute the changes. So if for example CivFan, Steam or whoever owns Civ3 right now wanted to set up an app server with my code and make private changes they might be able to do that.
 
I don't wan't to implement a user database, but perhaps (if I pursue this feature) I could authenticate OpenID or Twitter or Facebook APIs to allow annotation creation within user scopes, or maybe it would be better for annotation to be anonymously creatable?
Not sure I completely follow this. Would use of your utility require internet connection & access to twitter/facebook/etc ? Might make sense for Steam-dependent games. But not for C3.
 
I interpret the BIQ part of the map data, the most important/complicated of which is TILEs. Within tiles, the parts that are trickiest are the tile graphics (which it looks like you won't have to worry about), and probably rivers. I might still not have the "zero tile" or "corner of 4 tiles" rivers right; I remember looking into it but can't remember if I updated the code to make it work. There are a few binary fields that can be a bit tricky as well.

Almost 20 years of using Python and Linux... that's a long time! They aren't much older than that. I think the principles behind it are sound; my main concerns with it are economic, particularly within a capitalist country. But that's a topic for another thread.

Like Blue Monkey, the idea of integration with Twitter/Facebook, and to a much lesser degree, OpenID, sounds unappealing to me. I always choose non-Facebook sign-up options, for basically the same reason I never use services that require an e-mail sign-in (such as LinkedIn's options to find missing contacts). And there's still people out there with neither Facebook nor Twitter, including IIRC some at CFC. OpenID might not be bad; I've never used it since there's almost always a local-to-the-site option as well.

On the other hand, if you could get this hosted at CFC and have it integrated with CFC sign-ins, that might be pretty nifty. But considering that even GOTM has separate sign-ins, I'm not holding my breath.
 
Top Bottom