Civ3 Show-And-Tell

Puppeteer

Emperor
Joined
Oct 4, 2003
Messages
1,687
Location
Silverdale, WA, USA
Civ3 Show-And-Tell (C3SAT) is a web map viewer for your Conquests games. It reads save game files and produces web-browser-readable game maps. One of my hopes was to make it easier to discuss games online. You are now able to view and share a lot of non-spoiler game map info without having immediate access to a game PC.

In today's form it successfully converts save files into maps like this:


The link goes to a pan&zoom-capable map viewer, and a representative screenshot is shown here.

Only the parts of the map known to player 1 are shown. Terrain and cities are shown. Strategic resources are not shown--to avoid spoiler knowledge--but bonus and luxury resources are displayed, some with icons and some with text. (This is a work in progress.)

Web app: http://civ3.bigmoneyjim.com/

Windows app:
Spoiler :
The windows app is big and command-line and only can process uncompressed game files such as the autosaves, or you can decompress them with another utility first.

Download this huge honkin' zip file (5 MB, about 18 MB when you unzip it), then run makeSvgMap.exe <filename> or pipe it an uncompressed save.

More info here: http://forums.civfanatics.com/showpost.php?p=13259239&postcount=107

GitHub project: https://github.com/myjimnelson/c3sat
 
The original first post from 2013-04-16:
-----
Here is another cool idea that I'll tease and never follow through on. You have been warned.

Basically I keep having ideas that would require reading the save game files, so that's where I'll start. Many of these ideas I probably don't remember.

But some ideas in my mind today could be implemented a non-spoiling browse-able territory map.

Some of these ideas include:
  • (vague concept here) a web page you could point at a save file and browse territory for forum show-and-tell or SG illustration, sort of a super-screenshot that's more-or-less automatic
  • a 3D map of production factors
  • a 3D map of corruption/waste (either by corruption distance or by individual city)
  • some sort of jQuery map to locate items on the map, probably military by type and experience
  • Once I can parse the save file it will be too tempting/easy to start aggregating some of the data like CAII and MapStat do. It is not my goal to duplicate these, but wouldn't it be cool if you could use your tablet or smart phone as a status display for your Civ3 game?

Issue for the future if I actually get any of this started: I would probably need to use the installed graphics or find freely redistributable graphics for tiles and any other graphics I would need to display.

Another future issue: identify multiplayer games and have a mechanism for showing only the appropriate player's unfogged territory.

-----

First goal: parse uncompressed save-game files. At first my parser will just print the 4-char section name and report the bytes of data read.

Early challenges: While most sections have 4-char headers and 4-byte length, some have a count of subrecords instead of byte length, so there will be some conditional logic in this very basic parser.

Language of first attempt: JavaScript in Node.js. Why? Eh, I stumbled across it recently and it's my current tinker toy. Also there may be some unicorn magic letting me use similar code server-side or local if I try to make this a web-based thing. Or not. Edit: After solidifying some of my goals and ideas I realized I won't be parsing the save inside the browser, and I already knew JS isn't as smooth with binary data, so I am starting with Python instead.

Resources:
http://forums.civfanatics.com/archive/index.php/t-48270.html - Civ III save file format thread here on CivFan
http://apolyton.net/showthread.php/48062-Civilization-III-BIC-file-format-(2nd-thread)?s= - BIC file format (similar to save file)
https://github.com/codeboost/binaryparser - My starting Node.js module for parsing binary data; the simple.js example is the starting point I was looking for
dynamite - Since I am starting in Linux, I am using the dynamite program to decompress compressed save files before feeding them to my script ; A Win-based tool with scary dire warnings is offered here, but I figure if I get that far I'll decompress in the program/script
 
In my previous and current peeks at parsing the save file format it seems that those who have successfully done it guard their source code in the name of preventing cheating.

If anyone knows where some open-source save-game-parsing code is and feels like pointing me towards it, feel free! Otherwise it looks somewhat straightforward and similar to what I did in college 20 or so years ago. :eek:
 
So... you started this thread just to talk about an idea you had ? :dubious: Got no Utility Program to upload ? Then you're in the wrong Forum, you should ask a Mod to move this to the main C&C forum. Then we could discuss this brilliant idea :)
 
Yeah, it's a dev chatter / idea thread. If it needs to be moved, I am fine with that.

Actually until I get something that does something at all I expect to be talking to myself here for my own dev documentation. However if someone says "that's been done and you can find code here" then great!

Moderator Action: Moved to appropriate forum.
 
You can pm ainwood. I'm pretty sure he has tons of civ3 save data code laying around from the Civ3 GOTM.
 
Some cool ideas. Sounds ambitious. I have some questions/ideas about those, but first some resource information. I'm not aware of anything specifically in JavaScript, but I'm including language-specific stuff since at least the code logic might prove helpful.

For general BIQ format (though not SAV) information, the newer Apolyton thread is the one you probably want - it includes BIX/BIQ as well as BIC. There are a couple errors in it, such as the flag for allow cities and allow colonies being swapped.

Since Apolyton upgraded to vBulletin 4, the links to the sections in the thread I just linked don't work anymore. But I saved the thread while Apolyton was still on vBulletin 2, so I have a copy where the links work. I've attached it here. After unzipping it, you can open the .mht files in either Internet Explorer 6 or later, or any recent version of Opera (Firefox/Chrome don't support them), and they open like a normal page, complete with images.

If you're interested in it, I could send you the code I use (in Java) to process .BIQ files. I don't currently have any code for .SAV files - I can pluck the .biq rules out of a .sav, but haven't done much of anything for the .sav part itself. It's basically the Apolyton .BIQ thread implemented, with a few fixes and some comments for areas where the Apolyton thread was ambiguous. It is still changing, although fairly slowly.

Getting all the way to .sav files, even if you started at the .biq stage, would require quite a bit of non-exciting coding. The exception is if you could find someone with a working program's code for opening them and could use that language. Unfortunately, I don't have that.

You can pm ainwood. I'm pretty sure he has tons of civ3 save data code laying around from the Civ3 GOTM.

I'd second this recommendation.

-----

For the 3D maps, would they be kind of like overlay maps on an isometric grid? Kind of like what Sim City (previous as well as current ones) have? Or a completely new map view?

The option to find units on the map such as military unit reminds me of some Civ4 mods, where they have the ability to sort the military advisor to, for example, find units by level of experience, type of unit (horse, artillery, etc.), and so forth. I can't remember if the mod I saw that in was Realism Invictus or Caveman2Cosmos. Maybe both?

-----

Edit: For BIQ/SAV decompression, if you want, you can use the command-line utility I've attached below. It uses chiefpaco's decompression code from MapStat; all I've added is a front end so that you can use it at the command line and thus from any other program you want. The actual decompression is unchanged from what he posted in 2002, but I've never seen it fail. Which is good, as I don't know what it does behind the scenes.

The Windows version you linked to is, AFAIK, what Steph is using in his editor. In my experience it isn't quite as reliable.

----

Edit2: Good to see Chieftess in the Civ3 forums again! :wavey:
 
Thanks for the pointers.

I wouldn't characterize it as ambitious...more like pie-in-the-sky dreams with questionable ability and motivation to try to start implementing them. (Okay, I guess that could fit "ambitious".)

Basically I had a bunch of ideas in my head with vague ideas on how they might be realized, and got enough motivation to try to start organizing a blueprint on how to proceed.

Javascript is possibly a dumb idea, but what recently piqued my interest in programing-at-all lately was trying out a couple of Node.js modules to solve a problem I had. I can't immediately find the article I read, but LinkedIn moved to Node.js and can process server-side or client-side as resources and browser abilities dictate; if I could have that flexibility in a Civ3 save-game parser it would allow moving the processing around with the same code base depending on whether we want a plain HTML5 browser to not need any plugins or helper programs peek into a game file or have a local JS engine parse and push data from the gaming PC to a smartphone without using a public server. But that is the unicorn magic, and I don't know if I can pull that off.

On the other hand, with a couple of quick searches I found a promising binary parser module and isometric CSS libraries in javascript. The save-game-to-dhtml paradigm may be as conceptually simple as using prebuilt libraries to transliterate the pertinent save game data into HTML or JSON and letting style sheets and the browser handle the rendering.

I about wet myself when I found this video (not something I did, but damn if it doesn't look better than I imagined it):


Link to video.

Data graphing will probably be trial-and-error. I had some vision in my head of a flat grid with rotatable axes, and each city position would have a bar graph above it, but before I even find out if that's feasible I'm not sure it makes any sense to present data that way. It may make more sense as you say to highlight grids on an isometric map and/or have bar graphs to indicate factors of production. Check out these animations, especially the second one; perhaps I could have bar graphs that can be either above each city or fly into a front-and-center line bar graph as desired for clarity.

Random idea in the middle of typing: track individual unit(s) based on their names; if a unit is named in a certain manner (*trackme, track*), track its movements.

One of the many problems I have is that I'm trying to mush separate ideas into one idea, and it's not going to work that way. The rest of this post is me dividing my ideas up into implementable parts:

The main inspiration for starting this thread was the idea of the casual CivFan forum reader being able to peek into a save game without downloading it and firing up Civ3, and be able to do so even if the poster (SG participant, help seeker or other save-posting person) didn't take the trouble of making screenshots. To me the interface has to be HTML-in-browser.

(Posting interrupted, will edit later to finish thoughts)

And ideally there shouldn't be a download or plug-in. With modern JS engines in browsers and/or server-side code I think this is doable. If it were a browser plugin one might click on a save-game file and get an HTML isometric representation of the game state (map, city productivity, etc). But to avoid using a plugin (extra end-user effort) I would have to have server-side code, but since the save game file in most cases would be a URL, the results could be cached so if 200 people peek at the save file it will only need to be parsed once per app server.

So for the stock-browser-peek-at-save-file (or "Civ 3 Show And Tell") there would need to be an app server to fetch the save, parse it, cache the results (be they HTML or an intermediate data sink) and send to the client. Either the end user would copy the save game URL to the app server's interface page or a helpful poster would include a url like http://app.example.tld:9001/civ3vie...example.tld/forum/attachment/blahblahblah.sav . Come to think of it I should have a link text generator for that.

To summarize, C3SAT:
  • Modern stock browser as client
  • Intermediate app server to fetch sav files by URL, parse to HTML or intermediate format (JSON?), cache by sav URL and serve to client
  • Link generator for helpful forum posters
  • Basic needed non-spoiler info to display: Map, resources, improvements, cities, units, borders
  • Desired info: More detailed aggregate and specific info on factors of production, city production, unit data, corruption and culture
  • Pie-in-the-sky: Anything that can help show or teach someone how to change e.g. citizens or other settings to be more efficient or effective. (Probably lots of coding/calculating for anything like this)

Another precipitate of the bigger idea is a game content enhancer or analyzer in which the game state (based on the most recent autosave or save) to display on the same PC or on a secondary device such as a smartphone, tablet or any other browser device. I actually see this multiple problems:
  • Monitor the save/autosaves folder for new files
  • Parse save file
  • Update display based on save file data

Interrupting my train of thought I just realized that parsing in the browser is not going to happen; it's either going to happen on an app server (for C3SAT) or in a companion executable on the gaming pc (for enhancer/analyzer), so that more or less eliminates my reason for considering parsing in javascript as I can't sensibly offload parsing to the client browser. Python jumps to mind as a good candidate to parse on an app server and on the client and to share code between the two. Perl, Java or C would do, too. Possibly even a .NET app, but I was kind of hoping for any companion enhancer/analyzer to be usable on Mac, too.

But the javascript libraries can still be useful for the isometric layout and manipulating what is displayed in the browser interface.

Okay, continuing with the idea of the enhancer/analyzer game-side companion, mechanically it will perform similar work as C3SAT--parse the save and spit out HTML or JSON to a browser--except it will monitor the latest save file instead of from a URL.

However, the info the user would want is different. Map, units and city location are redundant to the game display, so aggregated game info, reporting and alerts would be the main interest. Plus, and this is from an idea I've had before, this should be able to display custom content for mod makers. For example a YouTube video might be displayed if a wonder is built.

At one point I had been envisioning the game info and mod content as separate things, but no, there is a companion display that displays game info *and* launches any custom mod content by Internet URL or as served locally from mod folders. I have imagined that a CSV or Excel file would list events and associated content to launch, but I wonder if it could somehow be rolled into the BI* file and read via the save?

Summary of enhancer/analyzer:
  • Companion executable that monitors for new save files
  • Displays to browser, on same PC or to secondary device
  • Could display custom content in the browser for mods (video upon wonder build or tech, etc)
  • Otherwise game alerts and game info graphs and overlays
  • Pie-in-the-sky: store history of game; historical productivity, unit tracking, ???

-----

Summary of this post's babbling:
  • Most of my ideas can be implemented in two different offerings: a server-side sav-to-html parser/cache-er and a game-side enhancer/analyzer program
  • They key on different events, but each has sav-file-in and HTML-out
  • So basically I should be able to reuse code to implement both
  • Using javascript to parse is a non-starter given where the sav parsing is happening but will be invaluable in the client browser
 
Interesting. I had no idea that someone already had an HTML5/CSS/JS isometric engine. And that does look pretty good.

I agree that an HTML-in-browser interface makes sense for this project. There could still be back-end processing, but it sounds like it would defeat the purpose if the user had to download the file - then they could just fire up Civ3 or CAII, etc.

I can't help with the development aspect of it, though. I've worked with JS a bit, but have never worked with Node.js. Similarly, I've worked with HTML, but that's HTML 4. And I don't particularly fancy picking up more web-based stuff right now... I've done that enough recently as an employee to be happy to stick with desktop stuff in my spare time.
 
I kept adding to my previous post after you posted. I talked myself out of javascript for parsing the save game (and thus out of Node.js), but of course I'll use the heck out of the isometric javascript modules for display.

And I agree there has to be a server to pull and parse the save file, but that makes caching a no-brainer, so parse-once-display-many. Huh, come to think of it there could be some data mining potential there, although I'm not sure it would be particularly useful data.

I might yet check out your Java code (which could be used server-side and client-side), but first I think I'll take a peek at what parsing with Python would be like.
 
That's quite an addition to the post! A couple of initial thoughts:

- I'm a bit unclear on the premise of the game analyzer/content enchaner, in particular with regards to why it's necessary when CAII already exists, and could possibly be enhanced. I see that this would allow viewing this sort of information on a second device, which could be helpful, but it seems like a lot of effort to go to for that. (I also realize that CAII's update frequency is very low recently)
- It does seem like you'd probably need a game-side program for auto-save monitoring, and sending them to the server. Perhaps it is possible in some way via the browser, but I'd be surprised if you could monitor for files, particularly without any plugins.

The idea sounds cool. It would certainly be a new feature for Civ as a whole (including newer versions).

Re: storing events in the .biq file. This may be possible. Currently, there are a few areas of the .biq file that are either unused (there's an area of the BLDG section that is unused, for example, and an "Unknown" text string in GAME). You could also tack something on after, for example, the Scenario Search Folders. All the strings are fixed-length in Civ3, but Civ3 ignores everything after the first null character in a string. So you could potentially store something after the null character in strings that are usually shorter than their fixed length. The Scenario Search Folders (in GAME) are 5200 characters, and I'm fairly certain that most are well short of that length, so it would be my go-to candidate if you consider doing that.

As far as I know no one's doing something like that with any utilty, though perhaps someone has and I don't know about it. I've thought that if someone ever makes a widely-functioning save game editor, a trick like that could be used to hide something in the file so that HOF games could detect if someone used the save game editor (assuming the editor would get rid of the the trick in the save file).
 
- I'm a bit unclear on the premise of the game analyzer/content enchaner, in particular with regards to why it's necessary when CAII already exists, and could possibly be enhanced. I see that this would allow viewing this sort of information on a second device, which could be helpful, but it seems like a lot of effort to go to for that. (I also realize that CAII's update frequency is very low recently)

This early in the thread I would describe the process as distilling, organizing and critiquing ideas that have popped into my head while playing. What's possible, what's practical, what might I be capable of implementing, etc.. Some of the ideas overlap, and some run off into the wilderness.

The enhancer/analyzer is something that's been in my head for a while, and I even started a thread about it before. It's not related in end-user utility to the C3SAT idea, but the code and implementation would share a lot. The core idea is to add external content for in-game events for mods or tutorials.

I keep telling myself I don't want to reinvent CAII, but whenever I think of parsing the game file it seems like it might be one more simple step to overlap CAII functionality, and CAII seems to be troublesome for many to install, and as far as I know the source code isn't available. Also, does CAII run on Macs?

On the other hand, I don't know enough yet about the save file contents to know how hard some things might be. For example, will reporting city factors of production be as simple as pulling a short integer for shields produced from the city section, or would I have to iterate each citizen of the city, the tile and improvements they work and the game rules for what each tile produces and then do corruption & waste calculations including figuring out whether or not the city is on the trade network? (Rhetorical question for now...I have other things to think on first.)

-----

I started trying to parse a decompressed save file with Python and have had the very most basic amount of success, printing out the section header and the length.

However, I am being tripped up earlier than I thought by inconsistencies. I am expecting most sections to be four characters followed by a 4-byte length, and depending on the section type the length may be in bytes or sub-records.

But the very first section in the file, CIV3, does not seem to follow this structure, and neither does the third section BICQ.

Code:
$ ./civ3parse.py
Hello, World
This is the message
Skipping first 30 byts because the CIV3 header doesn't seem to be followed by a length
Section Name: BIC  , Length:  524
Section Name: BICQ , Length:  592594262

Spoiler :
Code:
#!/usr/bin/env python

# 2013-04-19 Start of attempt to parse a decompressed Civ3 save file
# unc-test.sav

import struct           # For parsing runs of binary data

print "Hello, World"

message = "This is the message"

print message

saveFilePath = "unc-test.sav"

saveFile = open(saveFilePath, 'rb')

print "Skipping first 30 bytes because the CIV3 header doesn't seem to be followed by a length"
saveFile.seek(30,0)

# way to loop until EOF adapted from http://stackoverflow.com/questions/1752107/how-to-loop-until-eof-in-python
for buffer in iter(lambda: saveFile.read(8), ''):
        (section,) = struct.unpack('4s',buffer[0:4])
        (sectionLength,) = struct.unpack('i',buffer[4:8])
        print "Section Name:", section, ", Length: ", sectionLength
        saveFile.seek(sectionLength,1)

EDIT: Oh, I just looked at the file with a hex editor and see that BICQ is followed by ASCII "VER#" which I see in the BIQ format discussion. Hopefully the other headers will conform to the section name & length format.

Code:
0000230: 0000 4249 4351 5645 5223 0100 0000 d002  ..BICQVER#......
0000240: 0000 cdcd cdcd 0000 0000 0c00 0000 0800  ................
 
I started dividing the parser into functions, got a generic sub-record parser and handled the special cases of CIV3 and BICQ, but there are apparently at least two GAME sections, and the second one behaves differently than the first. I need to go do other things, but here is what I have now:

Output:
Spoiler :
Code:
$ ./civ3parse.py
Skipping 26 bytes because the CIV3 header doesn't seem to be followed by a length
sectionGeneric says: Section Name: BIC  , Length:  524
sectionBICQ says: This should say VER#: VER#
sectionSubRecord says: Section Name: BICQVER# Number of subrecords: 1
sectionGeneric says: Section Name: BICQVER# , Length:  720
sectionSubRecord says: Section Name: GAME Number of subrecords: 1
sectionGeneric says: Section Name: GAME , Length:  7581
sectionSubRecord says: Section Name: GAME Number of subrecords: 848
sectionGeneric says: Section Name: GAME , Length:  278
sectionGeneric says: Section Name: GAME , Length:  65536
sectionGeneric says: Section Name: GAME , Length:  0
sectionGeneric says: Section Name: GAME , Length:  0
sectionGeneric says: Section Name: GAME , Length:  0
sectionGeneric says: Section Name: GAME , Length:  1710592
Traceback (most recent call last):
  File "./civ3parse.py", line 67, in <module>
    main()
  File "./civ3parse.py", line 65, in main
    parseSave()
  File "./civ3parse.py", line 60, in parseSave
    sectionSubRecord(saveFile, sectionName)
  File "./civ3parse.py", line 24, in sectionSubRecord
    sectionGeneric(saveStream, sectionName)
  File "./civ3parse.py", line 15, in sectionGeneric
    sectionLength = readLength(saveStream)
  File "./civ3parse.py", line 11, in readLength
    (length,) = struct.unpack('i',buffer[0:4])
struct.error: unpack requires a string argument of length 4

Code:
Spoiler :
Code:
#!/usr/bin/env python

# 2013-04-19 Start of attempt to parse a decompressed Civ3 save file
# unc-test.sav

import struct           # For parsing runs of binary data
import sys              # for exiting program during development/debugging

def readLength(saveStream):     # I repeatedly need to read a 4-byte length integer from the file stream
                buffer = saveStream.read(4)
                (length,) = struct.unpack('i',buffer[0:4])
                return length

def sectionGeneric(saveStream, sectionName):    # Reads length, then skips that many bytes
                sectionLength = readLength(saveStream)
                print "sectionGeneric says:" , "Section Name:", sectionName, ", Length: ", sectionLength
                saveStream.seek(sectionLength,1)

def sectionSubRecord(saveStream, sectionName):          # Reads length in subrecords then iterates through them
                numSubRecords = readLength(saveStream)
                print "sectionSubRecord says:", "Section Name:", sectionName, "Number of subrecords:", numSubRecords
#               sys.exit("Stopping point in development")
                while numSubRecords > 0:
                        sectionGeneric(saveStream, sectionName)
                        numSubRecords -= 1


def sectionCIV3(saveStream):    # I don't know much about this section yet
        print "Skipping 26 bytes because the CIV3 header doesn't seem to be followed by a length"
        saveStream.seek(26,1)

def sectionBICQ(saveStream):    # This section needs slightly different handling; read the "VER#" then iterate subrecords
        buffer = saveStream.read(4)
        (thisSaysVerNum,) = struct.unpack('4s', buffer[0:4])
        print "sectionBICQ says: This should say VER#:", thisSaysVerNum
        sectionSubRecord(saveStream, "BICQVER#")

#def sectionGAME(saveStream):   # Figure out what to skip until I figure out the structure


def parseSave():
        saveFilePath = "unc-test.sav"

        saveFile = open(saveFilePath, 'rb')

        # way to loop until EOF adapted from http://stackoverflow.com/questions/1752107/how-to-loop-until-eof-in-python
        for buffer in iter(lambda: saveFile.read(4), ''):
                (sectionName,) = struct.unpack('4s',buffer[0:4])
                # Ensure sectionName is ASCII, taken from http://stackoverflow.com/questions/196345/how-to-check-if-a-string-in-python-is-in-ascii
                try:
                        sectionName.decode('ascii')
                except UnicodeDecodeError:
                        sys.exit("ERROR: parseSave(): sectionName is not an ASCII sting")
                # Apparently I need to learn polymorphism but for now I can make it work with chaned if-then-else
                if sectionName == 'CIV3':
                        sectionCIV3(saveFile)
                elif sectionName == 'BICQ':
                        sectionBICQ(saveFile)
                elif sectionName == 'GAME':
                        sectionSubRecord(saveFile, sectionName)
                else:
                        sectionGeneric(saveFile, sectionName)

def main():
        parseSave()

main()

Edit: I think I'll solve (temporarily) GAME by treating it as subrecords if length is less than some number, perhaps 10, and treat it as a length-in-bytes section if greater than 10.

Edit 2: I see now the GAME format is in the BIC format info and is more complex than length=bytes or length=subrecords
 
There are indeed multiple (two I think) GAME sections in the SAV... one is the same as the BIQ (at least, if the savegame is from a custom BIQ), the other(s?) is not. It looks like there are several other instances of "GAME" that don't correspond with sections, as well. I'm a lot less familiar with the SAV than the BIQ though.

You're right that there are several less-than-desirable limitations/pain points for CAII. It is Windows only, seems to have a lukewarm relationship with Vista/7, and a possibly cooler one with 8. And it has the misfortunate of using .NET 1.1, which isn't included with later versions and requires a separate download. So it certainly isn't as compatible as would be ideal. I guess I don't notice this as much as most people since I'm still on XP. On the other hand, I know there's been some work on it in the past year via what ainwood posted. Though that doesn't necessarily mean there will be a new release at any given time.

You're off to a start, though! There's still a long way to go to parsing the file. Probably a good idea to make it skip sections you haven't done yet - that'll make it take less time until it's at the point where it can do one or two things, and be expanded on from there.
 
First goal: parse uncompressed save-game files. At first my parser will just print the 4-char section name and report the bytes of data read.

Early challenges: While most sections have 4-char headers and 4-byte length, some have a count of subrecords instead of byte length, so there will be some conditional logic in this very basic parser.

The good news is I'm making some progress on my first goal, and that I correctly identified the early challenges, but it's proving to be a bit more complex than I anticipated.

There are several places so far where the format doesn't conform to section-title, section-length (in bytes or # of subsections).

I am currently reading the file info 4-bytes at a time for section names and integer counts and then skipping over the actual data (think of it as a stub for later expansion into processing the data) and reading the next section header. I'm using the same save-game file every time so far, so when I'm stumped I'm just skipping manually to the next (apparent) section header.

One of my happier moments was when all the WRLD, TILE and CONT section headers and lengths flew by without specialized parsing, but there is still quite a bit to figure out before I can start parsing arbitrary save files and extracting useful data.

I suspect the file is simply a binary dump of C structures, and if I were able to replicate the data structures I could slurp them up directly with no converting, but unless and until I figure that out I'll stick with parsing it section-by-section.

Output (shortened):
Spoiler :
Code:
Debug skipBytes Skipping 0x1a bytes hex offset 0x4
Debug sectionGeneric BIC  length 524 hex offset 0x26
sectionBICQ says: This should say VER#: VER#
sectionSubRecord says: Section Name: BICQVER# Number of subrecords: 1
Debug sectionSubRecord BICQVER# numSubRecords 1 hex offset 0x23e
Debug sectionGeneric BICQVER# length 720 hex offset 0x242
Debug skipToOffset Skipping to offset 0x32c6 hex offset 0x516
Debug sectionGeneric CNSL length 228 hex offset 0x32ce
Debug sectionGeneric WRLD length 2 hex offset 0x33ba
Debug sectionGeneric WRLD length 164 hex offset 0x33c4
Debug sectionGeneric WRLD length 52 hex offset 0x3470
Debug sectionGeneric TILE length 36 hex offset 0x34ac
Debug sectionGeneric TILE length 12 hex offset 0x34d8
Debug sectionGeneric TILE length 4 hex offset 0x34ec
Debug sectionGeneric TILE length 128 hex offset 0x34f8
[...]
Debug sectionGeneric TILE length 12 hex offset 0x606a4
Debug sectionGeneric TILE length 4 hex offset 0x606b8
Debug sectionGeneric TILE length 128 hex offset 0x606c4
Debug sectionGeneric CONT length 8 hex offset 0x6074c
Debug sectionGeneric CONT length 8 hex offset 0x6075c
Debug sectionGeneric CONT length 8 hex offset 0x6076c
Debug sectionGeneric CONT length 8 hex offset 0x6077c
Debug sectionGeneric CONT length 8 hex offset 0x6078c
Debug sectionGeneric CONT length 8 hex offset 0x6079c
Debug sectionGeneric CONT length 8 hex offset 0x607ac
Debug sectionGeneric CONT length 8 hex offset 0x607bc
Debug sectionGeneric CONT length 8 hex offset 0x607cc
Debug sectionGeneric CONT length 8 hex offset 0x607dc
Debug sectionGeneric CONT length 8 hex offset 0x607ec
Debug sectionGeneric CONT length 8 hex offset 0x607fc
Debug sectionGeneric CONT length 8 hex offset 0x6080c
Debug sectionGeneric CONT length 8 hex offset 0x6081c
Debug sectionGeneric CONT length 8 hex offset 0x6082c
Debug sectionGeneric CONT length 8 hex offset 0x6083c
Debug sectionGeneric CONT length 8 hex offset 0x6084c
Debug sectionGeneric ^D^@^@^@ length 4 hex offset 0x6085c
Debug sectionGeneric ^C^@^@^@ length 3 hex offset 0x60868
Debug sectionGeneric ^@^C^@^@ length 768 hex offset 0x60873
Debug sectionGeneric ^@^@^@^@ length 0 hex offset 0x60b7b
[...]

Code:
Spoiler :
Code:
#!/usr/bin/env python

# 2013-04-19 Start of attempt to parse a decompressed Civ3 save file
# unc-test.sav

import struct           # For parsing runs of binary data
import sys              # for exiting program during development/debugging
import inspect          # for listing calling function name in debug prints

def readLength(saveStream):     # I repeatedly need to read a 4-byte length integer from the file stream
        buffer = saveStream.read(4)
        (length,) = struct.unpack('i',buffer[0:4])
        return length

def printDebug(saveStream, debugInfo):
        readPosition = saveStream.tell()
        print 'Debug ' \
        + inspect.stack()[1][3] \
        + ' ' \
        + debugInfo \
        + ' hex offset ' \
        + hex(readPosition) \
        + ' ' \
        + ' ' \

def skipBytes(saveStream, skipNum):
        printDebug(saveStream, 'Skipping ' + hex(skipNum) + ' bytes')
        saveStream.seek(skipNum,1)

def skipToOffset(saveStream, skipTo):
        printDebug(saveStream, 'Skipping to offset ' + hex(skipTo))
        saveStream.seek(skipTo,0)

def sectionGeneric(saveStream, sectionName):    # Reads length, then skips that many bytes
        sectionLength = readLength(saveStream)
#       print "sectionGeneric says:" , "Section Name:", sectionName, ", Length: ", sectionLength
        printDebug(saveStream, sectionName + ' length ' + str(sectionLength))
        saveStream.seek(sectionLength,1)

def sectionSubRecord(saveStream, sectionName):          # Reads length in subrecords then iterates through them
        numSubRecords = readLength(saveStream)
        print "sectionSubRecord says:", "Section Name:", sectionName, "Number of subrecords:", numSubRecords
        printDebug(saveStream, sectionName + ' numSubRecords ' + str(numSubRecords))
#       sys.exit("Stopping point in development")
        while numSubRecords > 0:
                sectionGeneric(saveStream, sectionName)
                numSubRecords -= 1


# Replacing with generic skip function
#def sectionCIV3(saveStream):   # I don't know much about this section yet
#       print "Skipping 26 bytes because the CIV3 header doesn't seem to be followed by a length"
#       saveStream.seek(26,1)

def sectionBICQ(saveStream):    # This section needs slightly different handling; read the "VER#" then iterate subrecords
        buffer = saveStream.read(4)
        (thisSaysVerNum,) = struct.unpack('4s', buffer[0:4])
        print "sectionBICQ says: This should say VER#:", thisSaysVerNum
        sectionSubRecord(saveStream, "BICQVER#")

# using a skip function here temporarily
def sectionGAME(saveStream):    # A more complex system of subrecords
#       numScenarioProperties= readLength(saveStream)
#       while numScenarioProperties > 0:
#               scenarioPropertyLength =  readLength(saveStream)
#               print "Scenario Property #", numScenarioProperties, "Length:",scenarioPropertyLength
#               numScenarioProperties -= 1
#       sys.exit("Stopping point in development")
        print "Skipping 11384 bytes because I can\'t figure out the GAME section"
        saveStream.seek(11384,1)


def parseSave():
        saveFilePath = "unc-test.sav"

        saveFile = open(saveFilePath, 'rb')

        gameSectionsRead = 0

        # way to loop until EOF adapted from http://stackoverflow.com/questions/1752107/how-to-loop-until-eof-in-python
        for buffer in iter(lambda: saveFile.read(4), ''):
                (sectionName,) = struct.unpack('4s',buffer[0:4])
                #printDebug(saveFile, 'just read name')
                # Ensure sectionName is ASCII, taken from http://stackoverflow.com/questions/196345/how-to-check-if-a-string-in-python-is-in-ascii
                try:
                        sectionName.decode('ascii')
                except UnicodeDecodeError:
                        readPosition = saveFile.tell()
                        errorMessage = 'ERROR: parseSave(): sectionName is not an ASCII string. Offset:' + str(readPosition) + ' Hex: ' + hex(readPosition)
                        sys.exit(errorMessage)
                # Apparently I need to learn polymorphism but for now I can make it work with chained if-then-else
                if sectionName == 'CIV3':
                        #sectionCIV3(saveFile)
                        skipBytes(saveFile, 26)
                elif sectionName == 'BICQ':
                        sectionBICQ(saveFile)
                elif sectionName == 'GAME':
                        #sectionGAME(saveFile)
                        if gameSectionsRead > 0:
                                sectionGeneric(saveFile, sectionName)
                        else:
                                skipToOffset(saveFile, 0x32c6)  # Skipping; assuming this offset will not work with arbitrary save files
                        gameSectionsRead += 1
                else:
                        sectionGeneric(saveFile, sectionName)

def main():
        parseSave()

main()

I noticed while pasting this info that I'm not reading the first GAME section; I reversed the if-then-else; it should be reading the first GAME section and then skipping the second, but I'll fix it after this post rather than try to fix it, re-test it and re-paste.
 
I suspect the file is simply a binary dump of C structures, and if I were able to replicate the data structures I could slurp them up directly with no converting, but unless and until I figure that out I'll stick with parsing it section-by-section.

My thoughts keep going back to this. If I were to abstractly describe my strategy so far, it would be "read a stream of unordered sections generally beginning with a 4-char section name and then a 4-byte length (in bytes or subsections), the composition of each section being determined by its name."

However, even though I don't have the source for Civ3 I can probably safely make several assumptions about the (BIQ and SAV) data files:

  • They are simply a contiguous binary dump of native data structures
  • Civ3 was probably programmed in C
  • Within each file type (BIQ or SAV) the order of data structures will always be the same
  • The data is probably a series of structures, some including arrays or lists of sub-structures hierarchical
  • Given the previous two points, I might assume that there are a set of top-level structures, and SAV and BIQ files are ordered dumps of some top-level structures, some of which are shared between the files and some of which are unique to each file.
  • Given the last point, there probably exists a superset of top-level structures that non-repetitively define the data structures contained in all SAV and BIQ files

It is not in my set of goals to read BIQ files, but it seems more and more apparent that BIQ and SAV have a lot in common and likely share data structures, so in reading SAV files I am learning and implementing a lot about reading BIQ files.

A hierarchy of structs would help explain some of the apparent inconsistencies I am encountering, and as I start to peek at reading C structures I see mentions of data padding which would also explain a couple of the inconsistencies I've run into.

So I think I may start over and see if I can define C-style structures to slurp in the data. I might even do it in C even though I don't think that's what I want the final product to be coded in.

Edit: It is also not in my set of goals to write a SAV or BIQ, but if I figure out a data structure to read the file directly it would seem to be trivial to write it back again, even after changing values.
 
Question for others: Does each SAV file have an entire BIQ in it? Or are there some things that are in the BIQ but not the SAV?


Brain dump of some thoughts in my head about data hierarchy:

I have to assume TILE is not a top-level structure. WLRD may be a parent of TILE and CONT, or perhaps they all share a parent, given there are multiple WRLD headers.

CIV3 may be a header section, or perhaps it is the singular top-level data structure compromising the entire file. If the latter it may not matter as it would just mean some potential padding at end-of-file.

I'm guessing "BIC " or BICQ is a parent of a number of following objects, including the first GAME.

The second GAME may be top-level or possibly a child of CIV3. I don't think it is part of the BIQ data. I am going to make a wild guess that it is specific data on the game settings, and maybe that's why it doesn't conform to my previous expectations. Perhaps it's like this:

Code:
GAME
  difficulty
  barb aggression
  other settings for this game
  [sub-sections]
    [...]

If it is a parent structure with simple data members before other nested structures that would explain why I am having trouble understanding it using my previous methods.

I wonder if BIC and GAME are peer parent elements? That would seem to make sense, with BIC housing the BIQ rulesets and GAME having specific-game data.

That gives me an idea: I'll create a game based off of a custom BIQ and then see if the BIQ is included in its entirety in the SAV file. Also, I'll create a game from a BIQ with a predefined map and compare how it is stored differently between the BIQ and SAV.
 
My brain is starting to melt from decoding the save file format. Also, actual progress has mostly halted, so I'm eager to do something else that might do something neat, like display a rudimentary map in HTML by cheating and just reading the tile info.

But there seem to be four TILE sections per actual game tile. This is a 60x60 game, and if I understand the isometric layout that's 30x60 because each column is 60 tiles high, but going directly east/west is 2 grid moves, so there exists (from upper-left) a 0,0 and 0,2 and 1,1 and 1,3 but no 0,1 or 0,3 or 1,0 or 1,2. So 30x60 is 1800 tiles, but there are 7200 TILE sections with repeating lengths of 36, 12, 4 and 128 bytes (not including the 4-byte name or 4-byte length integer).

I thought I came here with a question, but maybe not, or maybe I worked out the answer during the writing of the post.
 
Your brain will melt if you try to decode the file structure for too long! Have lemonade or another cool beverage available to avert meltdown! Getting it to do something small, but neat, before finishing the file structure is also a good idea.

As for the format specifics, see my PM. I'm a fan of researching my own technologies, but the difficulty level here is high enough that I'd really recommend acquiring it via diplomacy instead. You are right about the 60x60 map actually having 1800 tiles, as well as the reasoning to get there.

Civ3 was programmed in C++, using Microsoft Visual C++. My guess is that it uses Visual C++ 6, although it could be a slightly earlier version than that. At least some of the Civ3 developers used CodeWright in conjunction with VC++, although this wouldn't have affected the compiled data structures.
 
Thanks for the PM. No programming-fairy-godmothers have appeared yet. I was about to walk away from it for a while (probably would have been a good idea), but I managed to accomplish something that made me happy today: I successfully created a (rudimentary, text-based) map of which tiles are visible to the player and which aren't.

I started over with a hybrid of my previous two approaches: I am assuming the data is in a predetermined order but am still decoding it pieces at a time. However I am using classes that self-init by reading in the data they expect, so I just instantiate a class by passing it the file stream and it takes what it needs/expects.

Currently I am jumping straight to the first TILE and only reading the expected number of TILEs for a 60x60 map, but this class is usable as-is when I expand what I am reading in.

I am also reading the section size into a buffer, unpacking any wanted data into a class attributes and then discarding the buffer, so the data that's left is only what I bothered/wanted to keep.

Anyway, enough of the boring stuff. Here's the cool:

'#' indicates a visible tile, '.' is a fogged tile, spaces are just to align the "tiles"
Spoiler :
Code:
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 . . . . . . . . # # # # # . . . . . . . . . . . . . . . . .
. # # # # . . # # # # # # # . . . . . . . . . . # . . . . .
 # # # # # . # # # # # # # # . . . . . . . . . # # . . . . .
# # # # # # # # # # # # # # # . . . . . . . . # # # . . . .
 # # # # # # # # # # # # # # # . . . . . . . # # # # . . . #
# # # # # # # # # # # # # # # # . . . . . . # # # # # . . #
 # # # # # # # # # # # # # # # # . . . . . # # # # # # . # #
# # # # # # # # # # # # # # # # . . . . . # # # # # # . # #
 # # # # # # # # # # # # # # # # . . . . # # # # # # # # # #
# # # # # # # # # # # # # # # # # . . . # # # # # # # # # #
 # # # # # # # # # # # # # # # # # . . # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # . . # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # . # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # . # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # . # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # . . # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # . . # # # # # # # # # # # # # # # # # # # # # # # # # #
 # . . . # # # # # # # # # # # # # # # # # # # # # # # # # #
# . . . . # # # # # # # # # # # # # # # # # # # # # # # # #
 . . . . . # # # # # . # # # # # # # # # # # # # # # # # # #
. . . . . . . . . . . . # # # # # # # # # # # # # # # # . #
 . . . . . . . . . . . . # # # # # # # # # # # # # # # . . .
. . . . . . . . . . . . . # # # # # # # # # # # # # # . . .
 . . . . . . . . . . . . . # # # # # # # # # # # # # . . . .
Compare that to the in-game F3 minimap:


The code:
Spoiler :
Code:
#!/usr/bin/env python

# 2013-04-20 Another attempt at reading a save file; this time I'm going
#   to try extracting just the map data

import struct   # For parsing binary data

class GenericSection:
    """Base class for reading SAV sections."""
    def __init__(self, saveStream):
        buffer = saveStream.read(8)
        (self.name, self.length,) = struct.unpack_from('4si', buffer)
        self.buffer = saveStream.read(self.length)

class Tile:
    """Class for each logical tile."""
    def __init__(self, saveStream):

        self.Tile36 = GenericSection(saveStream)
        del self.Tile36.buffer

        self.Tile12 = GenericSection(saveStream)
        del self.Tile12.buffer

        self.Tile4 = GenericSection(saveStream)
        del self.Tile4.buffer

        self.Tile128 = GenericSection(saveStream)
        self.is_visible_to = get_int(self.Tile128.buffer, 0)
        self.is_visible_now_to = get_int(self.Tile128.buffer, 4)
        self.is_visible = self.is_visible_to & 0x02
        #self.is_visible = self.is_visible_to & 0x10
        self.is_visible_now = self.is_visible_now_to & 0x02
        del self.Tile128.buffer

class Tiles:
    """Class to read all tiles"""
    def __init__(self, saveStream, width, height):
        self.width = width      # These may eventually be redundant to a parent class
        self.height = height
        self.tile = []          # List of individual tiles
        logical_tiles = width / 2 * height
        while logical_tiles > 0:
            self.tile.append(Tile(saveStream))
            logical_tiles -= 1

    def table_out(self):
        """Return a string of a simple text table of visible tiles."""
        table_string = ''
        for y in range(self.height):
            if y % 2 == 1:
                table_string += ' '
            for x in range(self.width / 2):
                if self.tile[x + y * self.width / 2].is_visible:
                    table_string += '#'
                else:
                    table_string += '.'
                table_string += ' '
            table_string += '\n'
        return table_string


def get_int(buffer, offset):
    """Unpack an int from a buffer at the given offest."""
    (the_int,) = struct.unpack('i', buffer[offset:offset+4])
    return the_int

def parse_save():
    saveFilePath = "unc-test.sav"
    saveFile = open(saveFilePath, 'rb')
    print 'HACK: Skipping to first TILE in my test SAV.'
    saveFile.seek(0x34a4, 0)
    print 'HACK: Instantiating the class that reads TILEs with width x height hard-coded to my test SAV.'
    game = Tiles(saveFile, 60, 60)
    print 'Printing something(s) from the class to ensure I have what I intended'
    #print game.name, game.length
    #print game.tile.pop()[1].length
    print game.width, game.height
    print game.tile[0].Tile128.length
    print game.tile[1000].Tile128.length
    print game.tile[0].is_visible_to
    i = 0
    while i < 10:
        print i, game.tile[i].is_visible
        i += 1
    print game.table_out()


def main():
    parse_save()

main()

Edit: I just realized a problem or two: Although I seem to have successfully interpreted the "is visible" and "is visible now" (line-of-sight fog) flags, I know in-game if I do not have line-of-sight the tile may change, but its appearance to the player is what it was last time it was in line-of-sight, so I will need to figure out how to show the non-spoiling tile info. But I am curious about multiplayer, or for that matter if the AI has the same line-of-sight tile issues. Assuming they do, there must be tile attributes repeated for each player to show what it looked like when they last had line-of-sight.
 
Top Bottom