Civ3 Show-And-Tell

[Puppeteer]

While CivFan was down I got the JSON version to feature parity with the previous SVG version. I think all that was missing was cities and BGs. I also cleaned up the code and repo a lot, so there are fewer files and less code overall.

Visually it doesn't look or behave much differently in the browser than before, so I don't have any live examples up yet. The biggest visible change is that I redid some of the colors which is easier to do now because it's just CSS styling.

I'm out of steam again, so I'm not sure anything is going to change soon, but the code is available in the repo, and the JSON output is pretty useful I think.

Edit: Oh, the other difference from the previous SVG version is that I tried to use Unicode characters for many of the resources, but their implementation on each browser and platform is different. In my screenshots thee are blue whales and brown & white cows as seen on a Chromebook and Android, but the Windows Chrome shows different glyphs if it has them at all. And of course IE and Firefox are a bit different, too. If/when I get to working on it again I'll probably go back to Creative Commons licensed SVG icons again.

 

[Quintillus]

The SVG icons sounds like the better option to me. Unicode is good in theory, but from the description it sounds like going that route would be like going back to the bad old days of "Works best in Netscape 4.7", with it only displaying as intended in a small number of browsers.

Out of curiosity, are you using Python 2 or Python 3? They're similar enough that it probably doesn't matter a whole lot either way, but I know there's a few things that are only compatible with one or the other.

 

[Puppeteer]

Python 2 because it seemed to be most popular at the time I started. I'm not real familiar with the differences. This is the most involved thing I've done with Python; anything else has been shorter script-y things, and usually Python 2.7 is the installed version. (Probably due to legacy OS scripts.)

 

[Quintillus]

Python 2 because it seemed to be most popular at the time I started. I'm not real familiar with the differences. This is the most involved thing I've done with Python; anything else has been shorter script-y things, and usually Python 2.7 is the installed version. (Probably due to legacy OS scripts.)

Yeah, I think 2.7 is still a bit more common. I did some updating of a Python program at work this spring, and while the material I studied to get myself familiar with Python was for 3.x, it was mostly seamless to apply it to the 2.7 program I was working on. Definitely easier than moving to 3.x would have been, since there were a couple dependencies that only worked on 2.x.

 

[Puppeteer]

The online converter/viewer has been rickety since I set it up. Until now. I finally made it so it doesn't switch ports every restart and will auto-start upon reboot. So it should stay up a vast majority of the time now.

It still doesn't show all the months on the front page, but the folder view works.

 

[Quintillus]

That's certainly a handy improvement. The bump also reminded me to put the link to the map in the first post of my story, along with credits for the Civ3 Show and Tell utility.

 

[Puppeteer]

I haven't played Civ3 in quite a while, and I don't think I will be soon, but something got me thinking about this project again. If I do anything in the short term it will be more about using this project as an excuse to play with other technologies like I did with d3.js last year and not necessarily to move the project towards any particular milestone.

My two new (-ish) toys I'm thinking about using here are Go and Spark.

Go(lang) is one of the trendy new languages I'm late to play with. I've only done "Hello, world!", read about how they set their workspaces up and look at other peoples' code. But what strikes me about it is that it compiles to native executables. A guy at work made a very cool utility, and it works the same on Windows, Mac and Linux with native executables. The code I've seen looks pretty efficient, too.

Apache Spark wouldn't necessarily be a component of this project but a way to help me figure out the remaining save file decoding. I got this far by parsing out sections, comparing hex dumps and even showing codes on maps to visually identify correlations. Which is really cool and makes me seem smart (maybe). But a few months ago I used Spark to try to classify log file entries as common or rare with the intention of finding useful (unusual) log entries out of a noisy Java app. As I did so I started realizing that my one-at-a-time comparisons could be multiplied by teaching Spark how to do it. I had been cleverly classifying and graphing log entries to visually look for patterns that could indicate problems, but Spark can be taught to do all that much faster than I can do it.

The problem doesn't apply exactly to decoding save game files, but I'm thinking I might be able to have it find patterns for me...which numbers change when the number of resources change, for example. Which number correlate with other data block size changes? Instead of hex dumping one map at a time, I should be able to feed Spark dozens or hundreds of maps and let it spot the patterns and differences.

Also, I've considered before doing things like toggling a game rule/option and then finding that setting in the file, but it would be tedious for me to do it. But if I can automate the creation of a save file with given options I can do it dozens of times and analyze with Spark.

Also, with Antal1987's data structure/class dumps, it would have been tedious for me to match up the unlabeled structures to data types, but Spark might be able to match at least some of those up.

And we're wandering into pie-in-the-sky territory here, but if I am able to automate starting a game with changing settings (which BTW I know is possible as there is a utility to find favorable start locations, and CAII can center the game on a city) then I should be able to move units and make changes in cities. What if I could get Spark to play Civ3? Or even analyze how the AI reacts to various moves and map conditions? (e.g. Move a stack of offense units towards AI city, read the autosave each turn to see what AI does.) Or even have a "human" AI played by Spark as a better (?) opponent.

Anyway, so possible refactor/re-language coming up, and possibly decoding more of the file with the help of Machine Learning or at least some automation in comparing files.

I'm also thinking about scraping the web--or at least this site--for Civ3 save files. If I'm going to analyze en masse it could be fun or useful to have more sources for save files. It could also help me figure out how to identify multiple save game files from the same game.

Edit: I'm having some other crazy thoughts, too:

- Have bot automatically pull save games from Stories & Tales and parse maps
- Have browser plugin that can find the c3sat maps for the current thread
- Have search engine for save games / stories threads
- Searchable Civilopedia with web browser interface

 

[Puppeteer]

Or...

I could get sick to take up the rest of the weekend, instead.

 

[Puppeteer]

Feeling better, but the weekend is over. I haven't coded (or decoded) a byte, but I've been thinking.

First, the whole decompression problem is still a problem, and I started wondering if Go could be used as a cross-platform decompressor utility/library. Maybe. First, there is https://github.com/madler/zlib/tree/master/contrib/blast which has a decompressor utility (source code only) in C. I could cross-compile it or try to translate it to Go manually or even with a converter script like https://github.com/rsc/c2go or https://github.com/xyproto/c2go . Or I could try to modify some of the existing Go code for LZW decompression to work on PKWare DCL (Civ3's save game compression.) Links in spoiler tag.

Spoiler :

https://golang.org/pkg/compress/lzw/
https://golang.org/src/compress/bzip2/huffman.go
https://golang.org/src/compress/lzw/reader.go
https://groups.google.com/forum/#!msg/comp.compression/M5P064or93o/W1ca1-ad6kgJ

My early thoughts for using Spark to analyze the save format are to take Antal1987's memory structure dumps, calculate sizes and match them to file data. But now I'm peeking at his header files, and there may actually be a log of useful info there without having to analyze & compare. I had recalled the structures being unnamed, but he has a lot of them named and many fields identified.

Furthermore, I think the variable-sized sections that confused me earlier are likely due to non-object arrays inside the larger class. So manually I should be able to look for repeating data and hopefully a record count can be found immediately preceding that.

It also just occurred to me that some of what I previously considered padding could be zeroed counters for object/section/class arrays that aren't present in a particular save file.

 

[Quintillus]

Those are certainly some interesting ideas (and a bummer of a way for the weekend to wind up). You've clearly thought about the possibilities of Spark a lot more than I have, but one additional idea I have that may lend itself towards that is figuring out what some of the remaining unknown BIQ values are. If we could automatically parse a good chunk of BIQs and find those with unexpected or repeated values for those, it might shine some light on what they represent.

Do you know if Go allows you to easily compile for platforms other than the one you are on? For example, I'm on Windows, can I compile for Mac while I'm still in Windows? That would be a nice feature as a developer; one of the reasons I've used Java is it allows the equivalent of that, so I don't have to have a Mac to create code that runs on OS X as I would in, say, C++. Granted, there's more flexibility with that now than 10 years ago between browser applications that are cross-platform, and .NET/Mono having improved cross-platform support, but a language that allowed native executables (not even virtual machines like .NET/Java) cross-platform would have some appeal.

Edit: Re: Decompression. A couple summers ago (really? that long ago already?) I posted the utility that I use for decompression in my editor, which works for BIC/BIX/BIQ, and SAV (and maybe some other Civ-related files). It's here. chiefpaco wrote the decompression Java code way back in the days of yore, and I just put a front end on it to make it easy to invoke either from a GUI, or likely more useful for your cases, from a command-line. What my editor does is invoke it as a separate process on the command line, just giving it the desired input and output file names, just as you would if you were invoking it yourself. I haven't verified the decompression code myself, but have yet to see or hear of it failing, so it seems to be pretty reliable.

So, if your system already has Java, that may be the easiest way around that problem.

 

[Puppeteer]

Go does allow for cross-compiling, although I haven't tried it yet.

Wow, that was easy. I just set the GOOS (windows/linux/darwin) and GOARCH (386/amd64) environment variables and did a "go build HelloWorld.go". I built these in Win10 64-bit. I tested the Linux ones and the Windows 386 one, but I can't test Darwin. It occurred to me that cross-compiling to Windows 386 will be important because a lot of folks are still running Civ3 on WinXP. (Haven't tested it on XP, though.)

http://lib.bigmoneyjim.com/civfan/gocrosscompile/index.html

The current Go version is 1.6, and apparently cross-compiling became much easier in 1.5:

http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5
http://golangcookbook.com/chapters/running/cross-compiling/

code:

Spoiler :

package main

import "fmt"

func main() {
    fmt.Printf("Hello, world.\n")
}

I want to play with Spark more, but after looking at Antal1987's data structures and seeing they are much better-identified than I remembered, I now think it will be easier to match them up by hand/eye than to try to tokenize those and the data file in a comparable way.

Previously I had little interest in the BIQ and even today my code seeks straight to the WRLD sections of the save. But now I want to parse everything, so I'm thinking I'll start at the beginning of the file which means the BIQ is the second section IIRC.

Once I get the BIQ reading in we can compare your knowledge with Antal1987's repo and then run mass comparisons on unknown values. I'm not sure Spark is needed, but I might use it, anyway. TBD.

I know there are decompression utilities, and I can manage to decompress stuff locally, but at the moment I would like c3sat to be able to handle both compressed and uncompressed saves without an external utility, so I'm still thinking I want to translate zlib/contrib/blast into Go and then it should work cross-platform.

 

[Puppeteer]

Oh, also, once I get to the map again, I'm going to try to use free SVG icons, but for resources without icons I think I'll have 3 icons for luxury/strategic/bonus resource and then hovering over that tile or icon will pop up or display in a designated area the resource name.

Font Awesome, Open Iconic and Octicons are three sets of no-BS SVG icon sets I'll look at, but honestly I don't think I'll find many to use in-game. In the long term I'll continue to either look for more CC-licensed icons or make my own crappy ones.

 

[Quintillus]

As it happens, I just rebooted into my XP x64 partition (just switched to 8.1 as my primary OS on my main machine a month ago; still have XP on my laptops and a partition on the desktop), and I was able to successfully run the Hello World program. Since my older laptop was right here, I tested it on XP 32-bit as well, and it works there as well. It does look like it's quite easy to cross-compile.

Font Awesome is a nice library, I've used it before at work. I don't know how much it will have that will be relevant to Civ, but they're good icons. Haven't used the other two yet, but taking a glance at them they look to be of similar quality.

Not having external dependencies is definitely nice. And converting it over could be an interesting exercise, so I'd say Go for it! (pun intended)

 

[Puppeteer]

I did some Go. After waffling about a bit on whether to try to read uncompressed data or somehow implement decompression, I went for decompression.

My c2go attempt failed miserably, so I started with a very basic open the file, read the first two bytes. That works, although for a while I was confusing myself by not realizing I kept running an old .exe and not seeing my code changes.

I found that all the compressed save files I have have the header 0x0006 which means literal data will always be handled one way and the dictionary size will always be 4k. In the general case there would be two literal options and three dictionary sizes, but in the specialized case of Civ3 data files I think I can focus more narrowly and write less code! And even better, the "0" literal format looks like the easiest to implement. Suddenly this looks a lot easier.

 

[Puppeteer]

This is starting to look very doable. I'm going to use https://github.com/dgryski/go-bitstream which should provide functions for me to pull a bit or a byte regardless of file byte alignment, which should prevent my having to keep tracking bit shifts myself.

Basically the code will be:
- Read the first two bytes. They should be 0x0006, "CI" or "BI" for files this program would ever read. Only 0x0006 needs decompression.
- Read a bit. If 0, take the next 8 bits as a literal byte. (And presumably add to dictionary.)
- If 1, it's more complicated, but do stuff. One of these token "words" means end of file/stream.
- Repeat from read a bit.

 

[Puppeteer]

Progress. I have the first two steps working (minus the dictionary), and it currently prints the 2-byte header and then each byte in hex until it hits the first non-literal value. (Instead it's an instruction to repeat data previously seen.)

0006
43 49 56 33 00 1a 18 00 00 00 0a

Those first four numbers on the second line say "CIV3" in ASCII, which is how all SAV files start. W00t!

The bitstream library I started using pulled the bits in the wrong order, so I had to modify the ReadBit() function ever so slightly and just swapped out ReadByte() wholesale for a loop that calls ReadBit() 8 times and right-shifts them into the byte to get the correct value.

Aside from implementing the dictionary and repeat decoding I need to make it a usable command line decompressor, and I might take a detour and learn the "Go" way of doing that, because they seem to have some scaffolding for command line interfaces and parameters.

Once I get it decompressing files I may take another detour and make a utility to read the map seed and perhaps the world options that matter for recreating the map. That's a utility often asked for, and I seem to be somewhat close to having a cross-platform native solution.

 

[Puppeteer]

Here is an early test version of readciv3 for Windows, Darwin (Mac) and Linux. You can readciv3 path/to/civ3.SAV, and it will decode the first few bytes, which so far have been identical on every SAV file for me, but that's expected. It currently displays them in a hex dump with any ASCII characters on the right.

It should produce output against any file, even if the output is "Uncrecognized file type." Its most interesting output will be against compressed and uncompressed SAV and BI* files.

This is not useful in itself yet, but it's a stepping stone to a cross-platform decompressor for Civ3 files.

http://lib.bigmoneyjim.com/civfan/gocrosscompile/index.html

(Actually I realized I left out the "Unrecognized" detection. Adding that in and will upload the fixed executables within a few mins of this post.)

Windows output:

Spoiler :

> .\readciv3.exe 'C:\temp\civ3saves\Im Screwed.SAV'
Early test program. Call this program with a Civ 3 SAV file or BIC/X/Q file.
It will report if the file is compressed or uncompressed,
and print the first several bytes of any compressed files in hex dump format.
C:\temp\civ3saves\Im Screwed.SAV opened
Compressed Civ3 file detected
Data hex dump:
00000000  43 49 56 33 00 1a 18 00  00 00 0a                 |CIV3.......|

Dictionary logic not yet implemented.

Linux output:

Spoiler :

$ ./readciv3-linux-386  ./Mao%20of%20the%20Chinese,%20130%20AD.SAV
Early test program. Call this program with a Civ 3 SAV file or BIC/X/Q file.
It will report if the file is compressed or uncompressed,
and print the first several bytes of any compressed files in hex dump format.
./Mao%20of%20the%20Chinese,%20130%20AD.SAV opened
Compressed Civ3 file detected
Data hex dump:
00000000  43 49 56 33 00 1a 18 00  00 00 0a                 |CIV3.......|

Dictionary logic not yet implemented.

 

[Quintillus]

> .\readciv3.exe 'C:\temp\civ3saves\Im Screwed.SAV'

I remember when that SAV was posted, and still have it on my hard drive. Definitely a challenging situation, but I'm glad it's proving to be a helpful test case, and that initial progress is going well. Map seed/world options sounds like a good milestone to shoot for; that is indeed a popular request.

 

[Puppeteer]

I can now parse all the bits and bytes properly and arrive at the "end of stream" token. I'm not yet actually copying the dictionary data to output, so the output is useless, but I can tell I'm getting the sequence and literal bytes correct because I see recognizable Civ3 strings in the data.

The remaining task is to simply copy length/offset sequences of bytes from the dictionary/lookback-buffer to output which seems quite simple in concept, but since I'm new to Go I have to figure a few things out first, and if any of my token data is wrong it's going to be a bugger to figure out. But theoretically I'm close.

Currently I'm doing a hex dump of the data, but I'm starting to think of what I want the command line to do. I think by default it should take a path as input as it does now. By default I'm not sure if I want it to decompress to stdout or...something else. Either require an out filename or assume a filename based on the input file (-unc.SAV).

But whatever the default is, I think I want options to hex dump, to decompress to stdout or to write to a file. And since I'm planning for the code to be able to seamlessly process compressed or uncompressed files I'll also have to figure out if I'm reading an uncompressed file if I still want to write the output file which would then be simply a copy.

For the decompressor utility, of course. The seed reader utility and later BIQ/map/game-data utility will all be able to use the same code package to read the file but will have different outputs.

Sample output:

Spoiler :

> .\readciv3.exe 'C:\temp\civ3saves\about to win English,
 1340 AD.SAV'
Early test program. Call this program with a Civ 3 SAV file or BIC/X/Q file.
It will report if the file is compressed or uncompressed,
and print the first several bytes of any compressed files in hex dump format.
C:\temp\civ3saves\about to win English, 1340 AD.SAV opened
Compressed Civ3 file detected
Data hex dump:
00000000  43 49 56 33 00 1a 18 00  00 00 0a 7a 1d 96 d6 27  |CIV3.......z...'|
00000010  ca 54 4b a2 76 96 d0 81  5d 1a f7 42 49 43 20 0c  |.TK.v...]..BIC .|
00000020  02 b8 44 65 22 f0 cd cd  08 47 41 4d 45 9d 1d 0f  |..De"....GAME...|
00000030  1f 00 03 04 05 06 07 09  0b 0d 0e 0f 10 11 12 13  |................|
00000040  14 15 16 17 19 1a 1b 1c  00 1e 01 60 f0 ff ff 1c  |...........`....|
00000050  02 28 32 64 32 19 0a 02  50 c3 f0 38 f3 a0 f8 d9  |.(2d2...P..8....|
00000060  c6 ba f8 6c 61 67 75 65  cd 00 44 fd ff ff 50 63  |...lague..D...Pc|
00000070  6f 6e 71 75 65 73 74 73  3f 08 24 89 13 6b 47 41  |onquests?.$..kGA|
00000080  4d 45 50 16 07 af 8e 05  df 81 04 59 29 02 9d 93  |MEP........Y)...|
00000090  8f 40 0b 11 04 1c 44 8e  16 05 01 01 05 00 10 1e  |.@....D.........|
000000a0  02 16 04 0d 00 01 40 80  00 1a 44 0d 08 44 80 0d  |......@...D..D..|
000000b0  44 22 a0 10 40 0f 80 40  4c 70 03 40 20 60 67 58  |D"..@..@Lp.@ `gX|
000000c0  41 04 01 40 4d 48 02 40  23 40 44 0e 40 98 98 50  |A..@MH.@#@D.@..P|
000000d0  98 03 80 24 08 46 50 02  45 fe 22 be 11 0f 04 20  |...$.FP.E.".... |
000000e0  40 20 20 40 88 90 1d 02  20 60 20 20 02 24 00 20  |@  @.... `  .$. |
000000f0  20 03 6c fd fe 02 29 e9  40 11 23 2c a1 70 80 80  | .l...).@.#,.p..|
00000100  4f 58 d8 6f 50 ef 03 67  60 21 74 26 00 20 30 60  |OX.oP..g`!t&. 0`|
00000110  82 11 43 03 4d 11 97 3d  30 60 2e 41 41 47 24 71  |..C.M..=0`.AAG$q|
00000120  64 84 64 45 81 84 98 42  a4 25 44 41 54 45 54 50  |d.dE...B.%DATETP|
00000130  09 7e a4 8b a3 61 13 01  10 07 62 00 37 05 50 4c  |.~...a....b.7.PL|
00000140  47 49 08 60 f0 10 5e 8b  4e 53 4c e4 2c 2c 01 01  |GI.`..^.NSL.,,..|
00000150  01 00 94 1f 67 05 3b 3b  2d 07 2a 78 57 52 4c 44  |....g.;;-.*xWRLD|
...
00002670  03 02 20 26 18 1e 43 55  4c 54 10 2f 05 45 53 50  |.. &..CULT./.ESP|
00002680  4e 20 64 03 21 02 15 4c  45 41 44 9c 15 18 b2 a5  |N d.!..LEAD.....|
00002690  77 32 0a 91 6d 07 bb 40  1a 25 11 17 0a 37 0a 01  |w2..m..@.%...7..|
000026a0  02 1d 01 06 09 fc c1 40  8b ba a2 40 f2 05 a6 42  |.......@...@...B|
000026b0  ff ff 04 02 01 01 01 04  01 03 26 1e 43 55 4c 54  |..........&.CULT|
000026c0  10 7b 05 45 53 50 4e 20  64 02 2b 02 1a 4c 45 41  |.{.ESPN d.+..LEA|
000026d0  44 9c 15 50 f1 e6 b7 e2  30 05 ff ff ff 04 02 43  |D..P....0......C|
...
000042a0  59 59 59 95 42 63 1a 92  91 d9 bb 29 34 29 31 37  |YYY.Bc.....)4)17|
000042b0  46 34 46 37 37 34 f3 3e  43 2b 7f a3 10 ce 76 02  |F4F774.>C+....v.|
000042c0  62 14 01 0e 01 c8 6e 67  65 20 70 75 62 6c 69 63  |b.....nge public|
000042d0  6c 79 20 72 65 6e 6f 75  6e 63 65 64 20 68 69 6f  |ly renounced hio|
000042e0  79 61 6c 74 e7 6d 65 65  65 20 15 00 25 03 c0 c0  |yalt.meee ..%...|
000042f0  14 65 24 72 6c 64 00 61  6e 64 03 07 07 01 61 74  |.e$rld.and....at|
00004300  65 69 38 0c 0a 45 6e 74  72 65 6d 6f 20 86 20 54  |ei8..Entremo . T|
00004310  70 68 06 07 48 61 70 70  79 20 4c 61 62 6f fc 31  |ph..Happy Labo.1|
00004320  52 c3 43 a1 11 22 82 88  b8 8a 78 54 78 88 da da  |R.C.."....xTx...|
...
00004bc0  f4 9c 1a be e2 24 8c 77  84 80 77 98 85 f9 f8 4c  |.....$.w..w....L|

End of stream token reached

Edit: Minor text fixes. Also, while the code repo is updated, my executables linked a couple of posts ago aren't.

Edit 2: Also, it turned out to be trivial to support all three dictionary sizes, so it does now. It will (sort-of at the moment) decode files beginning with 0x0004, 0x0005 or 0x0006. But The first byte still has to be 0x00 and not 0x01, but that appears to be fine for Civ3.

 

[Puppeteer]

For the decompressor utility, of course. The seed reader utility and later BIQ/map/game-data utility will all be able to use the same code package to read the file but will have different outputs.

Oh, after looking at the cli scaffolding I had an epiphany. These don't need to be separate utilities! They all do the same thing: read a Civ3 file and produce info, all of which falls under the thematic umbreall of "Civ3 Show-And-Tell". The command line will be either c3sat.exe or civ3sat.exe and have a number of subcommands, such as info (including seed, difficulty, map settings), hexdump and decompress.

Later I'll include various empire reports and map extraction commands, and perhaps have even a daemon mode that will monitor folders for new/modified save files and have a local web interface for the output (optionally).