Civ3 Show-And-Tell

[Puppeteer]

I just successfully loaded a save game file that was decompressed using my code.

I ran into a couple of snags:

- I had to negate the offset for the seek call
- I had to subtract one from the negated seek, because .Seek(0,2) puts the read pointer after the last byte, but for coded offset 0 I want to read the last byte, so the pointer needs to back up one extra byte first, so .Seek(-1-offset,2).

After that it looked like it might be working and indeed worked on a small test case spelled out in the specification. But from experience I could see the hex dump looked a little suspicious, and sure enough Civ3 crashed when trying to load it.

I was worried it was going to be hard to figure, then I recalled a special case I hadn't coded in: "There is one exception to this: if the copy length is equal to 2, then only two low-order offset bits are encoded instead of 4, 5, or 6." Once I coded that exception in, things looked more normal and Civ3 loaded the resulting decompressed save file.

The hard-to-figure part is done, but now I need to refactor the decompression logic into a package which might take a little time.

Sample output:

Spoiler :

000df310  52 50 4c 45 0a 00 00 00  06 01 14 00 08 00 03 ff  |RPLE............|
000df320  00 00 00 52 50 4c 45 0a  00 00 00 06 01 16 00 08  |...RPLE.........|
000df330  00 03 ff 00 00 00 52 50  4c 45 0a 00 00 00 06 01  |......RPLE......|
000df340  13 00 09 00 03 ff 00 00  00 52 50 4c 45 0a 00 00  |.........RPLE...|
000df350  00 06 01 15 00 09 00 03  ff 00 00 00 52 50 4c 45  |............RPLE|
000df360  0a 00 00 00 06 01 17 00  09 00 03 ff 00 00 00 52  |...............R|
000df370  50 4c 45 0a 00 00 00 06  01 14 00 0a 00 03 ff 00  |PLE.............|
000df380  00 00 52 50 4c 45 0a 00  00 00 06 01 16 00 0a 00  |..RPLE..........|
000df390  03 ff 00 00 00 52 50 4c  45 0a 00 00 00 06 01 15  |.....RPLE.......|
000df3a0  00 0b 00 03 57 00 00 00  52 50 4c 45 0a 00 00 00  |....W...RPLE....|
000df3b0  04 01 15 00 09 00 00 ff  d0 d0 44 61 72 68 61 6e  |..........Darhan|
000df3c0  00 52 50 4c 54 05 00 00  00 02 01 00 00 00 0b 00  |.RPLT...........|
000df3d0  00 00 52 50 4c 45 0a 00  00 00 06 01 0f 00 07 00  |..RPLE..........|
000df3e0  03 2e 00 00 00 52 50 4c  45 0a 00 00 00 06 01 0e  |.....RPLE.......|
000df3f0  00 08 00 03 65 00 00 00  52 50 4c 45 0a 00 00 00  |....e...RPLE....|
000df400  06 01 10 00 08 00 03 2f  00 00 00 52 50 4c 45 0a  |......./...RPLE.|
000df410  00 00 00 06 01 0d 00 09  00 03 2f 00 00 00 52 50  |........../...RP|
000df420  4c 45 0a 00 00 00 06 01  0f 00 09 00 03 2e 00 00  |LE..............|
000df430  00 52 50 4c 45 0a 00 00  00 06 01 11 00 09 00 03  |.RPLE...........|
000df440  2f 00 00 00 52 50 4c 45  0a 00 00 00 06 01 0e 00  |/...RPLE........|
000df450  0a 00 03 2e 00 00 00 52  50 4c 45 0a 00 00 00 06  |.......RPLE.....|
000df460  01 10 00 0a 00 03 6e 00  00 00 52 50 4c 45 0a 00  |......n...RPLE..|
000df470  00 00 06 01 0f 00 0b 00  03 ff 00 00 00 52 50 4c  |.............RPL|
000df480  45 0a 00 00 00 06 01 10  00 0c 00 00 ff 00 00 00  |E...............|
000df490  52 50 4c 45 0a 00 00 00  04 01 0f 00 09 00 00 6b  |RPLE...........k|
000df4a0  50 fc 4d 61 6e 64 61 6c  67 6f 76 69 00 00 00 00  |P.Mandalgovi....|
000df4b0  00 50 45 45 52 18 00 00  00 00 00 00 00 00 00 00  |.PEER...........|
000df4c0  00 00 00 00 00 ff ff ff  ff 00 00 00 00 00 00 00  |................|
000df4d0  00                                                |.|

 

[Puppeteer]

Here you go. Executables for Windows, Darwin (Mac) and Linux that can decompress a SAV file (or a BIQ/BIX/BIC) or hex dump it.

http://lib.bigmoneyjim.com/civfan/gocrosscompile/index.html

It still has rough edges, but it should be usable to others right now.

Rough edges to be fixed soon-ish:
- Will currently exit for uncompressed files
- decompress command only creates--or overwrites!--out.sav (even if a BIQ file is the source) in current folder with no options to change it
- Will gladly try to execute without a filename and then error out
- hexdump will only write to screen/stdout, and it's a lot of data--most likely too much to fit in your scrollback buffer. (but you can redirect to file e.g. > out.txt)

I've tested the Windows and Linux versions. I don't have a Mac.

Output with no arguments:

Spoiler :

> .\civ3sat-windows-386.exe
NAME:
   Civ3 Show-And-Tell - A utility to extract data from Civ3 SAV and BIQ files. Provide a file name of a SAV or BIQ file after the command.

USAGE:
   civ3sat-windows-386.exe [global options] command [command options] [arguments...]

VERSION:
   0.0.0

COMMANDS:
     decompress, d  decompress a Civ3 data file to out.sav in the current folder
     hexdump, h     hex dump a Civ3 data file to stdout
     help, h        Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Example output of decompressing:

Spoiler :

$ ./civ3sat-linux-386 decompress Mao%20of%20the%20Chinese,%20130%20AD.SAV
Mao%20of%20the%20Chinese,%20130%20AD.SAV opened
Compressed Civ3 file detected
Saved to out.sav in current folder

These files probably won't stay at the link for a long time. I expect I'll find a more permanent location from which to serve them once I smooth some of the rough edges and start adding functionality.

Edit: Oops, I just noticed the short 'h' alias for hexdump overrides the 'h' for help. Will fix in future version.

Edit 2: Oh, yeah, I'd appreciate it if people could test it out on a compressed SAV (basically *not* an autosave) and make sure the game can open the resulting out.sav file. Especially the Mac/darwin version. Thanks!

Edit 3: Aside from fixing the rough edges, this is the foundation for adding other commands like info or seed or map or similar, and eventually having the utility seamlessly pull info from both compressed and uncompressed SAV and BIQ files. The decompression should work on any Civ3 version (Vanilla/PTW/C3C at various patch levels). So far the data extraction has only worked on C3C, but I hope to eventually read PTW, too.

 

[Puppeteer]

Some save-file decoding thoughts:

Antal1987's GameData.h looks to be the information and order that is in the save files, and additional headers (.h) for various types are under the Civ3/ folder. This looks like it should be incredibly helpful in starting over and parsing all the game data.

What I have been calling "sections" of the save file in this thread so far are apparently C++ classes. Most of the classes seem to be descendants of a base class That has a 4-char class name followed by a count of either bytes or array records which has the effect of making the save file look like a list of named sections. But there is some non-class data outside of the "sections", too, so not all the data in an apparent selection necessarily belongs to the "section name".

I've also looked through Antal1987's header files several times and have yet to spot anything looking like the save game file header that starts with CIV3 and ends with the beginning of the BIC data. This now makes me believe that there is no game-pertinent data there, and I'm guessing it probably identifies some meta data like the version of Civ3 that saved the file, and perhaps--although unlikely--PC-specific data. IIRC, it's the same length on every file I've seen, so decoding this will involve comparing across saves.

Edit: The header file code bits that I think show the save game top-level format. Actually the last few items after class_City I don't think are in the save but are functions/methods, but the WrapX() and WrapY() items caught my eye. That's definitely info I want. Now I'm wondering if/how class methods would be in the save file, because there is definitely info in the save file after the CITY sections/instances.

Spoiler :

public:
	Game_Data(Civ3_Functions * Functions) : F(Functions) {}
	~Game_Data(){}

	class_BIC * Get_BIC();
	class_Map * Get_Map();
	class_Map_Renderer * Get_Map_Renderer();
	class_Leader * Get_Civ(int CivID);
	class_Game * Get_Game();
	class_Race * Get_Races();
	class_Race * Get_Race(int RaceID);
	class_Base_List * Get_Cities();
	class_City * Get_City(int CityID);
	class_Base_List * Get_Units();
	class_Unit * Get_Unit(int UnitID);
	struct_UnitType * Get_Unit_Type(int UnitID);
	struct_UnitType * Get_Unit_Type_Object(int Unit_TypeID);

	std::string Get_Unit_Name(class_Unit * Unit);
	std::string Get_Civ_Name(int CivID);
	std::string Get_Civ_Name(class_Leader * Civ);
	class_City * Get_Civ_Capital(class_Leader * Civ);
	bool WrapX();
	bool WrapY();
	void Correct_Coordinates(int * p_X, int * p_Y);
	void Get_Direction_Coordinates(int X, int Y, int Direction, int * p_X, int * p_Y);
	void Get_Unit_Direction_Coordinates(class_Unit * Unit, int Direction, int * p_X, int * p_Y);
private:
	Civ3_Functions * F;

Edit 2: Actually there isn't as much data after the City classes as I thought. I see CLNY, PALV, HIST, TUTR, RPLS, RPLT, RPLE and PEER. CLNY is colony data which is clearly part of the game data, but HIST and beyond is metadata like histogram, tutorial, replay and multiplayer info (?). (Not sure if I ever figured out what PALV is.) So I think the functions/methods shown aren't in the save file itself, and in fact the Wrap functions return game flag references. CLNY is easy to miss because so few games have them. The rest is metadata not necessary to play the game, so I'm guessing their apparent absence from Antal1987's data is due to his focus on live game mechanics. So I still think I'm good on using his stuff as a reference from the start of the BIC through the end of the cities info. That plus CLNY is all the pertinent game data. (But I want to figure out the metadata, too.)

 

[Puppeteer]

Rather than fix the output features of the utility I've focused on some internal refactoring and trying to decode the file.

I'm having trouble at the same spot as last time: from soon after the first "BIC " to "WRLD". Antal1987's dumps haven't been as helpful as I'd hoped so far, but now that the new program can seamlessly read compressed and uncompressed files, I can run the utility against many files at once.

Since the program has commands now, I have a "dev" or "z" command that does whatever hacky things I'm doing while the decompress and hexdump commands are unaffected and continue to work fine. I've been setting what I think is a common value, running against 151 saves and having it make output only if the given file is different. once I downloaded a couple of GOTMs (PTW saves), COTMs and a CCM SG, and each of those has had differences that helped me understand certain sections a little better.

One thing I've been wondering about for quite a while is weather a .biq file's contents is exactly the same as the embedded biq data. I'm about to find that out; I can already detect if it's a sav or a biq, so for the biq I can call the biq decoder directly where with a sav I can get the header info then call the biq decoder. Then run against a bunch of saves and biqs at the same time and see if they're the same or different.

Quintillus, some of my saves have BLDG after the BIQ headers and some go straight to GAME. Do you know offhand if there is an indicator/flag for when/why data would be present or skipped?

Edit: So far, the data in the BIC/X/Q file seems to be the same as the BIC portion of the save file. And I am able to compare several files at once including mixed BI* and SAV files. I've even thrown in the default .bic and .bix from vanilla and PTW to see where they diverge. I haven't found any of my PTW or vanilla saves yet, but I do have a couple of downloaded GOTM starting saves.

Edit 2: The BIC formats diverge at BLDG which has a different length in PTW vs. C3C. I'm going to have to go back to focusing on only C3C for now.

 

[Quintillus]

Quintillus, some of my saves have BLDG after the BIQ headers and some go straight to GAME. Do you know offhand if there is an indicator/flag for when/why data would be present or skipped?

The way I've always done it is to check if the first new section after the BIQ header is BLDG; if so, it has custom rules. If not, check for WCHR, which indicates a custom map (which may not carry over into SAV since the map will have changed during the game). The GAME (scenario properties) section then always follows, and the LEAD (custom player data section) optionally follows - with BIQs, I've detected that by seeing if the amount of data I've read after the GAME section equals the file size, but when it's embedded within a SAV you'll likely have to check to see whether the next four bytes are LEAD instead (if I tried that in a BIQ, I'd get an end-of-file exception when it didn't exist).

As far as I know there's no indicator in the file itself; simply the presence of absence of these headers indicates whether the section exists.

Focusing on C3C likely makes sense for now, since that's 90% of what's played nowadays. I added the various Vanilla/PTW versions (every one I could find) over the past year, and as a ballpark figure there's about 30 places where the code has to handle them differently (in the BIQ; probably a bit higher in the SAV). Not necessarily a lot differently, often just a couple additional fields, but it's probably better return on the time to focus on the Conquests 1.22 saves first and optionally support more once those are running. I unfortunately lost most of my Vanilla/PTW saves years ago, although I know there's still some on one of my hard drives... just not in my main Civ install location.

Fired up the Windows version on my latest Rise of Rome compressed SAV file, and the decompressed version loaded up just fine! My only Mac is a PowerPC one, which Go doesn't support yet, so I can't try that one... but the Windows one seems to be doing its job!

 

[Puppeteer]

Thanks for the info and for the testing.

I was convinced there must be an indicator, but through powers of deduction there are only 32 bytes where that info could be, and there is not as much as a bit that consistently is set based on whether or not there is a custom map or custom rules.

Perhaps the native code reads the file in much as us reverse-engineerers do, and that's why those 4-byte strings are there in the first place.

FLAV seems to be another "optional" section within the custom game rules between EXPR and GOOD. I was going to ask if there are others, but I've been thinking about how I'm going to refactor all this for utility, elegance and error handling, and I think I see how I can have it double-check itself while parsing and make it easier for me to decode it.

Also, now that I look, the BIQ format is pretty well documented on Apolyton, much better than the rest of the save.

 

[Quintillus]

I believe FLAV is present if and only if it is a Conquests scenario/save. It got tacked on to the end of the BIQ format since Vanilla/PTW didn't have it.

That FLAV was added at the end suggests to me that the Firaxis code does read the file in a set order - it looks for BLDG, then CTZN, then CULT, etc., and uses the strings as both a check, and where a section could differ, to see what section the next one is. It could also always read the 4-byte string and then run the code for that section. If that's the case, then one could swap section orders and it would run just fine, and the order would just be a convention. I'd be a bit surprised if this worked, but I don't think anyone's tried it, so I couldn't rule it out.

Apolyton's BIQ documentation is the best. Very useful when making anything that reads/edits BIQs.

 

[Virote_Considon]

FLAV = Flavours?

 

[Puppeteer]

I am also assuming the order is fixed. I hope we're right. I've spent many hours refactoring (from what I just coded in Go!) error handling, reading and internal API, but now I think I'm at a point where I can start making progress parsing the data again.

@Virote_Considon - Yeah, apparently they coded in "flavors" for buildings and one or two other object types...maybe units. But there doesn't seem to be any meaningful text in those sections, and I can't think of where flavors would appear in-game, so maybe it's an abandoned idea. But I still have to parse it so I can get to more pertinent data.

Edit: It's going slower than I thought. Meh. But after saying I assume the items are always in the same order, the .biq files that come with Conquests for "No civ traits" and "cheaper upgrades" have FLAV in a different position .

 

[Virote_Considon]

There is a Flavours tab in the editor, you can assign each civ any number of the seven available flavours, and the same with techs and buildings. It weights the AI's decision on whether to build/research something or not.

 

[Puppeteer]

Ok thanks. I didn't know that. Does it have any visibility in-game aside from AI behavior?

In the BIC portion of the saves, LEAD seems to be optional, too. I guess scenarios have LEAD and simple custom maps don't.

Curiously, a save from a custom map biq no longer has the custom map in the BICQ section but still has the LEAD section. (LK153-1100AD.SAV from LKCWorld-V3E.biq). It makes sense to me not to have the map in there twice, but not sure why LEAD is still needed.

I forget now if GAME is always the end section of the BIC or if I've seen LEAD after it. In any case, the second GAME appears to mark the end of the BIC section.

The second GAME is what gave me fits last time. It doesn't indicate its length or count. 0x350 is what would normally be the length or record count in all saves I've seen so far...oh, it's different in PTW saves, it's 0x1.

I should probably take a break from this and get other things done, but what are the odds of that happening?

Some recent output of me running it against several files with Powershell. This one helped me be sure LEAD is optional, my current code always parses up to the problematic GAME section, and that it's probably the end of all the BIC data:

Spoiler :

> ri .\civ3sat.exe; go build; "`n"; gci C:\temp\saves\ -File | % { .\civ3sat.exe z $PSItem.FullName; "`n" }


C:\temp\saves\about to win English, 1340 AD.SAV
File: C:\temp\saves\about to win English, 1340 AD.SAV   Compressed: true
GAME Name: GAME Count: 1
*** Debug output. Next bytes ***
00000000  47 41 4d 45 50 03 00 00  16 01 00 00 07 af 8e 05  |GAMEP...........|
00000010  df 81 04 00 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000020  59 00 00 00 29 00 00 00  01 00 00 00 00 00 00 00  |Y...)...........|
00000030  00 00 00 00 ff ff ff ff  ff ff ff ff 00 00 00 00  |................|



C:\temp\saves\cotm_121_open.sav
File: C:\temp\saves\cotm_121_open.sav   Compressed: true
LEAD Name: LEAD Count: 8
GAME Name: GAME Count: 1
*** Debug output. Next bytes ***
00000000  47 41 4d 45 50 03 00 00  33 02 00 00 ff ff ff 3f  |GAMEP...3......?|
00000010  1f 81 00 00 00 00 00 00  05 00 00 00 00 00 00 00  |................|
00000020  12 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 ff ff ff ff  ff ff ff ff 00 00 00 00  |................|



C:\temp\saves\gotm_149_ptw_open.sav
File: C:\temp\saves\gotm_149_ptw_open.sav       Compressed: true
LEAD Name: LEAD Count: 9
GAME Name: GAME Count: 1
*** Debug output. Next bytes ***
00000000  47 41 4d 45 01 00 00 00  28 15 00 00 00 00 00 00  |GAME....(.......|
00000010  00 00 00 00 18 00 00 00  01 00 00 00 02 00 00 00  |................|
00000020  03 00 00 00 04 00 00 00  05 00 00 00 06 00 00 00  |................|
00000030  07 00 00 00 08 00 00 00  09 00 00 00 0a 00 00 00  |................|



C:\temp\saves\gotm_150_ptw_open.sav
File: C:\temp\saves\gotm_150_ptw_open.sav       Compressed: true
LEAD Name: LEAD Count: 8
GAME Name: GAME Count: 1
*** Debug output. Next bytes ***
00000000  47 41 4d 45 01 00 00 00  28 15 00 00 01 00 00 00  |GAME....(.......|
00000010  01 00 00 00 18 00 00 00  01 00 00 00 02 00 00 00  |................|
00000020  03 00 00 00 04 00 00 00  05 00 00 00 06 00 00 00  |................|
00000030  07 00 00 00 08 00 00 00  09 00 00 00 0a 00 00 00  |................|

 

[Virote_Considon]

Ok thanks. I didn't know that. Does it have any visibility in-game aside from AI behavior?

Shouldn't do.

 

[Puppeteer]

Thanks.

I didn't fire up Spark, but I did combine a few tools to compare the non-BIC GAME section across Conquests saves from my own PC. I wanted to dump the GAME section and at least enough data to get to WRLD so I could compare different files with common offsets (0) and work with smaller amounts of data.

It took me longer than it should have to realize that my current progress already lands the file pointer right at GAME, so I just added a dump routine to it to dump 0x1200 (just over 4k, found by trial-and-error) from 12 different saves.

Then I ran strings against it to find the strings and offsets. Linux/unix has a strings command which is how I started this whole project, but I don't recall if it can show offsets. And at the time I didn't have automatic decompression and the easy ability to process multiple files quickly.

Then I took that output and did some Powershell-fu to determine the size of each section by subtracting offsets. The size includes the "GAME" header in this case.

Happily, the sizes of the other named sections between GAME and WLRD are consistently sized between saves.

Then I started wondering what makes the size different. World size? Number of civs? By sorting by size I spotted something unexpected:

FileName                                       Seq   Section   Length
Theodora of the Byzantines, 2470 BC.SAV.game   1     GAMEP     3839
Joan d'Arc of the French, 770 AD.SAV.game      1     GAMEP     3819
PPT-Huge-1-770 AD.SAV.game                     1     GAMEP     3819
about to win English, 1340 AD.SAV.game         1     GAMEP     3795
island Russians, 3200 BC.SAV.game              1     GAMEP     3791
Isabella of the Spanish, 2900 BC.SAV.game      1     GAMEP     3787
Tokugawa of the Japanese, 2350 BC.SAV.game     1     GAMEP     3787
Xerxes of the Persians, 2850 BC.SAV.game       1     GAMEP     3787
Wang Kon of the Koreans, 2850 BC.SAV.game      1     GAMEP     3783
Alexander of the Greeks, 3200 BC.SAV.game      1     GAMEP     3779
PPTTemujin of the Mongols, 170 AD.SAV.game     1     GAMEP     3779
Temujin of the Mongols, 1000 BC-redo.SAV.game  1     GAMEP     3779
Gandhi of the Indians, 3050 BC.SAV.game        1     GAMEP     3767

("GAMEP" is because the 0x50 value following game happens to be an ASCII value, so the strings utility thinks it's part of the string.)

While there are some larger maps near the top (you'll have to trust me), "about to win English" is a smaller than normal map, but late-game. So something about later-stage games would seem to make this section larger.

But so might map size. I need to load up the top one because it's relatively early game and I don't recall the map size. And the 170AD Mongols one which is later in the game but lower down the list in size.

So I think this game.h from Antal1987's dumps will be my guide on where to look to find the variable-length data. Although I don't see offhand any odd byte values that would cause all the lengths to be odd. And I'm not seeing any variable-length arrays until the Base_Lists at the bottom, but I think those are references to the actual map data and not in this ~4k I'm looking at. Oh well, it's a starting point, and I can see if I can match any of the field meanings.

Edit: The Mongol 170AD game is a normal map with 8 opponents. The Theodora save is a larger map.

 

[Puppeteer]

Oh well, it's a starting point, and I can see if I can match any of the field meanings.

I carefully set up a struct to match Antal1987's info, and at first it didn't look even close. Then I spotted 20000 and 100000 next to each other . Those have to be the culture win limits. They were close but didn't align quite right. Then I realized I missed one field, mis-typed another ([3]int32 instead of int32) and accounted for the Base class header having a length field even if it doesn't seem to be used here. After those adjustments (which shifted everything 3 or 4 fields depending on placement) The numbers look right!

DifficultyLevel:4
UnitsCount:79
CitiesCount:15
GlobalWarmingLevel:0
CurrentTurn:23
Random:142874715
CivFlags2:2
CivFlags1:511
GameLimitPoints:50000
GameLimitTurns:540
GameLimitDestroyedCities:1
GameLimitCityCulture:20000
GameLimitCivCulture:100000
GameLimitPopulation:66
GameLimitTerritory:66
GameLimitWonders:10
GameLimitDestroyedWonders:5
GameLimitAdvances:5
GameLimitCapturedCities:100
GameLimitVictoryPointPrice:25
GameLimitPrincessRansom:1000
DefaultDate1:9

(Heavily reformatted as I don't have it pretty yet. This is from one of my epic games; I presume it's standard size Emperor difficulty.)

My Go struct for the data so far:

Spoiler :

type Game struct {
	Name [4]byte
	_    int32
	// Above two fields count for "class base"?
	_                  [3]int32
	RenderFlags        int32
	DifficultyLevel    int32
	_                  int32
	UnitsCount         int32
	CitiesCount        int32
	_                  int32
	_                  int32
	GlobalWarmingLevel int32
	_                  int32
	_                  int32
	_                  int32
	CurrentTurn        int32
	_                  int32
	Random             int32
	_                  int32
	CivFlags2          int32
	CivFlags1          int32
	_                  int32
	_                  int32
	_                  int32
	_                  int32
	_                  int32
	_                  int32
	_                  [48]int32
	Value1             int32
	_                  [72]int32
	GameLimitPoints    int32
	GameLimitTurns     int32
	_                  [50]int32
	_                  int32
	_                  int32
	// this is 1... I guess it means the population at/under which cities are destroyed instead of captured?
	GameLimitDestroyedCities   int32
	GameLimitCityCulture       int32
	GameLimitCivCulture        int32
	GameLimitPopulation        int32
	GameLimitTerritory         int32
	GameLimitWonders           int32
	GameLimitDestroyedWonders  int32
	GameLimitAdvances          int32
	GameLimitCapturedCities    int32
	GameLimitVictoryPointPrice int32
	GameLimitPrincessRansom    int32
	DefaultDate1               int32
	_                          [27]int32
}

I haven't made it through the whole section yet, but to have Antal1987's data match up is some fantastic progress. I think I have the world random seed right there; I might be able to add that extractor soon. But sleep for now.

 

[Puppeteer]

Very busy so far this week with non-c3sat stuff. I kind of want to get a couple of improvements published as executables.

I just tested the random number field I have identified, but it's not the world generation seed. That must be the "preserve random seed" value I'm guessing.

 

[Puppeteer]

I looked a little more at the mysterious GAME section again. Antal1987's dump matches well up until the princess ransom field. After that it doesn't seem to make any sense.

I punted a little by reading the next 4KB or so, searching for the next section DATE, stuffing the intervening data into a hex dump titled "WTF" for later analysis, and then the subsequent sections seem to be easily readable with two extra int32's thrown in.

That reliably gets me to WRLD which is where my Python program begins, so I know I can parse a long way from there. So I can load data from the beginning of the file to the starting point of my Python with skipping-but-capturing a variable bit of data that seems to be around 3000 bytes.

So aside from lack of time I'm pretty close to mechanically parsing the file with a minor cheat for now. I need to work on extracting meaning, but with Antal1987's dumps (which aren't perfect for my purposes but are still quite instructive) and my previous analysis the meaning should come pretty quickly.

 

[Puppeteer]

The Go version can now output a the JSON map, and with minor text fixes modifications it is now viewable via d3.html. So it's at least at feature parity with the last screenshot I had.

Next steps: sleep. Then, who knows?

- Map features wanted: should now be able to show strat resources with proper spoiler avoidance (or soon when I figure out where the advances-per-civ are stored). Control over display features. Civ and border indicators. City names and stats. Units. Pop-out window for improvements and other tile detail.
- Tool features wanted: better output control for decompress, hex dump and JSON map. World seed extractor (trivial at this point). Info dump (turns, civ count, map size, difficulty, map settings, etc..). Perhaps output single-page html maps or have as option. Perhaps have integrated server to use local web browser. Summary reports (city/unit count, etc.)
- Stuff to parse: LEAD is next, then all the other stuff I never got to.
- Pie-in-the-sky: monitor save game folders, auto-parse new saves. Take actions...perhaps make map, make report, trigger outside content.

 

[Puppeteer]

Edit: The answer is that you can select random features, so the "final" value is the random choice, but the user selection was "random". Interesting.

Question: Does anyone know why there are world options and "final" world options? I'm in the process of getting the world options needed to go along with the world see to recreate the map. I definitely found the place, and it looks like it might match up with this dump, and sure enough when I change the barb settings, two numbers change. What's the difference between the final number and the normal one?

struct struct_World_Features
{
  int World_Aridity;
  int Final_World_Aridity;
  int Barbarian_Activity;
  int Final_Barbarians_Activity;
  int World_Landmass;
  int Final_World_Landmass;
  int Ocean_Coverage;
  int Final_Ocean_Coverage;
  int World_Temperature;
  int Final_World_Temperature;
  int World_Age;
  int Final_World_Age;
  int World_Size;
  int Continents_Data;
  int Tiles_Data;
};

Since my immediate purpose is the info needed to re-generate a particular map I guess I should go with the non-final value as I infer the first value is the one chosen at gen time.

 

[Puppeteer]

I found it curious there is no "final" world size, and the size and nature of the data seems to agree. So I tried another random-sized map with all other settings the same and the same world seed, and sure enough the random size was different.

So if you generate a randomly-sized map and save the seed and want to regenerate it, you have to next time pick the right size to get the same map. This is backwards from the rest of the map page settings where you have to leave it at random if it was random before.

 

[Puppeteer]

I'm nearly ready to put this in the utilities and tools section. If any readers have a chance to try it out and make sure the map outputs are correct and usable I'd appreciate it.

http://lib.bigmoneyjim.com/civfan/gocrosscompile/civ3sat-windows-386.exe
http://lib.bigmoneyjim.com/civfan/gocrosscompile/civ3sat-darwin-386
http://lib.bigmoneyjim.com/civfan/gocrosscompile/civ3sat-linux-386

It can now:

- Pull world seed and world settings needed to regenerate a generated map
- Decompress a sav or biq to out.sav in the current folder
- Hex dump to stdout a sav or biq (with automatic decompression if needed)
- Create civmap.json usable with d3.html in my code repo (not user-friendly yet)

The utility has native versions for Windows, Darwin and Linux.

Example seed:

> .\civ3sat-windows-386.exe seed 'C:\temp\saves\about to win English, 1340 AD.SAV'
C:\temp\saves\about to win English, 1340 AD.SAV

Setting         Choice          Result
World Seed      492735
World Size      Tiny
Barbarians      Roaming         Roaming
Land Mass       Continents      Continents
Water Coverage  70% Water       70% Water
Climate         Normal          Normal
Temperature     Temperate       Temperate
Age             4 Billion       4 Billion

Another seed against a file that was all random. As noted earlier, to regen the same map you have to pick the world size even if it was originally random. But the rest of the values, if the original gen was random you have to pick random again to get the same map.

> .\civ3sat-windows-386.exe seed "C:\Program Files (x86)\Steam\steamapps\common\Sid Meier's Civilization III Complete\Conquests\Saves\Auto\Conquests Autosave 4000 BC.SAV"
C:\Program Files (x86)\Steam\steamapps\common\Sid Meier's Civilization III Complete\Conquests\Saves\Auto\Conquests Autosave 4000 BC.SAV

Setting         Choice          Result
World Seed      156059380
World Size      Huge
Barbarians      Random          No Barbarians
Land Mass       Random          Archipelago
Water Coverage  Random          70% Water
Climate         Random          Arid
Temperature     Random          Warm
Age             Random          3 Billion

Here is the help screen.

NAME:
   Civ3 Show-And-Tell - A utility to extract data from Civ3 SAV and BIQ files. Provide a file name of a SAV or BIQ file after the command.

USAGE:
   civ3sat-windows-386.exe [global options] command [command options] [arguments...]

VERSION:
   0.3.0

COMMANDS:
     seed, s        Show the world seed and map settings needed to generate the map, if this map was randomly generated.
     decompress, d  decompress a Civ3 data file to out.sav in the current folder
     hexdump, x     hex dump a Civ3 data file to stdout
     map, m         Dump a JSON file of map data to civmap.json in the current folder
     dev, z         Who knows? It's whatever the dev is working on right now
     help, h        Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version