Civ3 Show-And-Tell

I now want to remove my dev/z command and even the map/m command before a "release'. Which led me to want to start using git branches which led me to discover I have a develop branch whose progress was abandoned. It apparently was reading LEAD and later sections. Note to self: this commit is the head of the abandoned branch. Aside from parsing more data (in Python) it also grabbed some file documentation for somewhere that I haven't looked at in a year or two.

Hmmm. Rather than try to merge the abandoned develop branch I think I'll base another branch off of it and delete/recreate develop.

Off to do some git reorganizing...
 
I cleaned up and released civ3sat v0.3.1.

I created an abandonedpydevelop branch to mark the abandoned branch. Then I deleted and recreated develop, deleted the Python files and moved non-deploy folders and files into a develop/ directory. Then I created an 0.3.1 branch where I deleted develop/, some dev code and the command line options dev/z and map/m and pulled it into master. Git branching #FTW.

Basically, the previous paragraph means I have a relatively clean release product and code, meaningful references to the abandoned effort and the current release, and a current dev area to move forward without messing up the release code.

The progress in the abandoned develop branch is kind of useless in my first review. It wasn't pulling data, just mechanically reading blocks or skipping unknown data. There was also a reference to another sav file analysis, but it's very limited, convoluted and incomplete. This file might be useful to refer to, but instead of copying into my code I just forked the repo.

Review time: Well, I've learned quite a bit about Go, and I even picked up some more on git branching strategy. And I thought I had more to say on this topic, but I guess not.

What's next? I think I want to continue changing strategy to reading in structs. Doing it this way, I think it will be relatively easy to write a file back out again, but it will take enough effort that I won't do it unless and until I am reading the entire file in and have a reason to want to write.

I also want to redo my JSON map output so the Go code handles the spoiler filters and makes friendlier-to-read JSON. I want to add features, but before I do that I need to settle how I'm reading and storing the game data.

I want to improve the output handling of the command line, but that's pretty low priority.

I want the JSON map to eventually be somehow usable; I might have to spit out a massive html file with the map data and d3 code wrapped inside it.

For helping decode unknown fields I'll probably code some sort of side-by-side comparison ability, although that might be a separate program or possibly even a mini web server so I can have really wide displays. Or maybe I can just dump to CSV and compare in a spreadsheet.
 
For helping decode unknown fields I'll probably code some sort of side-by-side comparison ability, although that might be a separate program or possibly even a mini web server so I can have really wide displays. Or maybe I can just dump to CSV and compare in a spreadsheet.

I briefly took a stab or two at this, but the differing data structures in a strongly-typed language made it difficult to do at all, much less to easily change the structure I'm looking at. I briefly wished I could manipulate the data like I could Powershell objects; after all I'm launching from Powershell and using PS to repeatedly execute civ3sat against numerous files.

Then I briefly tried to use Powerhsell to manipulate the output, but the easy-format printing of a struct from Go looks like this, which isn't easy to visually compare:

Code:
parseciv3.Civ3{Name:[4]uint8{0x43, 0x49, 0x56, 0x33}, A:0x181a00, B:0xa0000, C:0xa2ca0000, D:0xc73e533d, E:0xfe9547bb, F:0
x35127dc8, G:0x1061}

Then a clue stick fell on my head as I realized I could output the struct as JSON and then Powershell (or many other tools) can handle the iteration and formatting of the data, and the code doesn't need to change between struct types!

So now I can view like this, in CSV, or any number of permutations with various scripting and data tools. (And here I've even added via Powershell the base name of the file.)

Code:
Name     : {67, 73, 86, 51}
A        : 1579520
B        : 655360
C        : 970915840
D        : 226181814
E        : 1385647014
F        : 694160264
G        : 30150
FileName : Xerxes of the Persians, 2850 BC.SAV

Via an html-to-bbcode converter, a comparison table: ok, for some reason a table conversion is proving too difficult to bother. It's cool. Trust me.

Edit. Oh FFS. Threre is an HTML tag, but it highlights instead of render. :p Oh well, you can get an idea maybe:
Spoiler :
HTML:
<table>
<tr><th>Name</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>FileName</th></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>494534656</td><td>3391608470</td><td>1990347604</td><td>1568788630</td><td>63258</td><td>about to win English, 1340 AD.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>2082996224</td><td>3737146167</td><td>230575774</td><td>2497599718</td><td>30410</td><td>Alexander of the Greeks, 3200 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>1939865600</td><td>768679833</td><td>145641313</td><td>1255074369</td><td>27283</td><td>cotm_120_open.sav</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>16515072</td><td>537881997</td><td>1502956036</td><td>214430892</td><td>19372</td><td>cotm_121_open.sav</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>1217331200</td><td>2198151251</td><td>3917496705</td><td>3238901537</td><td>9014</td><td>Gandhi of the Indians, 3050 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>810352640</td><td>2717116138</td><td>3768732351</td><td>1909350322</td><td>47330</td><td>Isabella of the Spanish, 2900 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>1760559104</td><td>767887307</td><td>1234387303</td><td>2153661895</td><td>3036</td><td>island Russians, 3200 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>3999662080</td><td>1923868211</td><td>3736290733</td><td>1223456244</td><td>57803</td><td>Joan d'Arc of the French, 770 AD.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>1813774336</td><td>883246907</td><td>3316927498</td><td>611791203</td><td>40135</td><td>PPT-Huge-1-770 AD.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>3072000000</td><td>3595175410</td><td>2075739005</td><td>688430265</td><td>36511</td><td>PPTTemujin of the Mongols, 170 AD.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>2754871296</td><td>1892514423</td><td>1318798470</td><td>3999322574</td><td>62624</td><td>Temujin of the Mongols, 1000 BC-redo.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>2731147264</td><td>3342750525</td><td>4271196091</td><td>890404296</td><td>4193</td><td>Theodora of the Byzantines, 2470 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>927989760</td><td>4288629337</td><td>2712486103</td><td>3405451185</td><td>17941</td><td>Tokugawa of the Japanese, 2350 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>24051712</td><td>3551689832</td><td>1605255774</td><td>3979776023</td><td>52155</td><td>Wang Kon of the Koreans, 2850 BC.SAV</td></tr>
<tr><td>System.Object[]</td><td>1579520</td><td>655360</td><td>970915840</td><td>226181814</td><td>1385647014</td><td>694160264</td><td>30150</td><td>Xerxes of the Persians, 2850 BC.SAV</td></tr>
</table>

Edit 2: So this is the CIV3 header section at the beginning of the file, which I'm currently interpreting as 6 32-bit integers and a 16-bit integer as G. The first two numbers are consistent, then there seem to be no similarities between files, so I'm now thinking the third number might be a 16- or 8-bit value, so I'll try those and see if the other numbers start making sense. or maybe I'll make them all 16-bit numbers and combine where sensible.
 
Ah, I see now with the help of RTFM. A table here is basically a CSV except with pipes (|) instead of commas.

Here is the CIV3 header's data with each column being a byte.

Spoiler :
A|B|C|D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z|FileName
0|26|24|0|0|0|10|0|0|0|122|29|150|214|39|202|84|75|162|118|150|208|129|93|26|247|about to win English, 1340 AD.SAV
0|26|24|0|0|0|10|0|0|0|40|124|55|83|192|222|158|78|190|13|230|88|222|148|202|118|Alexander of the Greeks, 3200 BC.SAV
0|26|24|0|0|0|10|0|0|0|160|115|153|31|209|45|97|79|174|8|65|234|206|74|147|106|cotm_120_open.sav
0|26|24|0|0|0|10|0|0|0|252|0|141|109|15|32|4|74|149|89|172|244|199|12|172|75|cotm_121_open.sav
0|26|24|0|0|0|10|0|0|0|143|72|83|32|5|131|129|65|128|233|33|183|13|193|54|35|Gandhi of the Indians, 3050 BC.SAV
0|26|18|0|0|0|7|0|0|0|218|50|125|227|189|237|83|73|173|54|255|60|26|94|36|21|gotm_149_ptw_open.sav
0|26|18|0|0|0|7|0|0|0|41|168|46|67|68|29|173|73|181|130|34|13|4|12|239|120|gotm_150_ptw_open.sav
0|26|24|0|0|0|10|0|0|0|77|48|234|230|243|161|191|74|162|224|178|95|206|113|226|184|Isabella of the Spanish, 2900 BC.SAV
0|26|24|0|0|0|10|0|0|0|240|104|203|7|197|45|103|65|147|73|199|69|94|128|220|11|island Russians, 3200 BC.SAV
0|26|24|0|0|0|10|0|0|0|102|238|51|230|171|114|173|69|179|222|244|117|236|72|203|225|Joan d'Arc of the French, 770 AD.SAV
0|26|24|0|0|0|10|0|0|0|28|108|59|71|165|52|10|76|180|197|99|49|119|36|199|156|PPT-Huge-1-770 AD.SAV
0|26|24|0|0|0|10|0|0|0|27|183|242|5|74|214|125|67|185|123|185|156|8|41|159|142|PPTTemujin of the Mongols, 170 AD.SAV
0|26|24|0|0|0|10|0|0|0|52|164|119|122|205|112|134|68|155|78|206|209|96|238|160|244|Temujin of the Mongols, 1000 BC-redo.SAV
0|26|24|0|0|0|10|0|0|0|202|162|61|83|62|199|187|71|149|254|200|125|18|53|97|16|Theodora of the Byzantines, 2470 BC.SAV
0|26|24|0|0|0|10|0|0|0|80|55|89|74|159|255|215|64|173|161|177|15|251|202|21|70|Tokugawa of the Japanese, 2350 BC.SAV
0|26|24|0|0|0|10|0|0|0|111|1|104|124|178|211|94|66|174|95|23|144|54|237|187|203|Wang Kon of the Koreans, 2850 BC.SAV
0|26|24|0|0|0|10|0|0|0|223|57|182|66|123|13|166|75|151|82|136|11|96|41|198|117|Xerxes of the Persians, 2850 BC.SAV

Curiously, 26 is the length of the data in bytes including the first two bytes. But I think that's a coincidence as everywhere else a length is given it refers to the following data and not itself.

Clearly C is different for the two PTW games as is column G. I wonder if this is a float indicating the version number...no, PTW final is I think 1.27 where C3C is 1.22 . In any case there is some versioning info here.

Curiously nothing else in the header after column J seems to have consistency except maybe column R which has a relatively narrow range of values. Column S is also mildly less wild than the other columns.

Anyway, this is an example of how I can analyze the data across saves. In this particular case I would have to guess that the first few bytes indicate the version of civ that saved the game, and much of the rest of the data is some sort of hash or something, perhaps to protect from people editing themselves massive amounts of gold or maybe just a simple checksum for data integrity.

Edit: Also, now that I look at it, I think the data is laid out as a 16-bit int, then two 32-bit ints, then 16 bytes of gobbledygook. In fact, I think I'll make it a 16-byte byte array named Gobbledygook. :D

Edit 2: 26 is also the count of resource types in a normal game. Someday I'll run a save with a scenario with a different number of resources and see if this number changes.

Edit 3: Mesoamerica conquest has 21 resources. Started/saved a game. Number is still 26.

Edit 4: Gobbledygook:
Spoiler :
Always0x1a00|MaybeVersionMinor|MaybeVersionMajor|Gobbeldygook1|Gobbeldygook2|Gobbeldygook3|Gobbeldygook4|FileName
6656|24|10|3600162170|1263847975|3499521698|4145700225|about to win English, 1340 AD.SAV
6656|24|10|1396145192|1319034560|1491471806|1992987870|Alexander of the Greeks, 3200 BC.SAV
6656|24|10|530150304|1331768785|3930130606|1788037838|cotm_120_open.sav
6656|24|10|1837957372|1241784335|4104935829|1269566663|cotm_121_open.sav
6656|24|10|542328975|1099006725|3072452992|590790925|Gandhi of the Indians, 3050 BC.SAV
6656|18|7|3816633050|1230237117|1023358637|354704922|gotm_149_ptw_open.sav
6656|18|7|1127131177|1236081988|220365493|2028932100|gotm_150_ptw_open.sav
6656|24|10|3874107469|1254072819|1605558434|3101848014|Isabella of the Spanish, 2900 BC.SAV
6656|24|10|130771184|1097280965|1170688403|199000158|island Russians, 3200 BC.SAV
6656|24|10|3862163046|1168994987|1978982067|3788196076|Joan d'Arc of the French, 770 AD.SAV
6656|24|10|1195076636|1275737253|828622260|2630296695|PPT-Huge-1-770 AD.SAV
6656|24|10|99792667|1132320330|2629401529|2392795400|PPTTemujin of the Mongols, 170 AD.SAV
6656|24|10|1636723|1178301834|170507433|655230052|Smoke-Jaguar of the Maya, 300 AD.SAV
6656|24|10|2054661172|1149661389|3519958683|4104187488|Temujin of the Mongols, 1000 BC-redo.SAV
6656|24|10|1396548298|1203488574|2110324373|274806034|Theodora of the Byzantines, 2470 BC.SAV
6656|24|10|1247360848|1087897503|263299501|1175833339|Tokugawa of the Japanese, 2350 BC.SAV
6656|24|10|2087190895|1113510834|2417450926|3418090806|Wang Kon of the Koreans, 2850 BC.SAV
6656|24|10|1119238623|1269173627|193483415|1975920992|Xerxes of the Persians, 2850 BC.SAV
 
I have other priorities going on now and haven't done anything with this since my last update, but while consolidating files and cleaning hard drives I stumbled across my lost C# code in an SVN repo. I'll edit this post after I get it converted to Git and put on GitHub. It's a very early rewrite of this project and probably of little use to anyone, but I've wished several times I could find it, and I finally did.

Edit: https://github.com/myjimnelson/c3satcs
 
Last edited:
:bump:

I finally got around to writing some Civ3-SAV-reading code, and it got me wondering how Civ3 Show-And-Tell was doing, and also curious how you had handled SAV data reading. Hard to believe it's been 5 years since the first post and 1.5 since the last one! Although it was a bit sad that the site was down, after a bit of poking around I got a map for one of my save games generated locally. I also was impressed by the brevity of the Go code, although not versed enough in Go to understand quite how it's so small.

I'm curious about the reason for removing the map option in version 0.31. I used version 0.30 to generate the civmap.json file for display.

One other note is that the viewer code (with d3) is only in the develop branch. I'm guessing that means it's still at somewhat of a work-in-progress stage? I was happy to find it, though.

Lastly, I'm wondering if in the unlikely-but-more-likely-than-a-week-ago event that I start posting in the Stories and Tales sub-forum again, there would be any objection if I put the d3.html, svgmap.css, and (generated) civmap.json files on a web server and linked to it for viewable-online maps. Aside from the 56K warning due to the JSON size, I still think it would be a neat addition to stories.
 
Absolutely serve the files! That was the original intent, to allow people to view the game situation without needing to download, fire up Civ3 and poke around.

I've read the files using Python, C#, and Go. Is that all? I thought there was a fourth language in one of the rewrites. The holy grail would be to read the data structures in directly, but I was never able to quite get there. I generally wound up searching for the next expected 4-character "header" (which apparently is actually a C++ class/struct type) and extracting data into my own structure.

Although now that I look at it, the Go code does try to read it in with native structure.

The "reason" for removing the map option was that I decided to start over in Go *and* use defined structs, and I didn't quite get to enough map data before fizzling out. (Looking at code....) Well, maybe I did get that far but I seem to recall being undecided about how to handle the map generation, and fizzled out there. Also at some point I happily discovered the game settings info and wanted to publish a version that could extract the seed and settings which took some mental effort away from the map.

(Looking at commits / network graph....) Oh ok...I think I pulled the map code because I was trying to make a sort-of release-quality tool that someone wanted. I think this is the status from rusty memory and looking at develop: I seem to be reading enough to get the map rendered, but I think the output is just a JSON dump of the memory structures...possibly including all the non-map stuff. The d3 code exists in develop, but I don't think it's been updated to read from the Go c3sat map output. It should be semi-trivial to alter the d3 code to pull from the Go output, but I think it was my intention to...well, make it more complicated than that. Which probably helped lead to fizzling out.

Nobody every used the site, and it kept crashing, so it got forgotten about during some migrations. I think rather than setting it up the same way I'd do it all with modern cloud tools like S3 and maybe Lambda. Actually that would probably not be too hard.

Also, the last Python version will do maps...oh but it's not as portable. Oh but it's a native Lambda language. Hmm.

I have a lot of stuff vying for my attention right now. I haven't necessarily abandoned this forever, and I still have a little interest in it, but I have other higher priorities and interests at the moment.

Heh, like https://www.spellbackwards.net/ . After learning Angular 5/6 with Otda3 I made this silly utility. I saw the .com version sell on Flippa for $2k in September and decided to toy around with utility sites. This site is also my first real foray into Material Design, and some of those components will be useful if I start toying around with Otda3 again. Or maybe even if I resurrect the map converter/hosting site.
 
I remember the Python -> Go conversion, and saw the post about the rediscovered C# version. I definitely understand that it's been a way to try new things out as much as an ongoing project!

Alas, I didn't wind up getting back in to Stories and Tales, but briefly, while Civ3 Show-and-Tell was available. Turns out it's harder to keep a good story cadence going when you have a day job too. If I do try it again, I'll switch from the write-a-new-segment-after-the-last-one-is-posted strategy (which is great for interactivity, but not so great for when real life provides distractions) to a built-up-a-big-backlog-of-segments strategy, with a regular-but-not-too-frequent posting strategy, so that if I don't play for a month I'll still have story segments to post.

I have been learning about cloud deployment tools (particularly Docker, and tangentially things like Ansible), and it's an interesting though. It certainly simplifies deployment. I like (and am mildly horrified by) the newfangled strategy of "we used to treat servers like pets, and nurse them back to health when they were injured. Now we treat them like cattle, and shoot them when they are injured (and spin up a new one)". Horrified for the cattle, but it's sound for servers. I haven't gone that route for my personal web server project, but do have it as a self-deployed (and self-configured) .jar file, so it's not a whole lot more complex than that.

Upping the quality of personal projects can be so fraught. I remember how in 2014 or 2015 I got the great idea to bring all the 5-year-old code in my editor up to the standards that I'd learned in the past 5 years. They were better, but doing so was completely boring and, aside from a few rare cases where it simplified things, didn't advance the editor at all. In retrospect, I'm not surprised that burned me out on updating the editor for awhile. It can happen when you're being paid to work on something boring for too long; no wonder it can happen on a side project.

And yeah, I can identify with other stuff vying for attention, too. My Civ3 interest varies considerably, too. Right now it's at a high point, but there have been months at a time when I didn't so much as visit CFC.
 
Hello Puppeteer and Quintillus

Long time reader and a new poster here.
The reason I am posting is that I thought it would be really handy to know "when the cities are whippable (if that's a word)" without checking each one every time.

From what I read so far it seems that civ3 stores this data in save files with each section separated by a header string.
If you could give me a general advice on where to "begin" parsing save files or if there is already a tool that extracts city production data it would be awesome.

Many thanks!
 
If you could give me a general advice on where to "begin" parsing save files or if there is already a tool that extracts city production data it would be awesome.
If you can get it to work on your machine, then @ainwood's 'CivAssist' app does this, in a general sense.

It doesn't specifically tell you when you can whip, but it does have a 'City production' tab where you can see all your cites listed, along with how many shields have been accumulated into each city's current project — and hence (by mental subtraction) where it might be worth a whip (e.g. 19 shields left, in a town with 2 or more citizens).
 
Thanks tj.

Unfortunately I couldn't make CivAssist to work on my win10 machine, but really I want to create something much simpler.
I play the game in a windowed mode, so what I would prefer to do is to have a txt file beside the game window to show a list of the cities sorted by the number of people to be whipped.
This would be an easy script (+ whatever is cronjob in windows for the update maybe) if I could have a csv file showing production of my cities.
 
Should be doable. I've never tried to find out where that info is, but guess it would be in the city class, and should be easy enough to find out.

Save a game, use civ3sat to hex dump the save, whip a citizen then save and hexdump again and use a diff tool on the two hexdumps.

I may be able to do/type more later when I'm on a PC not a phone.
 
Okay there seems to be a lot of changes and as you said mostly in the CITY class and also CTZN. (There is also a change in TILE but the units will appear in the next turn after whipping so I don't know the reason for this).
Should I change the hex extensions and upload them here?
 
Okay there seems to be a lot of changes and as you said mostly in the CITY class and also CTZN. (There is also a change in TILE but the units will appear in the next turn after whipping so I don't know the reason for this).
Should I change the hex extensions and upload them here?

Makes sense. At first I thought the TILE might change if you whipped the city from size 7+ to size <=6, but since there's already some duplicate data (city location) in TILE it wouldn't surprise me if the population count is there, too, so even just changing the pop might update the TILE.

...Hmm, I just tried my test myself. There are many more changes than I imagined. And the city names aren't with the CITY objects, so there's more to decode than I imagined.
 
Since I last updated this program I made McpeTool which is basically a save-file reader for Minecraft PE/Bedrock, so similar to this in concept and also written in Go. I had also ignored it for months or a year at a time twice, but a week ago I started trying to do something with the information.

I found myself with the same quandary I ran into here more than once: where do I process the data, and how much info do I expose? They're the kind of decisions that can paint you into a corner down the line.

On New Year's Day I ran into such questions and thought, "I wish I could query for just the data I want," and, well, a lot of ideas started falling into place.

GraphQL is one of those things I had filed in the back of my head to check out someday, and that's where I went first. And it's freakin' awesome for my needs. It's a defined query language, and I write code for how to fill the queries. So I don't have to paint myself into a corner with decisions on what to expose and how to format the data; that can be done at query time.

I updated McpeTool with the GraphQL API and was able to duplicate the functionality I had already done with a REST API, but with more control. I should probably keep working on that as that tool has a much larger potential audience...

But my mind kept coming back to c3sat and how this could solve many of the problems I had, enable easier analysis/decoding, allow for arbitrary data retrieval, and perhaps a few other features. GraphQL has a subscription concept...I wonder if this could be leveraged for the thread-recent idea of watching for city changes and a much older idea of triggering outside-game actions when a new tech is discovered or other event happens, like opening a video or splash screen?

I also found myself mentally rearchitecting the whole save file decoding process. I worked so long and hard to try to get the data into structures, and that led to burnout a few times. Now my idea is just to slurp the file in as a byte array, index the four-letter class/section headers, and just pull data on demand as offsets from those markers.

Ok, here goes nothing...
 
https://github.com/myjimnelson/c3sat/releases/tag/v0.4.0
https://github.com/myjimnelson/c3sat/releases/download/v0.4.0/civ3sat.exe

I only made a 64-bit Windows binary because lazy. If anyone cares and needs other formats like for 32-bit WinXP or Darwin I can build them.

So what does it do?

It can execute GraphQL queries that I defined to pull data from the save file. Right now the "civ3" query can return map generation information. The other defined queries are lower-level and return data in various formats from offsets from section headers.

Example: "civ3sat.exe --path <path-to-save-game> api" starts a server at http://127.0.0.1:8080/graphql . You can use a standalone GraphQL client to point to this or browse to it with a web browser which will load the Playground GraphQL web client (screenshot attached). Queries can also be passed on the command line with the graphql command, but quotes need to be escaped.

The GraphQL clients include tools to explore the schema and available queries. The output is JSON. The input is NOT JSON but GraphQL language which is basically a hierarchical recipe for the data you which to retrieve.

This is the friendly query of which I hope to make more in the future. It returns world gen info and the results are in the screenshot.

Code:
{
  civ3 {
    worldSeed
    size
    barbariansFinal
    landMassFinal
    oceanCoverageFinal
    climateFinal
    temperatureFinal
    ageFinal
  }
}

The lower-level queries can return lists of bytes, int16s or int32s. "base64" and "hexString" queries are just like "bytes" but return strings in the noted format.

The program internally finds all the four-character "section headers" you see when digging through the save game like TILE, GAME, WRLD, etc.. The lower-level queries find the nth instance of the specified section then return a number of values at an offset from that section.

This query returns a list of starting locations (as world tile index). Specifically it returns 32 int32s starting at offset 36 (bytes) from the 2nd WRLD section .

Code:
{
  int32s(section: "WRLD", nth: 2, offset: 36, count: 32)
}

Example output:
Spoiler Spoiler because kinda long :

Code:
{
  "data": {
    "int32s": [
      3712,
      3466,
      3005,
      2456,
      1468,
      1969,
      3557,
      2620,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      2147483647,
      1391838083
    ]
  }
}

This query returns the trade network ID by civ for a tile which is a list of 32 int16s at offset 26 of the 4th TILE section for each tile, so nth must be a multiple of 4, or 4 times the tile index plus four. I won't bother with results; it's just like the other except a list of all 32767 because this game has no trade on tile 0. I just wanted a list of int16s to check my code.

Code:
{
  int16s(section: "TILE", nth: 4, offset: 26, count: 32)
}

You could in theory decode the entire save file with these low-level queries...maybe. But I hope to define more game-intelligent queries and enhance the low-level queries with loops/repeats for predictably-repeating sections like TILE and CONT.
 

Attachments

  • 2019-01-06 22_14_47-Playground - http___127.0.0.1_8080_graphql.png
    2019-01-06 22_14_47-Playground - http___127.0.0.1_8080_graphql.png
    52.5 KB · Views: 99
Last edited:
:bump:

Civ3 2020!

I semi-retired a couple of months ago, and in my increased leisure time I have come around to civ3 again.

It seems like the Windows updates environment has changed since I last tried to play much. I had to download something to make the audio work right (especially in diplomacy), and I can't make either Civassist II or MapStat work anymore. Or at least they won't monitor files automatically anymore, and I'm not going to manually load the autosave every turn!

I'm not yet sure what this means for this thread/program. I did play around a little today with a new way of representing a map, this time in modern vanilla HTML/CSS/JS. (Code in the isocss.* files on GitHub.) The attached image is a 3×4 manual layout with some relevant unicode characters on each tile. But for the moment I'm not in the mood to fully implement the map generation again. One reason is I constantly change my mind on where in the flow I want to extract and render the data. I've also historically gone all over the place on how I want it to look, but I'm liking the simplicity of the Unicode characters and wondering if there are enough of those plus web icons to make useful maps.

When I started this thread years ago I explicitly did not want to recreate CivAssist II. But now it doesn't work anymore, and neither does MapStat. And as far as I can tell, I'm the only one who currently occasionally dabbles in parsing the save game files (from which is where these tools get their info).

So I'm toying with the idea of making my own save-game file monitor ./ game reporting & advisor program. That will probably be a different thread even though it would start from the same code base. And it's more than a matter of coding; I never figured out how to properly parse all the data. I had mostly figured out the map data, but parsing the city and unit data properly, for example, I never accomplished. (It's inconsistently sized/shaped, and I have yet to figure out where the telltales are to structure the data.)

My own gameplay priorities would be happiness and diplomacy alerts, such as civ at war is willing to talk, or new techs available. I've seen more than one person looking for a new way to monitor forest chops for shields, and that's data I should already be able to parse, and making a simplified map for that should be trivial.
 

Attachments

  • 2020-01-09 19_14_47-Civ3 Show-and-Tell Isometric CSS Map.png
    2020-01-09 19_14_47-Civ3 Show-and-Tell Isometric CSS Map.png
    10.9 KB · Views: 48
I think I may actually make a start on the new program. But the hardest part is first: picking a name. I kind of need that to start a new dev thread.

Some early ideas, coming from imagery of a person or department that gathers governmental info and summarizes it for an executive (and later randomly branching out):
  • Ministry of Civ3
  • Bureau of Civ3
  • Civ3 Conquests (or Complete) Virtual Assistant (C3CVA)
  • Civ3 Analytics
  • Civ3 Crutch
  • Civ3 Micromanager
  • Civ3 Rollup
Anybody else have ideas? What would a ministry/bureau/department be for a Despot?

Edit: Hmm, maybe Civ3 Intelligence Agency (CIA3, just because), or MIciv3.
 
Last edited:
But for the moment I'm not in the mood to fully implement the map generation again.

So what did I do? Spent a few hours on getting map rendering working.

Well, there's still a lot missing, and I actually don't intend to work more on map rendering. This is literally 9 bits (as in binary bits) of information from each tile: the terrain type byte, and the forest chopped flag. But getting the tiles aligned right and styled up took some work.

The tiles where a forest was chopped have a "C" on them in this image. This was a game I was playing a day or two ago, and it matches with the forests I chopped, so I think I have the right value.

I also had to refamiliarize myself with the details of the GraphQL code. I'm liking its versatility.
 

Attachments

  • 2020-01-10 12_47_32-Civ3 Show-and-Tell Isometric CSS Map.png
    2020-01-10 12_47_32-Civ3 Show-and-Tell Isometric CSS Map.png
    42.8 KB · Views: 47
:wavey:

I caught up on the thread just now (missed the 2019 updates). There are several interesting things in the past 13 months.

GraphQL looks interesting. It took a few times of reading it, but then your WRLD example clicked. That's definitely handy when you don't necessarily know/want to implement the parsing of an entire section. And in SAVs, more so than BIQs, that's likely to be the case. In my (still somewhat skeletal) SAV parsing, I've implemented a fair amount of skipping bytes in the parser to get a similar effect, but with far less convenient syntax than what GraphQL is giving you. The subscription idea is interesting too... if indeed it would watch the result of the same query as the save file (e.g. autosave file) changes, that could get you a lot of functionality along the lines of what a CAII replacement would need for essentially free.

Depending of the definition of "occasionally", I also dabble in SAV-game parsing at times. But I'm not aware of anyone who hasn't posted on page 10 of this thread who is still doing so. As such, I've received a few requests over the years to build out partial replacements for CAII from people who can't get it to run on their systems. I've never actually reached a stage where I had anything useful along that line, due to various factors, which in order include time dedicated to it (or lack thereof), competing ideas (i.e. editor functionality, pie-in-the-sky Civ3 ideas), and challenging in parsing SAVs (Diplo tables being a particular intersection of difficulty, with oft-requested functionality). Suffice to say, if you want to take on the CAII replacement task, by all means, go ahead! I'm not going to be making any progress on it this year, but there is some demand for it.

The interesting technical thing I've done with SAV-parsing in Q2 2019 (which would still be required for my pie-in-the-sky Civ3 ideas, to some degree at least) is experimenting with Java's annotation processing, which gives a more declarative way of reading BIQs, SAVs, or other such things.

Spoiler Technical Details :

Instead of (in pseudocode) doing something like:

Code:
public class CTZN {

    int shields;
    int happiness;
    int anticorruption;
    ...

    public void read(Reader in) {
          shields = in.readInt();
          happiness = in.readInt();
          anticorruption = in.readInt();
    }

    public void write(Writer out) {
         out.writeInt(shields);
         out.writeInt(happiness);
         out.writeInt(anticorruption);
    }

I can instead do something like this:

Code:
@Civ3Section
public class CTZN {

    @Civ3Field int shields;
    @Civ3Field int happiness;
    @Civ3Field int anticorruption;
    ...

And (with one-time code to tell Civ3Field how to read ints/shorts/strings with defined lengths, and Civ3Section how to read a whole section), the manual read/write effort is taken care of. Much quicker overall! And a technique which I never knew I could implement, until learning about it from a colleague who was reading COBOL records with it, and realizing that the COBOL records and Civ3 data both were fairly highly-structured data formats.

There were some caveats, though, notably some fairly complex areas of the BIQ and (moreso) SAV that still required custom parsing (or simply aren't documented); and that one thing I didn't do upfront was create a method to parse a file partially with the new method, and partially with the old, more labor-intensive one. Instead I was working on moving it all to the new method. But while quicker, the new method still requires annotating probably well over a thousand fields, and writing a few dozen read and writer parsers, and dealing with various thorny parts of the file formats that I hadn't revisited for a long time. Thus, predictably, I eventually lost momentum, and stalled out. I think I got to reading (and writing?) all the non-map sections of the BIQ with annotation processing. Which is great, except that I already could read all those sections. I think if I revisit this in 2020, the first step will be making it so I can mix-and-match parsing methods, and then using annotations for parts of the SAV that I don't already have figured out (and as I don't have writing for any of them, that could be potentially all SAV sections).


The new map renderer looks nice. I'd have to compare it to the previous iterations to say which one looks best. I think you had a D3 one at some point? IIRC it was the previous high point. I also recall a CSS one, which IIRC looked nice, but had some performance challenges. It would be interesting to try out Mozilla's new Quantum CSS engine on it, and see if it now performed like a hot knife through butter.

I like the Civ Intelligence Agency 3 name. The reference is clear but works well, and the full name accurately conveys its use.
 
Top Bottom