Civ3 Show-And-Tell

[Quintillus]

Hi! A few thoughts as I'm reading through:


I tried Alpha 1. Seems to be working pretty well! Tested it in both Vivaldi and Firefox and both are happy. Tree-chopping detection is working. The new auto-save pickup works instantly. Pretty nifty for a first alpha!

I'd agree with focusing on making something useful rather than the shotgun approach. Even Alpha 1 gives a glimpse of a useful tool in the direction of a CAII successor, and as one of my colleagues says, "Working code is king." I.e. he will suggest various improvements to the coding style, ways of improving the code quality/maintainability, ways to avoid tech debt, but in the end none of that matters if you can't accomplish what you want to do with the code. One of the biggest motivators for working on CIA3 is likely to be people using CIA3 on actual game and providing feedback, so getting it to the point where it's somewhat broadly applicable for that would make sense as a target.

Spoiler Thoughts on technical aspects and SAV details :

• You are correct that the BIQ is only embedded if it's not the default one; I load the default conquests.biq for SAVs that lack a BIQ. Separating the SAV into the BIQ section and the actual SAV section probably will simplify queries.
• There is a double-digit percent chance I may be using GraphQL at work next week. So it's interesting following your adventures with it.
• I suppose my thoughts on which browsers to support are, I wouldn't spend much time on IE11 (most Windows users will have at least one alternative browser); Edge is switching to the Blink (Chromium) engine soon, as in Q1, so you probably don't need to spend too much time supporting its old/current-as-of-January engine, although that's probably an easier task than supporting IE; and my main focus would be on not making it Chrome-specific. Supporting Chrome, Firefox, and Safari (and thus implicitly derivatives thereof) gives a pretty wide swatch of support, and should be enough to avoid browser lock-in.
• On that note, lorca looks like an interesting conundrum. It may make it easier to write the application in Go, but also provides browser lock-in. I'll be downloading the next alpha and trying it out, but as I don't have Chrome (for philosophical reasons), will be providing feedback on how it works in other Blink-based browsers and Firefox instead. I'm somewhat confused by the combination of the effort expended towards looking at IE/Edge support, as well as looking at integrating lorca, which would make that IE/Edge effort non-applicable.
• I've done WebSockets once before (in JavaScript on the front end, Java on the back end). It worked pretty reliably, but it's been a couple years and I don't remember the details. I do remember that the Chrome Dev Tools have a pretty decent WebSocket debugger (in the network tab; it lets you see each WebSocket communication), and Firefox added WebSocket support to their debugger a few months ago. Although, it may well not be the effort switching back now that long polling is working. I can look up and share the details if you're interested in trying it again.
• The "spin up the server and start the browser" seems acceptable to me from a UX standpoint, at least early on. If it had to be command-line, that would probably be too steep a threshold, but opening up in a tab in the user's default browser seems okay. Maybe errors could be piped up to the client if need be. The only caveat is if it requires Chrome, and the user's default browser is not Chrome, it wouldn't be a very smooth experience.
• Sounds like you've already made error handling smoother. Excellent!
• I'm using Mongo at work currently; it's pretty nice to work with. I haven't heard of or used LevelDB, so no thoughts on it.

I'm about to try Alpha 3 now! Will update once I've given it a whirl.

Edit: Alpha 2/3 don't seem to work on my system. Maybe because I don't have Chrome? Alpha 3 (which I tried first, and started from the Command Prompt) gave me a Windows Firewall popup, which I allowed on private networks, but then it exited. Alpha 2 did the same, minus the firewall pop-up. I tried browsing to localhost:8080/isocss.html in both, as well as plain-old localhost, but both failed, probably because cia3 had already exited.

I'd somewhat expected it to still work in a Chromium-derived browser (I have Vivaldi, which includes the Chrome Dev Tools; Chromium and Brave are two other examples that would fall in this category), but it looks like it's hard-coded to require Chrome itself?

 

[Puppeteer]

Thanks for trying it! Yeah, it definitely motivates me if people actually use it.

The browser support is slightly confusing. There are two aspects: the feature support of the browser, and the UI hooks for Go.

For feature support, after my refactor, only webcomponent-supporting browsers would work, but I could add polyfills (and perhaps transpilation) to support more browsers. (Which is trivial but nonfun repetitive work and doesn't really move anything forward towards my goals.)

But to make it more user-friendly I was trying to make it launch straight into a browser, and that's where the Go UI libraries come into play. Alpha 2 and 3 are using lorca which requires Chrome. There is also webview which uses the OS' go-to web engine, but for now it uses MSHTML on Windows which causes feature support problems.

After reading your post and your satisfaction with pointing your own browser at it, I have an idea. I could try the launch-to-browser but fail back to a 'native' GUI window with a link to the server. Then I realized this gives me a place to display errors pre-browser. Then I realized I can and should delay as many potentially-error-producing actions as possible (registry queries, file opens, file watches) until I have a window to which to display errors. That would allow it to open the browser directly if successful but still work if e.g. I'm using lorca and Chrome isn't detected.

That's when I realized the GTK libraries require, well, libraries to be installed. I'd really like to stick to one downloadable, executable file if possible. And I just need a fallback/error window, so now I'm checking out fyne.

Spoiler Techie(er) stuff :

• Thanks. I'll start loading the default BIQ where one isn't present in the SAV.
• I'm really liking GraphQL for my purposes. On the server/intermediate side, you basically get to code how to fill each query, so you can pull from different data sources or refactor without affecting the query API. It's like an API where the user can get only the data they want from one endpoint, and it's self-documenting. Go's gql is pretty verbose, but Apollo looks much more friendly to code for. Actually Go's gql isn't bad, it's just...verbose. Copy-paste is my friend. The biggest "gotchas" I've run into with GraphQL is when translating a query from the Playground to an API call, because escaping and formatting the non-JSON-but-JSON-looking query into a JSON request object.

Huh, while typing about gql I started thinking about mutations (writes). I had long assumed with this codebase that writing isn't an option because I was thinking in terms of properly being able to read the sav in natively and then writing it back out natively. But especially with gql I could allow arbitrary in-place rewriting of any data. I don't have to know the entire SAV file format to change "Chieftan" to "Git gud", "Cleopatra" to "Cleo", change all mountains to hills, or all grasslands to bonus grasslands. Not sure what I would do with that, but that should be quite doable.

Edit: I also had the idea where I could failback on the port numbering: try 8080, then 8000, then go ephermeral.

 

[Puppeteer]

Here is CIA3 alpha 4 (28.5MB win64 .exe). If Chrome is not present, it will open a small GUI with a hyperlink to the web UI. Otherwise it opens straight into a Chrome window

The allStrings GQL query is also new if you dig into the Playground. Also, it will now try ports 8080, then 8000, then 8888, then pick an ephermeral port if the others fail. This lets it use a consistent port while allowing for semi-common dev ports to be in use.

What does it do?

• Looks in the registry for the Civ3 Conquests (or Complete) install location
• Finds the "Latest Save" value from conquests.ini and loads it
• Watches <civ3 location>\Saves and <civ3 location>\Saves\Auto for new SAV files
• Tries to open in a special Chrome window
• If it can't find Chrome, opens a small GUI with a hyperlink to the web UI for other browsers to use
• Loads any new SAV (including autosaves) automatically and refreshes its info

It still doesn't handle pre-GUI errors gracefully yet, but this release is a bit of a proof-of-concept, and I haven't actually tested it on a non-Chrome system yet. (But I did test the GUI before moving it to a fallback-only part of the code.) The executable size has grown by 16MB or so. Oh well, GUIs are expensive. Still way smaller than Electron!

Screenshot is of the fallback GUI with hyperlink.

 

[Puppeteer]

Ugh, I hit a slog and then a brick wall today.

For some reason, reading a list section gave me fits. I think it was logic errors presenting as library errors, but I'm not sure. I basically had to recreate a bit of code a tiny piece at a time with debug output to get it working. But it's working now, so I can get data from each item of a list section, including a 0-terminated string, an array of int32's, or hex dumps of each entire item. Currently this only seems useful if a custom BIQ is in the save.

The brick wall is that I decided reading in the civs would be a good start to the stuff I want. That's the LEAD section(s), but so far they make no sense. The int32 following LEAD does not seem to be either a count or a length

There is also a suspicious gap between the last CONT section and the first LEAD.

I think I know how to approach this: look for the civ count which I kind of expect to be in GAME, or certainly someplace between the last GAME and the first following LEAD. Also compare the length of each LEAD section between games to see if they're always the same or vary.

I can do this manually, but it might be time to code some batch query operations and/or automatic difference finders. Don't know yet. I think the biggest challenge is to avoid burnout. Getting new data is fun, but analyzing not-yet-understood portions of the save are not fun.

Spoiler List section query and result examples :

{
  difficulty: listSection(section: "DIFF", nth: 1) {
    name: string(offset:0, maxLength: 64)
    content_citizens: int32s(offset:64, count: 1)
  }
}

  "data": {
    "difficulty": [
      {
        "content_citizens": [
          4
        ],
        "name": "Chieftain"
      },
      {
        "content_citizens": [
          3
        ],
        "name": "Warlord"
      },
      {
        "content_citizens": [
          2
        ],
        "name": "Regent"
      },
      {
        "content_citizens": [
          2
        ],

{
  tech: listSection(section: "TECH", nth: 1) {
    name: string(offset:0, maxLength: 64)
  }
}

    "tech": [
      {
        "name": "Bronze Working"
      },
      {
        "name": "Masonry"
      },
      {
        "name": "Alphabet"
      },
      {
        "name": "Pottery"
      },

Edit: Oh, there are 32 LEAD sections in this 4-player save. Huh. Maybe there are always 32. OHhhhh, I'm being a dumbass and looking at the embedded BIQ LEADs when I mean to look at the SAV LEADs. No, that's not right; there are no LEADs in this embedded BIQ.

Edit 2: Thinking out loud here. Each LEAD has two ESPNs and a CULT. The ESPNs and CULT seem to have length ints heading them. And then a suspicious gap between that and LEAD. So maybe the "gap" is part of the civ data?

Edit 3: Ok, the gap between CONT and LEAD I think are resource counts. 26 integers, 26 resources, and I recall finding such a list of counts before. There are still "gaps" at the end, including the last LEAD.

Edit 4: Antal1987 to the rescue, I think. This data dump I think will help me decode each LEAD section. I think I can assume that every SAV will have 32 LEADs and that the default game will always be the same length per section. I suspect that BIQs with different resource type counts (and/or unit type counts, improvement type counts, etc.) may result in different LEAD lengths, but one step at a time.

Edit 5: I now think that LEADs *do* have a length. So there is extra stuff between the end of the LEAD length and CULT, and then some more after ESPN. Guessing that's unnamed lists. But this is good news! I can skip to each LEAD section and start pulling out bits of data. When I get around to figuring out the unnamed lists...well, stuff for a later time. I suspect the techs are there, though.

 

[Quintillus]

Hmm, I'm running into an issue that I also saw with Alpha 1, but which I thought probably was due to my hex editing of SAV files. Looks like it probably isn't, as I haven't been doing that lately. For some reason, CIA (alpha 1 and 4) is giving up because it tries to find the auto-save file without the "C" at the end of "BC" - e.g. "Conquests Autosave 3200 B.SAV". I'm not sure why it's losing the C yet. Alpha 1 worked yesterday when I created a new game to try it with, so I suspect it's something to do with trying to resume a game rather than creating a new one.

 

[Puppeteer]

Huh. I have had it get confused tracking autosaves when I start a new game. If the .ini has an autosave as the "last save" and you start a new game, the referenced autosave is delete but the .ini still points to it.

I hadn't noticed the missing "C".

I guess I need to add some controls for files and error handling.

 

[Puppeteer]

To be clear, is this problem when first starting cia3? Or does it happen while running and watching for new saves? I have yet to see any problems with watching for new saves.

 

[Quintillus]

So far, only when first starting it up. Once it detects a save properly once, it keeps tracking it as expected as the game progresses. I fixed it (with alpha 1) in my most recent test by adding an extra "C" to the Latest Save in conquests.ini.

Haven't quite got Alpha 4 up and running yet. Alpha 1's console logging enabled me to figure out what was going on with it. With Alpha 4, the GUI does spin up, and the link opens just fine in my default browser (Vivaldi), but it isn't getting any data, and I haven't found a cause when looking through the browser's debug tools. It could be the "C" issue, though it could be something else.

 

[Puppeteer]

Thanks! Those kind of problems should become less problematic once I improve pre-GUI error messaging and/or code an interface to load files or watch/unwatch dirs manually.

It doesn't surprise me that I might ...

Oh, might the filename have had any non-EN-US characters in it? There is definitely a string encoding incompatibility between Civ3 and Go, but I didn't expect to run into it yet. Although I did myself yesterday, but not for a filename. With the city name "Chichén Itza" in a query. In fact I just finished the Win1252-to-UTF-8 converter to translate Civ3 strings to JSON/UTF-8. (I hope Civ3 is using Win-1252...it works for Chichén Itza at least.) And that would be a problem that would only present in reading from the .ini, because watching for file changes needs to translation. On the other hand, not sure how adding a C to the .ini file would work around that root cause.

I was going to say that it wouldn't surprise me that there is a length issue, but that I was surprised *I* haven't stumbled across it myself. yet.

Speaking of string encoding, that would make any writes a little more complicated. But I have yet to think of any reason to add write ability to the project. In fact it goes against the purpose of both Civ3SAT and CIA3.

Yesterday I got really, really frustrated. I ran into the same sloggy problem 3 times and had to spend a long time copying bits of code that worked the second copy but not the first, but appeared to be absolutely functionally identical. And recompiling and retesting each step of the way. Three. Times. I finally noticed that when I had spaces in a quoted name field it would cause the problem, but the error message gave no clue whatsoever that the problem was there. I just happened to name the second instance "debugging" or "ugh" or "bogus" instead of the friendly name I had the first time.

It almost burned me out, but I feel like I'm really closing in on pulling some useful new info out.

The RACE section in the BIQ may actually turn out to be slightly more of a challenge than LEAD, because the city list comes first and is an arbitrary length, so even though RACE is a list section I can't use my list section code as-is because there is no reliable offset for the leader and civ data. But it should be trivial, just a bit more work.

But I really need an encouraging result, so maybe I'll hard-code default BIQ values temporarily so I can present some per-turn civ data.

On second thought, it might be nearly as simple to read the data in natively than recreate the list by hand. We'll see.

 

[Puppeteer]

Here is CIA3 alpha 5 (28.6MB win64 .exe). No new user features since alpha 4.

But instead of bombing out on errors it stuffs them into a queue (and currently ignores the queue...one step at a time). Currently it should only panic exit if it can't open a TCP listener port. I figured there isn't much point in continuing if that happens, at least until I have an error display to tell you it's not serving.

The error queue is 20 long, and I think there are only 9 places that can send it errors, so it shouldn't lock up waiting on a full queue.

On the GQL side, the listSection and civs queries are implemented (or partially so). Whatever I talked about having implemented since alpha 4 is baked into alpha 5.

Oh, it also runs the "Latest Save" .ini line through a Windows-1252-to-UTF8 conversion since Go's file functions expect UTF8 encoding.

So in short, it still fails silently, but it won't bomb out of its own accord unless it can't open a port for the local server.

Edit: The 'good' news is that the web UI should handle this well as it will only pull data when it sees a refresh event. If the program's start-up attempts fail but the directory watcher succeeds, the web ui will just wait to query data until the watcher sees a new SAV file. (Extra side note: upon page load, the web ui looks back 24 hours for any events. After that it polls since its last refresh or last timeout. More detail: the web UI doesn't know about files or file dates; it just knows about when the last SAV file load happened and/or its last polling timeout.)

Edit 2: For grins, I just added the logic to try two locations for the registry key to support a 32-bit version. The logic seems to work, but it complained about missing source files for one of my dependent libraries. But my main focus is on 64-bit Windows as any 32-bit systems out there can probably still run CAII or MapStat. This should also be able to go cross-platform to Linux or (modern Darwin/Intel) Mac. But it's not my primary focus, and I'm not sure how many are running Civ3 with Wine. Right now the primary blocker to cross-platform CIA3 is not having a UI to set the install and save directories.

Edit 3: Because I want to brag on my progress a little: I'm loading the default BIQ and can query it, the file, or the game. At least for a couple of the queries...lots of little migrations/refactoring to do. But difficulty names and world size names are being pulled from the BIQ! It's a hackish setup right now where the BIQ queried is always the default, but I'll get there.

Spoiler gql query for DIFF & WSIZ :

{
  difficultyNames: listSection(target: "bic", section:"DIFF", nth: 1) {
    name: string(offset: 0, maxLength: 64)
  }
  worldSizeNames: listSection(target: "bic", section: "WSIZ", nth: 1) {
    name: string(offset:32, maxLength: 32)
  }
}

{
  "data": {
    "difficultyNames": [
      {
        "name": "Chieftain"
      },
      {
        "name": "Warlord"
      },
 [...]
      {
        "name": "Sid"
      }
    ],
    "worldSizeNames": [
      {
        "name": "Tiny"
      },
      {
        "name": "Small"
      },
[...]

 

[Puppeteer]

I got quite a bit accomplished on the server/GraphQL side today, but right now the code is in a hacky state where it always reads from the default BIQ and doesn't try to look for an embedded one.

But I can iterate (and cross-reference) the LEAD and RACE data now and have a pretty good general idea of how they're structured.

I have long assumed (and still do) the turn order is the index order, so barbarians go first, then humans, then AI players. I briefly wondered about whether the order of civs taking turns was spoiler info, but then I checked, and the F10 "view space race" screen is in the index order, so I won't be spoiling anything to list all civ names in the order in which they appear.

Next steps in finding useful data is to look for contact/peace/war data and tech data. I used to worry about how to look up tech prerequisites, but I'm not worried about that now even though I haven't looked into it yet. I know the info is in the BIQ, and I'm pretty confident it will be in a list section format that I can iterate and pick out what I need.

Screenshot is of a temp civ list table. Left to right are the first of each LEAD's data presuming they're all int32s. I cross-referenced the race ID with the BIQ and pulled the civ name instead of the number. I'm happy about that.

Spoiler GQL query for cross-referencable civ data :

Only parts of this query are in alpha 5. The next test release can run this.

{
  civs {
    int32s(offset:0, count: 32 )
  }
  race {
    leaderName: string(offset:0, maxLength: 32)
    leaderTitle: string(offset:32, maxLength: 24)
    adjective:  string(offset:88, maxLength: 40)
    civName: string(offset:128, maxLength: 40)
    objectNoun: string(offset:168, maxLength: 40)
  }
}

{
  "data": {
    "civs": [
      {
        "int32s": [
          0,
          0,
          1,
          0,
[...]
    ],
    "race": [
      {
        "adjective": "Barbarian",
        "civName": "A Barbarian Chiefdom",
        "leaderName": "",
        "leaderTitle": "",
        "objectNoun": "Barbarians"
      },
      {
        "adjective": "Roman",
        "civName": "Rome",
        "leaderName": "Caesar",
        "leaderTitle": "Emperor",
        "objectNoun": "Romans"
      },
      {

I also fixed signed number handling. I recognized encoded -1 but today wondered why it wasn't coming through as -1. I was doing the conversion wrong, but now it displays as -1.

 

[Quintillus]

Tried out Alpha 5... still isn't detecting the saves as Alpha 1 does, but there's probably an error in the new error queue. I'd thought perhaps the UTF-8-ifying might have resolved whatever issue it has on my system, but it seems not. Although I'm suspicious about whether the "Latest Save" in Conquests.ini is actually being updated, as I started a new game but it still had the save file from yesterday (where I'd added an extra "C") listed. Going to the next turn, and creating an auto-save, doesn't seem to cause that file to be refreshed.

The English (and West European more generally, AFAIK) version of Civ does use Windows-1252. The Russian version uses Windows-1251, and I've been interpreting the Chinese version as GBK, which seems to be correct and fits the chronology of Civ3's release. With registry hacks, it's possible to convert the English version to use the same encoding as the Russian or Chinese versions, and thus display scenarios with Cyrillic or Chinese text correctly; presumably the reverse applies as well.

I'd be curious what the two locations for Civ install site you are using are, for 32-bit and 64-bit. I'm pretty sure I only have the 32-bit one, as my install location detection doesn't work on 64-bit systems. I remember having a rather painful time getting it to work reliably across both XP and Vista back in the day, and that was before I had a 64-bit system to work with.

Tomorrow I have an uplift session on GraphQL at work, as we're considering using it as part of a project. So it will be interesting to learn more about it!

I wonder if this is about the right time to split out a separate thread for the releases? There's a lot of technical detail here, and while can allow things like verification of the encoding being used, it can also hide the updates from people who may want to try them, but may not follow the technical details. I suppose it's the reason a lot of projects wind up with both a "development" and "discussions" mailing list - most consumers of the application will only care about the latter, but there are technical users and collaborators who may be interested in the former.

 

[Puppeteer]

https://en.wikipedia.org/wiki/WoW64:

HKEY_LOCAL_MACHINE\Software\Wow6432Node is the 32-bit equivalent of HKEY_LOCAL_MACHINE\Software (although 32-bit applications are not aware of this redirection).

So the location as seen by win64 programs on win64 systems (and is the 'real' location) is
SOFTWARE\WOW6432Node\Infogrames\Conquests\install_path , and as seen by win32 programs is
SOFTWARE\Infogrames\Conquests\install_path .

If alpha 1 behaves better than alpha 5 I might be able to see the difference. If I can remember which commits those were.

I'm also confused by autosave / "Latest Save" behavior. It doesn't seem consistent.

The English (and West European more generally, AFAIK) version of Civ does use Windows-1252. The Russian version uses Windows-1251, and I've been interpreting the Chinese version as GBK, which seems to be correct and fits the chronology of Civ3's release. With registry hacks, it's possible to convert the English version to use the same encoding as the Russian or Chinese versions, and thus display scenarios with Cyrillic or Chinese text correctly; presumably the reverse applies as well.

Thanks! That's good to know. International support is not my focus for now, but I have vaguely thought about international versions as Civ3 I think is popular (relatively speaking) in various places. There's surely a way to pull the locale from the environment. Come to think of it, there's probably something in the SAV for different code page indicators.

For my purposes, GraphQL has been great. It does seem like the kind of thing that could become a pain if someone with different philosophical point of view controlled the API / query definition side of it. Because that team control the algorithms and information flow. But in my situation where I control both client and server side, it's a great way to normalize how I pull data out of the SAV. I just reused the same subqueries code for a generic list section for the RACE data, but for RACE I adjust the offset parameter to be zero just after the inconsistently-sized city and leader string arrays.

I do want a release thread, and I think we're getting close. I hadn't formalized my thoughts on it, but I was vaguely thinking I'd like the following for a release thread:

• UI for SAV file watches / loads
• Actually displaying errors rather than swallowing them
• An idea of where the startup UI is headed; a path to consistency, although I may be there or close enough now
• Available features work as expected. Huh, this actually is true now. I was going to say it's not, but the hackish parts are only exposed when querying GQL directly...except the difficulty and world size names are pulling from the default BIQ.
• OK, kind of combining two other items here, but I'd like better handling and messaging of which BIQ is being used and if the default loaded correctly
• Some sort of feedback prompt(s). Just linking back to this or a release thread would seem to make sense, but I have another thing in mind. (Coral Project Ask . I've used Coral Talk for comments, and I think Ask would be good for optionally-anonymous feedback. But the CIA3 part of this would just be a form submitting to a yet-to-be-installed Ask instance. Well, actually just a JS inclusion.)

Or, in short, I want it to be actually useful to someone and generally predictable in its behavior.

I thought about putting it out there more as just a forest chop watcher, but the UI behavior (console window, direct-to-Chrome, now fallback GUI) was very turbulent, and I have been closer and closer to adding info *I* actually want to use in-game.

But it is getting close to time. I'm really driven to find per-civ info right now—literally I just got out of bed and was thinking of it and heading towards it while fixing/eating breakfast—but once I accomplish that or get frustrated with it, perhaps I can see about ensuring errors display to the user so they can be reported. The rest of the bullet point items don't have to be in the first release thread release.

Oh, I'll also want in-app version reporting so I know what version I'm getting reports on.

 

[Puppeteer]

This is today's developer babble post.

(Screenshot) I am hex dumping the civs (LEAD) data in the browser, and I had the idea to style the '00's dim, and that worked great! I also dimmed the periods, and the 'ff's are slightly dimmed but not as much as the zeroes and dots. That makes it a lot easier for me to spot things. But Chrome keels over and dies on this, and it's not my JS, it's the CSS spans. Firefox is handling it fine where Chrome won't even scroll sometimes. Wow.

Notice the "Portuguese" text in the screenshot, lower-right. I think this is a bug in Conquests, although admittedly that's a bit of a reach from the limited info I have. I can see no reason for *any* strings to be in the LEAD section, and they're only present sporadically in other LEADs and SAVs. And the first few times I noticed them, they were incomplete like "nglish". Taking a further leap of thin logic, I'm wild-guessing that has something to do with scientific leaders or scientific golden ages as I know the latter is bugged. And if I'm right that the string doesn't belong there, then it must be overwriting something else, and the game isn't horribly broken.

So, the styled hex dumps are neat, but I'm very quickly realizing I want to:

• Compare values between civs
• Compare values between turns/saves

That's a bit much for my eyeballs even with the styling as the lengths of the dumps are quite long.

My first thought was to treat everything as one long int32 array and then auto-show the changes or differences civ-to-civ / sav-to-sav. And that may still be best.

But now I'm wondering if I can do more targeted/limited hex dumps.

Or...why not both? Yeah, I can do that. Ok, I think I have a plan.

But discussing the LEAD section: For the most part it appears to be 4-byte int32 aligned, although there could be a char or int16 array or two in there, but if there is their length is a multiple of 4 so the alignment holds when it goes back to int32s. Nothing really looks bitmask-y so far. The TILE sections used all sorts of data formats, but LEAD seems to mostly be satisfied with int32s for everything so far, aside from what I think is a bug string.

Edit 1: A few hours later, been watching LEAD values change while playing. I only just now realized Antal1987's dumps' unnamed values are named with offsets! I thought they were just weird references to elsewhere in the headers, but no, they're hex offsets. His dumps don't seem (so far) to exactly match what I see, but they are pretty close and generally in the right order.

Second screenshot is the "friendly" view of civ data I'm decoding. The named items are actually being read from the BIQ. Much of this for the other civs is spoiler info, but it's handy to look at now to make sure the values are sane and correct as far as I can tell.

I've identified the "contact" and "at war" lists and maybe/sort-of know where to look for "willing to talk", but I'll have to declare war on someone else to pin it down for sure.

I may have been getting too deep in the weeds noting every change that happened. If I back out and look at the bigger picture, if I can pin down the "willing to talk" indicator then I'm ready to move on to the tech lists which I think are after the length-designated portion of LEAD.

As far as civ-to-civ vs turn-to-turn differences, I just expanded the table I was using to 111 fields wide (seems to cover the area I need to look at) and put the integer index and hex and decimal offset in the header to take care of the civ-to-civ differences (3rd screenshot of the expanded table). Then I just used jsdiff and diff2html in javascript to run the last and current hex dumps through a diff function and show it all pretty, color-coded, and side-by-side in the browser. The diff engine skips most of the unchanged lines, so that's working great.

 

[Quintillus]

Awesome, sure enough I don't have the WOW6432Node path in use. I can likely incorporate that to make Civ auto-detection easier for the most-likely-90%+ of folks running on 64-bit systems these days.

It looks like I won't be using GraphQL at work real soon, as we punted on it for now until some higher-priority things are in order. Could still wind up making use of it in a month or two though. The group consensus was more "this would probably be helpful in certain areas, but won't be revolutionary enough to jump to it before doing these other important things" than not wanting to use it overall.

I'd agree that non-swallowed errors would be very helpful before having an official release thread. I don't know how many times someone has noticed something in my editor, and either they've looked at or I've asked them to look at the log file, and it really helps in pointing out what went wrong. I suspect the "C" issue may have been solved by now with a non-swallowed error, as it's probably some off-by-one case, at least if it isn't conquests.ini being entirely untrustworthy.

The hex-dumping with color-coding sounds interesting. I've been using HxD as a hex editor for years, and I'll open multiple files, and copy sections of a file that correspond to a particular PRTO or something like that, and display things side-by-side, but it doesn't have color-coded output of what's changed since the last turn. For understanding what's changing in the SAV - which is significantly less-well-understood than the BIQ - that sounds quite useful.

One related question, the About CIA3 page indicates the source code is on GitHub, but it appears to be the Civ3 SAT code that is linked to. I don't know if that is intentional, but it stymied an attempt to see if I could un-swallow the error and gain some insight into what's been tripping up alphas 4 and 5. (Although having never used Go, it's also likely that the attempt would not have wound up being successful)

 

[Puppeteer]

That's the right repo, but all the CIA3 work is still in the develop branch. The cia3.exe code is in cmd/cia3/main.go (and *.go). In fact, I just updated the readme in develop to kind of explain where stuff is, and yeah it needs some reorganization.

The fallback GUI code is in cmd/cia3/fynegui.go and is called from near the end of main.go. I haven't looked into it yet, but I'm hoping I can add a UI element and just write the error messages from errorChannel into it. A gofunc with `err <- errorChannel` that somehow pushes each `err.Error()` to the UI widget should do it, but I don't yet know how to update a fyne UI widget.

Or you could just yank out the `fyneui()` call, build without the windows GUI flag, and add something like this at the end:

Spoiler untried code for spitting errors to console :

for {
    switch {
    case err <- errorChannel:
        fmt.Println(err.Error())
    }
}

Actually, that's halfway to error logging in the fallback GUI. I just don't know yet what the other half looks like.

Hmm, I wasn't going to put the diff code in the repo, but if you might use it I'll leave it in there. It sure has come in incredibly handy for me today! (screenshot). Code is in the latest develop commit. It's in use in civs.html.

---

So, I *think* I found the contact data and "will talk" data, but they have some extra features.

The contact array seems to be a bitmask. 0x1 is have contact, 0x2 might be "have contact, but never actually spoke" as I got a 3 value for contact until I contacted them and saved again. 0x8 for their contact array means my unit is in their territory. I didn't get a chance to see if it works the other way around, too, yet.

"Will talk" seems to be about an 8-turn countdown timer. But it counts down from 8 or 7 again after making peace. But it's the only thing that seems to make sense for a "will talk" signal. So maybe the true internal function is a countdown timer to prevent the AI from making peace too quickly or redeclaring war too quickly. So I think I need to say "will talk" only if at war with the civ *and* "will talk" is nonzero. (Or maybe I should call it "won't talk", but it looks like maybe it's "AI crazy war debounce timer".)

Somewhat interesting: Civs don't have contact with themselves, and barbs have no contact with anyone. I guess that makes sense. Oh, actually I turned off barbs for this trial. (I turned off barbs and picked several expansionist civs on a pangea...but nobody found me, I had to go find them all .)

 

[Puppeteer]

Here's CIA3 alpha 6. (29.8 MB win64 exe)

There is now an errors.html page which reads any errors from the exe's error queue. And it's linked from the main page. That was easier than figuring out how to turn fyne into a console log.

There's a lot going on in civs.html (linked from the develop page). It's spoiler-y but quite interesting to watch while playing. This page had given me trouble in Chrome (but FF was ok), but it's behaving better now. Consolidating spans and using opacity instead of different colors both seemed to help.

The back end does pull data from the BIQ, but it's currently hacked to only read from the default BIQ even if a custom BIQ is present in the file, simply because I've had tons of "to dos" and finding the custom BIQ marker hasn't been to-done yet.

Edit: The contact "unit in territory" flags only seem to happen when at peace. When at war, the contact value remains 1. I just made peace with a civ who has units in my territory, and now I have value 29 (0x1d) with them having two archers in my territory. Haven't seen bits 0x4 or 0x10 until now. Hmm, maybe one of them means I can demand they withdraw immediately or declare, because that is the case. They advanced right next to my city instead of retreating due to barb chasing. Other thoughts are that maybe offensive unit has its own flag, but warriror was the same as scout in the other game. Or maybe intrusion during a 20-turn peace treaty?

 

[Puppeteer]

I started the day off curious about trying to use push notifications for alerting. I briefly looked into it, but I doubt I'll be able to get the behavior I want. Browsers make it really hard for web pages to forcibly get the user's attention, and for good reason.

Fyne can pop up alert dialogs, but I don't know yet if it can pop in front of other apps or make a noise. In any case, I think I'm going to go with fyne for any popup alerting. (To be explicitly enabled by the user first.)

Also, I'm thinking of defaulting into the fyne gui window with the link instead of trying to launch directly into a browser. The only two known cia3 users currently would prefer it that way, and it would present a consistent user experience for all users in all situations.

So I still have lots of shiny things distracting me, but I'm trying to steer myself towards making what I have more useful and consistent, and that means settling the UX, making the newly discovered civ info into an unspoiling table, and getting the proper BIQ loaded for all saves.

I just realized I need to check something. Not now, but it needs to go on my to-do list: I know a civ can be "persuaded" within a human turn to go from unwilling to speak to willing to speak, but I just realized I don't know if that willingness is reflected in the value I'm looking at or if the value I have is a modifier for other behavior. Without combat, the location I'm looking at seems to be a simple number of turns until they're willing to speak.

 

[Puppeteer]

Here is CIA3 v0.4.1-alpha 8 (29.6MB win64 .exe).

This is the most exciting update for me so far. The civ list is useful per-turn information that *I* want to use in my games.

What does it do?

• Looks in the registry for the Civ3 Conquests (or Complete) install location
• Finds the "Latest Save" value from conquests.ini and loads it
• Watches <civ3 location>\Saves and <civ3 location>\Saves\Auto for new SAV files
• Opens a small GUI with a hyperlink to the web UI for browsers to use (currently they need to support WebComponents, so not Edge or IE, but Chrome, FF, and derivatives will work)
• Loads any new SAV (including autosaves) automatically and refreshes its info

Main features:

• Consolidated non-spoiling list of opponents showing contact, war, and "will talk" status, government, era, and city count (when info is available in-game)
• Map view highlighting tiles which have previously been forest-chopped for shield production
• World generation settings including world seed, barb, and map settings.

Screenshots:

• Launch GUI window
• Main game info page
• Chopped forest map

 

[Puppeteer]

This is getting really, really close to having its own release thread. I'd like to play with this version for a bit before doing so to make sure it's working as expected.

There may be numbers in parentheses for contact or "will talk", and relationship may show as a number (although I haven't seen this yet). That is the value in the file, and I'm not entirely sure what they mean yet, but the "will talk" number seems to be the number of turns until they'll talk to you absent combat results. The contact numbers appear to be bit flags, 0x1 meaning contact, 0x8 seems to indicate that civ has units (saw this (decimal 9 total) for both scout and warrior) in your territory. I saw 0x4 and 0x10 plus the others (decimal 29) after making peace and the other civ had two archers immediately approach my city while chasing barbs. I figure those might mean you're able to demand withdraw or declare, and maybe the other is offensive unit.

I haven't seen any numbers for the atWar value yet. So far its either 0 or 1 which I have displaying as "Peace" and "WAR".

The version number is more complex; since I'm getting closer to a "real" release, I'm putting the repo's semantic version in there, and v0.4.1 alphas are where we're at now, but I might bump to 0.5.0 because so many changes have happened since 0.4.0.

I also reorganized the repo. "civ3satgql" was a horrible name for the main workhorse code package; it's now "queryciv3".

Oh, and it is reading from the appropriate BIQ! There seems to be an int32 with value 0xcdcdcdcd to indicate using the default BIQ, so I'm keying on that. I created DYP and Ancient Mediterranean scenario games from the civ-content menu, and the appropriate civ names , tech names (when I was still showing current research for all civs), and era names were clearly from the custom BIQ. Difficulty and world size names are also being read from the appropriate BIQ, but I have yet to try a game with those values customized.

I think I want to playtest the current version a while. I'm not sure what's next as far as dev work, but the next feature I'd really like is to show available tech trades. I think I know roughly where to look for that (between the end of the length of LEAD and the start of ESPN), and I feel much more confident in my ability to read the BIQ for tech prerequisites even though I've only pulled the tech names so far.

Edit: I think I see the problem with Quintillus' "latest save" load. Except I can't figure out why it works for me. I should be having the same problem. I "fixed" it, but now it's broken for me. If I figure it out soon I'll update my last post to link to alpha8 with the fix.

Edit 2: Huh, I have a space (0x20) after my filename and before the carriage return in my .ini . That's why mine works when yours doesn't and vice-versa. But I don't understand why it's there. I guess I'll try removing any spaces from the end of the filename.

Edit 3: I updated my previous post to point to alpha 8 which has the "latest save" fix. It gets the length correct, converts from Win1252 to UTF-8, and trims whitespace from the beginning and end of the filename. (Well, partial filename; the .ini doesn't include the ".SAV" for some reason.)

Edit 4: The world gen settings text is wrong, on the part about how to recreate the map. I'll fix that some other time, probably putting it in a different page. I'm only displaying the 'final' settings, but if the map had been generated with any "random"s those would need to be set random.