Good info, I'll run through it after work tonight. I hadn't been aware of "go get", and am not sure if the GO_PATH is set up on my home machine. It looks like I was misaligned on my understanding of the save monitoring; I thought it was looking only at the conquests.ini, or at least had to succeed there first, which is why I'd focused on it. But indeed, it isn't picking up autosaves, either, or at least nothing has loaded in the web UI for me since Alpha 1.
I'll look at the watercheck.exe too. That output looks like it might be quite helpful in figuring out what's going on.
All right, I got a locally-compiled one working. It must not have been working because I did a "git clone" into a not-the-GO_PATH-folder previously. With "go get", it's now working, or at least working as well as it did before. Still not loading any saves, but the program compiles and runs. Enough that I can start investigating, adding print lines, etc.
The watchercheck.exe is indeed picking up the autosaves. So that seems to be working just fine. It must be something in between the save-watching code and the user interface that is the issue on my machine.
Edit: I've been adding fmt.Printlns throughout the code; so far none have indicated an abnormal condition. However, I did find this in the console of the web browser (in both Vivaldi and Firefox):
It seems that the server is sending the JS file as a plaintext type, and it appears the browser is not parsing the JavaScript as a result. Which might be why the UI hasn't been loading anything. It isn't working in SeaMonkey or Internet Explorer either, despite not printing the error, but they might just not have as good of error messages.
Edit 2: This might be the issue: https://github.com/golang/go/issues/9156 It looks like for some reason Go looks up MIME types in the Windows registry, and Windows has it defaulted to "text/plain". I need to restart to see if changing the registry entry fixes the problem. That would be a really odd one if so.
Interesting. Line 58 of fswatcher.go is the loadNewSav function that gets called when a new file is detected. It makes sure the filename is >4 characters and ends in '.sav' (not case-sensitive), then checks to be sure the path exists and is a file (not a directory). If it passes all that then it calls code in the queryciv3 library to load the new save data and then triggers the refresh message to the long poller.
So if the new file is detected, then my first intuition would be that os.Stat(s) call and then fi.Mode().IsRegular() are somehow failing. After that the file does a compression auto-detection and then slurping in the file, but those are standard file i/o libraries, so I think it would fail at os.Stat or Mode() first if the reader streams were to fail.
Another possibility is the default conquests.biq read. I don't know offhand what would happen if that went wrong, but it might make the UI look like nothing loaded, but there'd be an error in the browser dev console. The call that loads it is in main.go line 26 . I can't imagine why that would fail or at least why it would fail where other stuff didn't, though.
Success! It was the MIME type of the JavaScript file that was being served by the server (as described at https://github.com/golang/go/issues/9156). After I opened the registry and went to HKEY_CLASSES_ROOT\.js, set the Content Type to text/javascript (from text/plain) and restarted, everything worked just fine.
I don't know why that registry entry was set to text/plain, but there it is. I also don't know why the http package in Go delegates to the Windows registry for this; the other servers I've used have always served JavaScript as JavaScript. Maybe there's some sort of option to tell it to always do that, but until then, that's how to fix at least one "it's not working" issue on Windows.
Now that it's working, I'll have to try it out on the new Game of the Month. Who knows, I may adopt it for the story game I'm playing, too... but I want to learn what info CIA3 provides before deciding that.
Interesting. Line 58 of fswatcher.go is the loadNewSav function that gets called when a new file is detected. It makes sure the filename is >4 characters and ends in '.sav' (not case-sensitive), then checks to be sure the path exists and is a file (not a directory). If it passes all that then it calls code in the queryciv3 library to load the new save data and then triggers the refresh message to the long poller.
So if the new file is detected, then my first intuition would be that os.Stat(s) call and then fi.Mode().IsRegular() are somehow failing. After that the file does a compression auto-detection and then slurping in the file, but those are standard file i/o libraries, so I think it would fail at os.Stat or Mode() first if the reader streams were to fail.
Another possibility is the default conquests.biq read. I don't know offhand what would happen if that went wrong, but it might make the UI look like nothing loaded, but there'd be an error in the browser dev console. The call that loads it is in main.go line 26 . I can't imagine why that would fail or at least why it would fail where other stuff didn't, though.
That more or less follows the path of where I was adding logging to see if it worked, and occasionally "else" clauses with logs in case it didn't. But that all wound up being okay in the end.
My two Win10 build systems are serving with the right mime type. Very unexpected that Go's net/http would check the registry; never even crossed my mind. My systems are original Win10 installs instead of upgrades; wonder if that has anything to do with it? No content-type explicitly set in my registry for .js.
Since you seem to have no memory of having previously set that yourself, I'm going to assume this is probably a widespread problem. I already have a middleware bit setting CORS headers, so I can throw in some explicit content-type-by-extension-mapping there.
I know you've used c3sat before, so I was trying to figure what changed. But I think the stuff you used had hard-coded handlers which would have explicitly set the headers instead of the bundled pkger filesystem and file server setup currently in use.
Ok, here's cia3-v0.4.1-alpha10.exe . The only change from alpha9 is the explicit setting of Content-Type headers for static files served for .js, .css, and .html files. The code is also pushed to the develop branch on GitHub.
(The GraphQL and long poller handlers manage their own headers well enough.)
This should fix Quintillus' issue, and presumably the same issue for lots of others who have yet to try it.
I just checked two other computers, one running Windows 10, and the other running Windows XP, and neither one of them has the content type set for .js files in the registry. I don't know why my 8.1 desktop did, but it was installed in July of 2017, so it's entirely possible it didn't when it was installed, and something changed over the past two and a half years .
Well, since there seem to be multiple other people running into similar problems with net/http in other applications, I'm guessing it's something more common than just your particular machine. Maybe it's a dev tool, or maybe it's some weird antiquated default.
---
So I'm kind of out of steam at the moment. I'm not even playing right now. Next steps should be cleaning up for a beta release, but that's no fun. I could also use a LOT of code cleanup. The JS client in particular is getting a bit hairy and needs refactoring to better deduplicate queries between similar components. The UI also needs organization and polishing. Oh, and I'd really like to find out where the "civ is eliminated" info is. A lot of the GQL query code also needs to finish migrating from my old single-file-target way to the newer bic, game, or file way.
But I'm not feeling like putting effort into any of that just now. I'm vaguely interested in mass-sav-scanning to find particularly interesting data points using statistics, but that's going to take some extra smarts to match section offsets between files, and perhaps identifying a game.
Babbling intensifies: Based on the data I'm gathering so far, identifying saves of the same game probably involves the sequence of race IDs (same civs in the same player slots), a continent count...well that might be a good start. Maybe also something from the BIQ. Aggregate city and/or unit count and the date/turn# might be something to key on, but it would involve making a judgement of how likely those numbers are to change and by how much over time. But those limits might become apparent after the first round of game identification. It wouldn't surprise me terribly to find a key value in the save somewhere that is unique per game; maybe in the first section or in the GAME section. Oh, the world seed is another thing to look at!
I've mentioned a time or two I could actually modify a save. That doesn't fit within the aims of either c3sat or cia3, but I'm getting curious enough to see if e.g. turning all grass into BG and all mountains into hills will work. Not entirely sure what a use case for that would be; maybe it's just a stepping stone to taking a random map and changing all plains and deserts to grasslands with an appropriate distribution of BGs. Or otherwise statistically curating/shaping a world.
I wound up taking it for a spin in my story game (which is a lightly modified BIQ), and it works really well with it. The query code may have been written for a single target, but the SAVs from my BIQ are, at least, close enough for it to work with.
Finding enough steam is the perpetual challenge. I haven't written any Civ-related code in several weeks at least; my steam seems to have run out sometime in February. But I have started play Civ III again (after a stint with IV earlier this year).
No, it should be reading from the BIQ now! Whichever BIQ is embedded in the save, or else the default Conquests BIQ.
I haven't done extensive testing, but it's worked well on several tests. I do wonder about e.g. original PTW games later opened and saved with C3C and some other edge cases, but it should either find the embedded BIQ and use it or pull from the default C3C BIQ.
One thing I noticed is that my hex diff page is locking up. I'm not sure when/why that changed, but in thinking about separating the dev/investigative code from the 'real'/user cia3 code I'm thinking maybe I need yet another executable in the project to handle some or all of the dev/diff and batch functionality.
I'm noticing a common pattern in my save data analysis: isolate the repeating pattern (map tiles, tech civ mask, race list, leader list), parse the repeating pattern into comparable chunks, then compare between turns and saves against what I know is happening in the game.
That last comparison step I'm typically doing by looking at hex dump diffs between two saves, although sometimes I'm looking at int arrays or other structured lists. But I should be able to automate that comparison pretty effectively with batch-scanning all saves in a folder and then spitting out some statistics.
e.g. if a value occasionally changes and usually in a power of two, I may have found a bitmap related to civs (or trade networks, apparently, or other items like improvements in a TILE). If it increases over time or does so cyclically it may be a timer, factor of production tracker, tech research progress, etc..
So I'm currently thinking of a Go executable that runs the same query against all saves in a folder, saves the results in a LevelDB (or similar) store, then spits out some sort of statistical analysis. Or maybe I'll have a separate query program for the organized data.
Yeah...because sometimes I'm hunting for a value that changes on a specific turn (such as the elimination of a civ for my current data hunt), so I could just show values that changed between those two turns but didn't change in most or all other saves.
In these data comparison dumps I also want to use my ideas for automatically identifying games. So I could run against literally every save I have, and it could hopefully sort the results by particular game. And after I got that down pat I could even look for branches of the same game where I reloaded and went in a different direction. I don't do that much, but it's not unheard of. Or maybe I started the same world seed with the same players over again. There should be a way to figure out which branch a particular save likely belongs to by city position, relative unit count, or similar.
Oh yeah, city positions would be a fantastic way of differentiating saves! Now I have another reason to want to parse cities. And I fairly recently found the total city count, so maybe it's time to go looking harder at that, because I can count the CITY sections, divide by the city count and see if that helps me figure out how to parse the CITY entries.
I suppose relative unit locations could also be a way to identify branches of the same game between turns.
Ok, all this stats stuff is starting to sound like fun.
Ok, I just did a little thing I'm excited about: I made a Windows DLL with Go. And then I executed a function in it from PowerShell.
SpoilerGo Code :
MinGW gcc (or TDM-GCC) is needed.
Build with flag "-buildmode=c-shared" and then rename the extensionless output to .dll.
In any Go code, exported functions must begin with a capital letter. For the c-shared build (and therefore DLL targets), apparently the comment "//export <FunctionName>" is needed, too, to make the function available to call in the DLL.
Code:
package main
import "C"
import "fmt"
//export HelloDll
func HelloDll() {
fmt.Println("Hello from a DLL!")
}
func main() {
fmt.Print("Hello from the main function. You only see this if you compiled an executable.")
}
SpoilerPowerShell Code :
This works in native PowerShell (PS 5.1 and earlier) but not PowerShell Core (PS 6+).
Of course ".\godll.dll" should be replaced with the path of the dll. The "public" is needed in this case to access the function from the rest of the PS script. Replace "void" with the appropriate return type, if any. And I guess parameters if any would need to be defined, too.
I can't tell that the namespace matters except to not collide with an existing namespace. Assigning the type to a variable seems important as I was unable to call the function from the namespace ([GoDllTest]::HelloDll() didn't work).
Note: As long as the PowerShell instance that loads this is active, the dll file is locked.
Why does this excite me? My programming approach has generally been to use Go (or whatever appropriate language) to extract/generate/format data and then make it available over http. And then I use JS and HTML to automate the client side and present info. But for cases like CIA3, both the client and server are on the local machine. There is convenience in the paradigm but a bunch of unnecessary middle stuff with the http queries in this case. Well, with a relatively easy way to export my function calls, I can now skip the middle stuff and have my client grab data from my Go code more directly.
That frees up the client side of the equation a lot. The obvious front end for me has been a web browser, but often for automation I've used PowerShell, NodeJS, or just bash and curl to batch-run http queries to my more complex back end process.
Well now I can skip the http part. But that also means it will be much easier to use any other programmable framework to act as a client...like Godot. I had toyed with the idea of using Godot before, but the thought of having Godot use http queries to a separate server just seemed even clunkier than what I have now. But having Godot use a DLL would probably take a lot of complexity away.
But even for just batch operations, calling functions directly will be a lot less clunky than running everything through http requests.
And now that I think of it, this opens up the ability to pass native data (int32s, byte arrays, strings, etc.) and skip the JSON serialization and deserialization. It's not immediately clear to me if that's beneficial to this project in particular as I'm already heavily invested in the GraphQL paradigm, but it's a new angle to ponder.
The most immediate benefit of all this to this project in particular is for batching scans of lots of SAV files which is a thing I've been wanting to do but hadn't quite figured out how to do it in a sane and convenient manner. Well now I can export a couple of functions from CIA3 in a DLL and have PS or bash collect a list of SAV files, tell the DLL to load each in turn and make requests and handle the output however I like.
In fact the simplest thing now is probably turn-to-turn diffs: I could compile this in WSL and just pipe two SAVs to xxd and diff which is kind of what I wanted to see in the first place.
Another random but related thought: I haven't really called DLLs from programs before, but it's occurred to me more than once that compressing and decompressing civ3 SAVs should be pretty a pretty simple DLL call. Well, scratch that. I just looked and don't see an obvious dll for PKWare DCL, so it may be compiled into the exe. There is a binkw32.dll which seems to be a video codec...that could be interesting to poke at. The other dlls don't look immediately exciting.
Edit: And while I'm thinking deeply on shared libraries and accessing them from .NET, there's this little nagging idea I get in my head from time to time: CivAssist II is a .NET app. A .NET app is a collection of assemblies. My thought is to wonder if an external program could "drive" CAII around its problems on modern OSes. At the very least, externally monitor the save files and tell CAII to open them each time its modified/created. At the most, take over for CAII's main program loop, call what works, and avoid calling what doesn't work. At least some of that may be doable without the source code. I'm deducing the main conflicts are attempts to touch the live game like displaying a CAII icon in-game or the ability to locate a unit or city in-game from an action in CAII. Also Ainwood says the big problem is that some of the ,NET methods used are deprecated now and would need to be rewritten. I'm guessing that file monitoring may be one of them. In any case, the gears are turning in my head....
Edit 2: The name of this hypothetical project would be "CivAssist II Assistant" (CAIIA) or "Assistant to CivAssist II" (A2CAII).
Edit 3: I've been playing with making simple "hello world" dlls and started looking into getting Godot to use them. It's not as straightforward as I hoped; Godot is pretty darn particular about its shared libraries. And the Go bindings for Godot are 2-3 years old and won't work for me on Windows. It looks like the path of least resistance to getting Godot to use my Go dll may be to make a wrapper library in C or C++ either manually or using the better-maintained Godot bindings. I understand the concepts well enough, but the plumbing is complicated. (Edit to the edit: Oh, it may be possible to include the C or C++ bindings in Go. Or at the very least include all the C/C++ code in the repo and one go build command can make a single dll out of the wrapped mess.)
As for my DLL talk last post, I tried passing parameters and returning data from the DLL programs but ran into inconsistencies across languages and platforms, and I fizzled out on the DLL angle.
However, for scripting / batching I have a new toy: Lua, an embeddable scripting language. I haven't used it with Civ III yet, but I used it in another Go project to good effect.
But I haven't quite figured out what this means for the web UI & GraphQL API. I like GraphQL in concept, but for the very targeted application I'm doing it doesn't necessarily make sense. It does help get the data in an organized manner, but when making large queries and then trying to massage the data for display, things got pretty cumbersome.
My idea is to expose the save data in a Lua environment. But I'm really not sure what the interface is. Come to think of it, most of my "interface"s are just tables. The map is the only outlier.
Oh, I've been worrying about formatting data in Lua, but I really don't need to worry; I can do it in Go and expose the function call in Lua.
My idea is to expose the save data in a Lua environment. But I'm really not sure what the interface is. Come to think of it, most of my "interface"s are just tables. The map is the only outlier.
I've been giving this some thought, and I'm starting to think I should quit worrying about the UI at all. Let's face it: I'm not good at UIs and don't particularly want to be. I'm good at batch processing data and making different components work together.
I've been letting the UI concern slow down other progress. Most or even all of the data I'm surfacing is tabular, and 'user input' is just picking which table(s) to look at. Even the map is just a layered 2D tabular matrix of data, even if it's easier to digest in graphical form.
And I really had the map display already figured out at least two or three ways, anyway.
But perhaps a bigger problem was separating data extraction and data processing with a REST and/or GraphQL API in the middle. There are reasons I like this—mainly that the back end extractor doesn't have to be too 'smart' and the processing side can be user-driven—but it's proving a bit cumbersome in real use.
So for the moment I think I'll start a third executable, call it maybe clua, c3lua, or civ3lua, and try to bring it up to feature parity with what GraphQL is doing now. Note that cia3 is a combination of the GraphQL queries and some rather unoptimized and ugly javascript on the client side, so clua won't be replacing cia3 the moment it reaches feature parity with GraphQL. (In fact, cia3 will adopt the clua way of gathering data, not be replaced by it.)
I have a pretty cool proof of concept going for the Lua-Civ3 interface. I don't have an executable yet; I'm using go tests to run things quickly while developing.
So it currently sets up a Lua environment with some global modules:
- install_path - If it can find the Civ III install path in the registry, it's here
- bic.load_default() - Loads a BIC into memory for use as the default; if no parameter it tries to load install_path + "/conquests.biq"
- sav.load() - Takes a path as a parameter and loads a save into memory
- get_savs() - Takes a directory path and returns a table array of .sav files in that directory
- civ3 - contains some values found in the file header
- tile.width, tile.height, and tile[1] through all tiles - each tile value has a base_terrain and overlay_terrain, so tile[1].base_terrain, etc. . Note that civ3 data has 0-based arrays and lua has 1-based arrays, so some +/-1 translation is required
It's going a bit more smoothly than I imagined so far. I have high hopes for expanding this quickly.
Spoilerlua code for printing a text map :
Code:
bic.load_default()
sav.load(install_path .. "/Saves/Auto/Conquests Autosave 4000 BC.SAV")
for k, v in ipairs(tile) do
-- newline for end of map row
if (k - 1) % (tile.width / 2) == 0 then
io.write("\n")
-- indent odd map rows
if math.floor((k - 1) / (tile.width / 2)) % 2 == 1 then
io.write(" ")
end
end
-- tilde for water, blank for land
if v.base_terrain > 10 then
io.write("~ ")
else
io.write(" ")
end
end
io.write("\n")
Here's an example of accessing a batch of saves. I'm hoping this can be used to compare lots of saves at once to find statistics on various parts of the save file, like how often it changes, the average and standard deviation, etc.. "Looking" at dozens of saves at once may make some things more clear than one save at a time.
Spoiler:
Code:
foo = get_savs({install_path .. "/Saves/Auto"})
for _, v in pairs(foo) do
print(v)
sav.load(v)
-- print(sav.dump())
-- print(civ3.always26)
-- print(civ3.maybe_version_minor)
-- print(civ3.maybe_version_major)
-- for k, v in pairs(civ3) do
-- print(k, v)
-- end
print(tile.width)
print(tile.height)
end
The lua-based code is currently in the lua branch of c3sat , but now that it seems to be more than just an experiment I'll probably merge it to the develop branch soon-ish. I might consider merging it to master as it doesn't interfere with any existing code.
Edit: The lua branch is gone. The lua code is now in develop, and I'm about to merge it into master. It's not done, but it's not breaking anything, either, and it's at least interesting if not quite useful yet.
There is a clua executable now, although I haven't released it yet. The code is in the repo.
My decompression routine worked fine, but when I started processing dozens of saves at once, I noticed it's slow. The autosaves went by much faster than the regular saves, and that's why.
So I got side-tracked trying to speed it up, and I wound up writing another decompressor function, and it is *much* faster.
Then I worked on another project for a bit.
I'm not sure what I want to do on this project next. I know where I want to get to, but getting from here to there will take some doing.
SuedecivIII recently posted a question or two that this project would eventually be great at answering: what triggers barb camps to spawn, and what starts the tier 2 barb units? His testing suggests it's based on city count. My clua program should eventually be able to scan a lot of save files and note barb camp, barb unit type, and city counts, among other things. So that might be a medium-term goal to try to reach.
Meanwhile, here's another take on a text world map, because maps are kinda easy and rewarding:
SpoilerAnother text map :
Each "map tile" is 2 wide, but since every other x location is blank, four characters are printed to fill the space. This makes the aspect ratio a little closer to the game than the first text map. Also using blocks instead of blanks for land.
I decided to go straight for the cities vs barbs question without "doing it right". UNIT sections look easy enough compared to other stuff I've looked at.
One thing I just noticed is more stray text in a UNIT section where I think it has no business being. When I saw this in LEAD I speculated it might be an error of some sort and wildly guessed it might have something to do with the broken scientific age. Hah.
Now I think it's just 'dirty' memory, and the places where this stray text is are unimportant/uninitialized. Perhaps unused string space or unused array space. That could be useful info for the future! Anywhere I see 'stray' text is probably universally unimportant in similar sections.
There is a global unit count, but sometimes there are a few more UNIT sections than that. When is a UNIT not a unit? I guess I'll find out. Maybe the global unit count doesn't include barbs? Just speculating.
For now, I have a 'suede' module that lists the global city and unit count (as in the GAME header) and counts the UNIT sections. Next I'll start poking more at UNIT to identify barbs and unit type. Oh, that reminds me, there is a per-civ unit count in the LEAD sections...I'll have to see if the barb 'civ' keeps that number, too.
Code:
25 81 86 large Henry of the Portuguese, 2590 BC.SAV
206 1180 1210 megasigh Tokugawa of the Japanese, 800 AD.SAV
206 1060 1092 metallurgy Tokugawa of the Japanese, 600 AD.SAV
206 1085 1115 monarchy Tokugawa of the Japanese, 700 AD.SAV
0 19 20 nice start Lincoln of the Americans, 4000 BC.SAV
206 1085 1115 sell happy Tokugawa of the Japanese, 680 AD.SAV
Huh, that output is from a script only showing where unit count doesn't match the number of UNIT sections, and for the first time I see that the UNIT count is lower than unit count in one of the saves...odd. (Columns are city count, unit count, count of UNIT sections, save name.)
SpoilerLua script for the above output :
This is/will-be the user-scriptable part of clua.
Code:
function lpad(s, l, c)
local res = string.rep(c or ' ', l - #s) .. s
return res
end
function mass_scan()
bic.load_default()
foo = get_savs({install_path .. "/Saves/Auto", install_path .. "/Saves"})
for _, v in pairs(foo) do
sav.load(v)
if suede.unit_sections ~= suede.unit_count then
io.write(lpad(tostring(suede.city_count), 4))
io.write(lpad(tostring(suede.unit_count), 5))
io.write(lpad(tostring(suede.unit_sections), 6))
io.write(' ', save_name,'\n')
end
end
end
mass_scan()
Wow, so I made a lot of progress with clua yesterday. LEAD, PRTO, RACE, and UNIT sections are being parsed into Lua and have modules with good info available for them. GAME exists but only gives global city and unit counts so far.
I was able with that to batch-scan all my saves and made a Google spreadsheet (well, csv) with the results. It seems to agree that barb counts seem to jump at city count intervals, but more saves need to be scanned, and perhaps map sizes need to be taken into consideration.
SpoilerLua code to make the csv :
This works for the default bic. With some more work it could be able to pull the barb tier 1 and tier 2 unit info from the Bic itself rather than hard-code prto_id's 6 and 11,
Code:
-- left padding for easy terminal viewing
function lpad(s, l, c)
local res = string.rep(c or ' ', l - #s) .. s
return res
end
bic.load_default()
io.write("\"cities\",\"bar wars\",\"barb horses\",\"save name\"\n")
civ3_save_files = get_savs({
install_path .. "/Saves/Auto",
install_path .. "/Saves",
})
for _, v in pairs(civ3_save_files) do
sav.load(v)
local barb_horseman = 0
local barb_warrior = 0
for _, v in ipairs(unit) do
if v.civ_id == 0 and v.prto_id == 6 then
barb_warrior = barb_warrior + 1
end
if v.civ_id == 0 and v.prto_id == 11 then
barb_horseman = barb_horseman + 1
end
end
io.write(lpad(tostring(game.city_count), 4))
io.write(',', lpad(tostring(barb_warrior), 4))
io.write(',', lpad(tostring(barb_horseman), 4))
io.write(', \"', save_name,'\"\n')
end
That stray string data caused problems with parsing the unit sections. The civ3 text files include section-header-like strings like UNIT, BLDG, and PRTO in particular. The latter two happen early enough in the bic that it hasn't been a problem for me so far, but unit sections are very late in the save, so all those stray "UNIT"s in the uninitialized spaces mess with the search strategy. For now I'm searching for the 64th IDLS section and then looking for the first "UNIT" after that, and then I hard-increment 536 bytes between units and use the global unit count to decide how many to read. I think that strategy is safe, but I don't know yet.
So the stray "UNIT"s explain why I saw more "UNIT"s than the global unit count in many saves, but I don't yet know why some saves would report fewer "UNIT"s than global unit count. I might have to do with how I'm parsing and comparing; e.g. maybe the last value before a proper "UNIT" happens to be the ASCII code for a capital letter, but I'll have to dig into those specific saves to find out.
The Lua code is not at feature-parity yet with the cia3 code, but it could get there quickly. I now think I can change cia3 to use lua and make it simpler.
All the data cia3 uses is tabular in structure, even the map. So the Lua code could just spit out text tables or CSVs. If I stick with the web browser UI, that will make the display code much simpler.
While this has come a long way in a short time, I still don't have the ability to diff/compare unknown values in an automated way. And at the moment I'm having trouble figuring out how to go about that. Maybe just diff hex dumps? Either that or just straight byte arrays. Maybe both, and maybe they can be enabled/disabled by a Lua call.
Edit: Ok, for the hex and byte array dumps I'll just use functions.
I started looking more into OCN. I haven't understood it well, and one of my hangups was trying to find the 'real' OCN based on custom map sizes. But I found out (again) there are no custom map sizes. The available map sizes are right there in every Bic, and each has an OCN specified, and each difficulty has a % modifier for OCN. I'm getting a bit ahead of myself, but figuring out corruption suddenly seems obtainable, in the manner of listing each city's corruption and waste and whether the next shield will be corrupt or not.
Speaking of cities, that has always looked hard. So did units until yesterday. And I'm pretty confident I have the global city count right, so that should make figuring out how to untangle all the CITY sections more doable.
Also, I started thinking about some of the bit masks and whether I should expand them into boolean arrays. But I think just keeping them as bit masks and using bitwise operators in Lua should b simple enough...except I found out the Lua I'm using has no bitwise functions. So I'll probably partially implement Lua 5.2's bit32 module for clua.
Speaking of cities, that has always looked hard. So did units until yesterday. And I'm pretty confident I have the global city count right, so that should make figuring out how to untangle all the CITY sections more doable.
Holy crap, OMGWTFBBQ. There appear to be 91 "CITY" section headers per city.
There are some outliers, but they are custom bic games I downloaded. One has the global city count at 1 which is wrong, so I guess that value can move around depending on the bic. Huh, I guess that could be the case with the other...well, hard to tell. The city count is increasing and the "CITY" section/city count ratio is steady.
Anyway...91??? WTF? I keep expecting to find that I massively goofed up somewhere in the counting, but I haven't found such a goof yet.
I wasn't even sure where to go with that next, but I guess the next step is to see if there's a constant byte count between every 91st "CITY" section. That would be the first domino to fall (or the second). I'm pretty sure that's not the case, though, as I believe I've observed before there's at least a "CTZN" for each citizen which would not be a constant number. Buildings are another thing cities can have varying amounts of; I'll have to see if those are bitm asks or arrays...a bit mask would probably make more sense, but we'll see.
Edit: Ok, looks like 84 of the "CITY"s are back-to-back 4-byte-long sections. There are 83 building types in the default game, so a guess is this is the build date of the city itself and any building present. Culture might be another candidate. But it would help explain why so many, and why it differs in custom scenarios.
Edit 2: Ah, no it's a list! The first 4-byte CITY is an integer 83, so that's the list header with count for the 83 other 4-byte CITYs. W00t!
I've been getting more done. I can now parse the whole city list! Antal1987's dumps suggest there is a corruption value in the city, so I may be close to my "will the next shield(s) be wasted" answers without having to figure out the entire corruption calculation.
The ability of the Lua code is way, way ahead of the documentation, and I haven't even posted a binary release yet because I'm adding features daily.
Yesterday I tried and succeeded in replicating my trade report and diplomacy report from cia3 in Lua code. In fact it's already better as it will list the at-war/won't-talk civs but show that they won't talk. There are a couple more things to replicate before I can start migrating cia3 from GraphQL to Lua behind-the-scenes, but it shouldn't be hard.
I had been pondering a while how to tell if a civ is eliminated, and I was thinking today about how to look for it, and then it struck me: if a civ has 0 cities and 0 units, they're eliminated! I don't know if that's what the game keys on, but it should be good enough for cia3, and I have those per-civ values exposed in Lua already.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
.
