What I'm Up To

[Quintillus]

I'm back on C7 stuff (while using my newly discovered mono tools), and I realized I've probably been pretty dumb. We don't have to name the C7 file .json, especially for the static "new game" save. I think if we name it something Godot recognizes, like .gd or .gdscript, it will throw it into the PCK file and we can find it reliably on any platform. (Possible complications, but I think it's doable.) It shouldn't try to run the .gd file because it won't be assigned to any nodes.

Also, I just stumbled on the user:// file reference in Godot. That is probably where we'll want natively saved files to go. In Windows this goes to the user's roaming AppData folder, and in other platforms it may go to a .godot folder in the home directory, or wherever user data is supposed to go on that platform

....

https://godotengine.org/qa/52307/can-you-read-a-textfile-resource-with-gdscript
https://github.com/godotengine/godot/blob/master/scene/resources/text_file.cpp
https://www.google.com/search?q=godot+load+text+resource

You're probably right about this, but it seems a bit odd that you can't throw just any old file in a PCK bundle. Doesn't that limit Godot's flexibility a bit? Oh well, it's no more of a hack than renaming .exe files to .zzz files so you can GMail them to your friends (something I used to do all the time).

Thoughts on the user:// and where to save natively supported files. It's a mess on Windows as to where games put their saves. Locations I have seen many games use include:

- Their own save folder, within their install folder; Civ 3 is such a game. Officially discouraged by Microsoft, has been for decades, but never really caused any problems until Windows Vista, where it caused a lot of headaches. The obvious appeal was it was easy to find the saves if you knew where the game was installed.
- C:\Users\<username>\AppData\Local - Cities: Skylines, Burnout Paradise, etc. use this location
- C:\Users\<username>\AppData\LocalLow - A ton of games use this location. Offworld Trading Company (from Soren's company Mohawk Games) is one example
- C:\Users\<username>\AppData\Roaming - Not as many games as the above, but still a good amount. Factorio being one example. Godot also uses this location for its local configuration.
- My Documents - A whole bunch of games. Empire Earth II, all Paradox games since Victoria II, etc. IMO this is not a great option as it clutters up the My Documents folder which is intended to be used for storing the user's documents. IMO, a save game folder does not belong at the top level here. I no longer store my documents in the My Documents folder as it got overrun with game-specific folders.
- My Documents\My Games - A bunch of games, such as Civ IV through Civ VI, Galactic Civilizations II, etc. IMO this is the best option, as it makes it easy to find the save games/mod files/etc., but doesn't take over the top-level My Documents folder if the user installs a bunch of games.
- C:\Users\<username>\Documents, or C:\Users\<username>\Documents\My Games - A few games such as Van Helsing, plus a few tools such as Visual Studio 2010. These games don't use the system variable for referencing where My Documents lives, although they do for the user folder. Don't be one of those developers (looking at you, Microsoft!).
- C:\Users\<username> - Really it's development tools and cross-platform, Linux-first programs such as GIMP that put their config here, i.e. C:\Users\<username>\.gimp-2.8 . IMO, while not "native", this is preferable to having to dig through the AppData folders. Few users know to check here, but fewer still know to check in AppData, let alone its various sub-folders.

From a user standpoint, I have no idea what should be in Local/LocalLow/Roaming, and there seems to be no rhyme or reason as to what goes in each location. MyDocuments\My Games makes sense from a user standpoint.

From a developer standpoint, I don't know what Local/LocalLow/Roaming should be used for either. I could probably look it up, but I don't think that would help our users figure out where their save files are.

Does it show that I find AppData, and how it's used inconsistently by third-party (and sometimes even first-party) developers, to be one of the most annoying parts of the Windows platform?

Oops, I branched my current feature branch from MountainsAndHills I just noticed.
17f3bed5f45172f4a43a942c5e0ea104b2ad81be in particular. Hopefully that's going to get merged in the not too distant future. It looks like it's working to me, and it looks good.

I started off just making a console app for Lua to drive QueryCiv3 against a number of files, but now I'm taking what I'm learning and modifying QueryCiv3 some, but nothing we're currently using.

I was searching hard for a byte that tells me if the bic has custom rules and/or a custom map, then I realized I'm a dummy and am now just keying on whether BLDG or WCHR exist, respectively.

That can pretty much be merged at any time. I'd been keeping it open as I (slowly, due to moving plans coming together this afternoon and recovery from my booster shot's side effects on Monday) work on the finer points of mountainous terrain. But you're right, it's not so much a half-baked cake as a cake that doesn't have the icing yet, but would still taste fine and have a good texture.

You are correct that there are no bytes for custom rules/map/player data, it's just based on whether those sections of the file exist. I thought those bytes existed too, but that's because I added flags for them in my editor code for ease of reference. I just read the first header, and if it's BLDG, there's custom rules. If it's not, I check if it's WCHR, etc.

You're welcome. Now I want to play some SMAC again

It's got me comparing the two. Jungles in civ3 are always worthless as a jungle. In SMAC, xenofungus is a huge PITA early game not unlike jungles, only it's worse as the "barbs" native life in SMAC use xenofungus like roads and are more or less invisible in them until they're in your zone of control. But you can follow a tech tree route (and societal choices analogous to government type or the multi-option stuff you have in Civ4) that makes xenofungus very productive late game.

I like that compared to civ3 jungles and marshes. You can still clear xenofungus and terraform like crazy, but it tends to make the native life more aggressive when you do that. Interesting mechanics.

Having visible terrain elevation is interesting, too.


Some of you seem to have networks of games-knowledgeable people. Aside from the name recognition of the Civilization series, I think a huge part of Civ3's resurgence is that it's available on Steam. Sure, SMAC is on GoG and cheap, but I think if it were on Steam at the same price, more people would notice it and pick it up. It doesn't have "Civilization" in the name, but it is branded "Sid Meier's". Does anyone have a chain of friends that could poke someone to see about putting SMAC on Steam?

You're making me want to dive into SMAC. Which might endanger my progress on C7. Then again, I've also been pondering jumping into Call to Power 2 in recent days. Maybe C7 is making me think more about other 4X games.

I don't have such friends, but I do suspect Civ III's availability on Steam at prices lower than the average game on GOG, couple with regular sales, has likely provided a significant boost to its user base over the past decade.

Looking at the publishers, I see:

- CivIII and later are Infogrames Interactive/Atari/2K, which all roles up to 2K today. 2K doesn't have its own game store, and puts its stuff on Steam, and a good chunk of its older stuff on GOG.
- SMAC is published by Electronic Arts. They want you to buy on Origin, although they put some stuff on Steam as well. They also are pretty decent about putting older stuff on GOG, if it still runs okay on modern systems. They don't seem to put their own older stuff on Origin for some reason. Maybe GOG gives them a higher cut if it's only on GOG, and they figure their old-school game audience is more likely to be on GOG than Origin?
- Civ II is published by Microprose, which I believe rolled up into Atari/2K. But on PS/2, it was published by Activision, so maybe it would up with Activision/Blizzard, which has their own store as well. AFAIK you can't legally buy it anywhere other than a disc on eBay/your local secondhand store.
- Call to Power II is published by Activision, and is on GOG, which suggests Activision is not opposed to GOG as a venue for their old games. So where did Civ II fall through the cracks?

Perhaps a relevant question is does the CD version of SMAC/CTP II work on modern systems as-is? If GOG did some work to iron out gremlins on modern systems, that could be why they are only on GOG. I know they do such work on some older games.

But yes, it does seem reasonable that if Electronic Arts wanted to maximize their SMAC revenue, it would make sense to have it on Steam and Origin in addition to GOG. Whether that's enough of a drop to register on their bucket... I wouldn't guess so, but I'm not sure I would have guessed GOG sales would have either. Maybe the GOG release was part of their PR department's initiatives to make them seem less evil.

A random idea immediately relevant to C7: We could allow comment-only lines in JSON. This just occurred to me. Comments aren't allowed in JSON proper, but what's to prevent us from running a C7 save through a "remove comment lines" filter? To allow a comment header.

On the other hand, maybe we can just lay out the JSON in a way that's self-documenting, with the map tiles near the bottom and the player count, game settings, number of human players, and such near the top of the file. Maybe have a field for player notes or just some sort of freeform text in case the player wants to put a note in the game to remind them of what it is about in the future. HoF run or something, not that I expect a C7 HoF anytime soon.

This makes me ponder, what is the main point of having the save be in a human readable format? Is it for humans to manually fiddle with the save? Or for third party tools to be able to make use of it easily, and without relying on Apolyton documentation from 2004 that covers a binary format?

I'm sure there's some people who will want to do it manually, and that could be really useful if you accidentally made the Medieval Infantry have an attack of 44 instead of 4 in your mod, and only realize it when one shows up at your border. But thinking about the times I've opened a Europa Universalis IV save and realized the enormity of the data, maybe the latter is at least as relevant? I think I did try to manually modify things once or twice (probably ending a permanent war in EU3, before automatic white peaces were implemented), but never with a lot of confidence.

 

[Puppeteer]

Local and roaming Windows AppData folders behaved differently if you had a remote/roaming profile like some companies used to do before cloud drives became the standard. No idea why there is also a LocalLow; the previous is from ancient experience.

The nice thing about user:// is that it would take platform detection out of our hands. Or the bad thing, depending on your point of view. If we're lucky enough to have people use the game, somebody will complain about where the saves are, and at least with user:// we can say "built into Godot that way" instead of defend our reasoning. On the other hand, having a Godot-dependent reference to save files ... oh yeah that's a problem since we're loading from C7Engine. Never mind.

I'm beginning to understand why putting stuff in the PCK and referring to it is hard, and I think Unity is the same way based on some of the comments I've seen...maybe WW can confirm. In Godot, again we're approaching it "wrong". The Godot way is to load everything as a resource from res:// , and res:// is apparently the PCK file. And you don't just put arbitrary stuff there; it's supposed to be stuff that Godot knows what it's for and how to deal with it. So there aren't really levers to pull arbitrary data out of PCK or put it in.

Not that I'm saying we're actually wrong; I just see why the PCK file is not easily accessible as a general-purpose file store. But I do think we need to understand (I'm slowly getting there) that many things that we would want to do in the PCK or with loading files from outside the PCK and putting them into Godot are going to be a bit harder than we think they should be.

I don't think stuffing a JSON into the PCK is a good long-term strategy, but I like the idea of using it as our "new game" thing until we have map generation which doesn't seem like is anytime soon. And for that if I can get that reliably working on all three platforms by renaming it and working around Godot's expectations, I think it's a good (enough) idea.


I presume SMAC and CTP do not work on modern systems. Anything with copy protection won't, or at least the SafeDisc & SecuROM stuff used back in the day. I didn't try it, though; I have a SMAC CD from way back when, but I just bought it on GoG.

You're making me want to dive into SMAC. Which might endanger my progress on C7.

SMAC sucks! There are only seven factions, the xenofungus drives you crazy, and mind worms just bypass all your armor with psy attacks! But seriously, I've been pressing pretty hard on C7 for a week and am turning at least partially to other things, too.

As for human-readable save format, WildWeasel was really adamant about that. I wanted to play with protobuf or flat buffers as a save format, but I see the appeal of human-readable. As far as the json schema right now, I just needed a target into which to import a Civ3 save, and serializing GameData seemed to be the easiest way forward. But I've always imagined per-game data and rules/per-mod references being in separate hierarchies in the save...they kind of have to be to allow modding.

 

[Quintillus]

Interesting, I've never had a remote or roaming profile. Just an LDAP-based one, and once a computer that I could log in to remotely while connected to a VPN (but still using the same account that I'd use in the office). But you're right, saving is (probably) going to be below the Godot level. Hmmm...

You are probably right about the disc versions of SMAC and CTP, too. It's interesting to reflect that many late 90s/early 2000s games won't work due to SafeDisc/SecuROM, and I found out last year that many mid-90s games won't work due to 16-bit installers, or sometimes graphics API changes, or processors being too fast nowadays, or depending on something low-level that isn't accessible anymore. The ones that do work well, or have few/easily resolvable problems, tend to be the ones that are most likely to be on GOG or Steam.

By comparison, I was able to run almost, though not quite, all of those mid-90s games on XP.

I think I have a disc copy of SMAC, and my desktop is on 8.1 without the update that removes SafeDisc/SecuROM (have to be able to play Civ3 Vanilla, after all). After I move, I'll have to see if it works there. Probably more likely than on 10. If not, SMAX is on GOG. I don't know if I can handle just seven factions, though .

I agree that the save format should be human readable. I was just thinking that the intended use cases for that human readability could influence some decisions. Kind of like the guid vs integer vs key discussion - keys are more regular human friendly than integers, which are probably more so than guids. Comments can also be human friendly, but having non-standard JSON, while we could run it through a parser, would have the downside of making it harder for any other tools, written by other humans, to consume the data. Which had me wondering, if comments are "worth it", is there an alternative to JSON that would achieve that, and if so is that alternative "worth it"? E.g. XML would allow comments, but I wouldn't want to switch to XML over JSON for that, at least in most cases.

 

[Puppeteer]

I kinda hate xml. It's the worst of all worlds and doesn't generalizably translate to other formats. Apropos of nothing. Well, I guess XQuery is OK....

I don't know if I can handle just seven factions, though .

SMACX actually doubles that I think. The GoG package has both. I've never played SMACX, though, not even since having the GoG version...I just don't have the hang of the basics, and the expansion seems even more complex with native aerial and "artillery" units.

 

[Puppeteer]

Some thoughts while replying to the 20-year thread in C&C: I suppose if we just want to allow 32-bit graphics we could look for a PNG or WebP image before the PCX.

I'm not sure how modders go about making graphics; if they're using a pixel art editor then this may not be of use, but if they're working in full color then making a palette at the end, they might want to just use the full color image in C7. So maybe they could just put the full color version alongside the PCX version, and C7 looks for PNG/WebP/whatever first?

I see this as an intermediate step towards a future that has a complete C7 native graphics scheme.

At whenever is a good point, you might want to have a glance at Plotinus' "Undiscovered Worlds" project.

Since semi-accidentally getting interested in terrain generation, I've been more open to looking into it more and thinking about how to apply it to C7. I just rediscovered this link and am looking at it. It's a lot to read, and I can't see yet if there is any public code or utility to play with. (I'll probably end up reading it all, eventually.) Do you know if there is public code associated with this?

Ok never mind, there is no public code. I Googled the heck out of this and found that even in recent months he's posted on social media and stated there is no public code when people show interest. Still, the blog looks to be great reading and probably applicable to future C7 terrain generation.

 

[Quintillus]

I have to keep reminding myself that there are two broad categories of graphics:

- Civ colored graphics (units, etc.)
- Non-Civ-colored graphics (cities, terrains, resources, etc.)

The first one is tougher because is depends on a palette to a fair degree to change the civ color. I'll ignore that for now.

For the non-civ-colored ones, I tend to agree that supporting full color would be nice. Use whatever is available, PNG, WebP, 24/32-bit PCX, 8-bit PCX. I could even port my Java library of obsolete Atari/DOS/early Windows formats if we really want to go overkill on graphics format support for no compelling reason.

I also agree that it is a nice intermediate, and one we could support incrementally. I think a really nice proof-of-concept would be to work with a graphic artist such as Kyriakos to add a high-res, high-color city set to C7. It's smaller than a whole terrain set, but enough to show a difference in detail, and non-civ-colored, so it bypasses potential tripping points there.

I'm not sure exactly how modders go about making graphics either, although for the little programmer art I've done, I've largely followed a guide Kyriakos wrote around 2007 for using GIMP with Civ3 graphics (at the time, GIMP 2.4.7 with a Greek interface, IIRC... the problem now is the menus got rearranged in 2.6; the Greek wasn't much of a barrier). But I've read about other modders using PaintShopPro, and I'm sure there are other preferred programs, such as Blender for units; I also see tutorial threads for Poser and Bryce, though I'm not sure if those are still used often in the present day. Still, I suspect this is an area where there may be some enthusiasm from the artist community to showcase higher-res versions of their work in a proof-of-concept.

(Edit: zo-zo wrote a similar guide in 2011 that shows how to do it in GIMP 2.6+. Also, alas, the images in Kyriakos's 2007 post no longer work)

(Edit 2: Kyriakos's guide is preserved at the Wayback Machine, with images. I now have followed the most important step by playing some Metallica)

One other consideration is alpha transparency. In theory this should be fairly easy in formats that support alpha natively, and nicer than reserving a couple slots of a PCX palette for that. But it's still worth a mention IMO.

I may have to read up on Undiscovered Worlds as well.

 

[Puppeteer]

A discussion in Discord plus my recent work on the "save format" gave me a new framework for approaching the SAV (and BIC) files: read them in serially, but use the ASCII headers as a type discriminator.

This almost has to be what Civ3 is doing as I've seen multiple examples of there being absolutely no forewarning whether a certain thing exists in the file or how many of them there are. And it would explain why the 4-char headers exist in the first place as that *never* made sense to me from a tech point of view. Unless they're a type discriminator.

This is more or less how I started trying to read the file in way back, but I didn't really think of it that way. I looked at it as "Ok, CIV3 first, then BIC, then BLDG" or whatever, but sometimes things just aren't there or the count of them isn't where you might expect to find it.

I'm not entirely sure if this new idea is useful, though. It might be if I wanted to actually write the SAV back out again so that Civ3 could read it, but that has never been an interest of mine so far. And it would still involve writing a bunch of extra code for that. The type discriminator framework might be more compute and/or memory efficient, but we're not doing enough with the save file to make that even worth the bother at this scale.

Plus, I'm increasingly enamored with the convenience getter construct I've come up with, and it appears to be relatively memory efficient if I'm understanding how C# handles the data references.

 

[Quintillus]

I still need to dig my Discord sign-in off of my laptop with a semi-broken screen. Which is better than my laptop with a broken screen in some ways, but less good in others (I have parts for fixing the one that's all-the-way broken)

I'm 99% sure that how Civ works is:

1. Read a header. Is it one that I can work with? Yes? Cool. No? Throw an error.
2. Read the data length.
3. Read however many bytes the data length says I should read. Process them according to the header.
4. Repeat from step 1, if we haven't reached the end of the file.

The reason is that one of the recurring errors I made in my editor early on was updating how much data a section had (e.g. adding another city name to the RACE section), and forgetting to update the data length for that section. My editor could open the resulting BIQ as it just read in whatever data the section should have, regardless of the data length parameter, but Civ3 would invariably blow up. This lead to being more disciplined around having setters/adders that integrated updating the data length variable, as well as having some integration tests to try to catch the issue, and it's been years since I introduced one of those bugs. But in the first few years, they were a recurring annoyance.

The headers are still useful in this model as depending on the structure of the file, you may or may not have e.g. a BLDG section. If Civ3 sees a BLDG section, it knows there are custom rules. If it gets WCHR instead, default rules are used but there is a custom world map.

 

[Puppeteer]

I think we're on the same page. That means the header is a type discriminator. Not all sections have a length, though; some have a count, and some have neither. At least in the SAV. And then there is definitely data without headers here and there. But yeah, the reader must see the header and then expect a certain data format based on that.

It might be fun to try to rearrange sections, but not actually fun, just tedious and informative, but possibly not worth the effort unless otherwise motivated.

Repeat from step 1, if we haven't reached the end of the file.

I recall pretty recently experimenting with adding arbitrary data to the end of either a BIQ or SAV, and Civ3 didn't crash or error. Must've been a BIQ because we can't rely on extra data propagating from one turn to the next in a SAV, but it would live long enough in a BIQ for Civ3X to do something with it.

So I must conclude that for the BIQ at least at some point it just says "oh, that's all I need, I'll stop reading." Or maybe it does error internally then go "actually, I have all the data I could want, let's just ignore that and move on."


Which reminds me, I had been speculating that we could inject arbitrary data into a SAV that we could persist across turns by stuffing it into "unused" TILE bytes as I believe there is vestigial space left over from vanilla and PTW, that successive patches and editions just added new data without reclaiming the space used for e.g. PTW terrain replaced by C3C terrain.

Well, I haven't tried that, but I see possible evidence that Civ3 is already doing that! When using the heat map to examine tile bytes, there are a couple of offsets that only have a little data starting at the top left tile, lasting a few rows, and then everything else is zeroes. That sure as heck doesn't look like map data; it's what I think my idea would look like if I tried it. On the other hand, the data doesn't really look like any pattern I've seen before. When I get around to investigating more I'll have to see if its serialized length varies with turns in the game, size of the map, stuff like that.

I don't recall if I mentioned that finding here...maybe I did.

 

[Quintillus]

Interesting re: arbitrary data being acceptable at the end of a BIQ/SAV. I hadn't tried that yet, nor rearranging sections, since I agree that it would be tedious for no apparent benefit.

It is possible to inject arbitrary data in unused parts of the BIQ, and have it persisted by Civ into SAV files, and to do the same with SAV files. I use this functionality in my editor to mark if the user has used the SAV modification functionality, so that if they do so on a Hall of Fame/GOTM save, the submission will be rejected. The PTW fields would likely be valid places for that; another one is after any null-terminated string. You can see this in my editor, where it shows the full data for null-terminated strings on the ERA tab, for scientist types by era. Civ never shows anything after the null, but the data sticks around. For most scenarios, the largest such area would be the Scenario Search Path, which is over 5200 characters long, but rarely if ever exceeds 200 characters in practice. You could fit a moderate sized event script in that area if you wanted.

It makes sense when considering Civ's C++ lineage. Freeing memory doesn't mean zeroing out its data, strings end whenever there is a null, and you're reading everything into fixed-length buffers, so whenever there's extra space in that buffer, it can be used for something else. Of course that's also why we have limits on things such as city name length - put one too many characters in there, and you've got a buffer overflow, something I've tested a couple times in the hex editor just for fun.