1. Firaxis celebrates the "Asian American and Pacific Islander Heritage Month", and offers a give-away of a Civ6 anthology copy (5 in total)! For all the details, please check the thread here. .
    Dismiss Notice
  2. Old World has finally been released on GOG and Steam, besides also being available in the Epic store . Come to our Old World forum and discuss with us!
    Dismiss Notice

What I'm Up To

Discussion in 'Civ3 Future Development' started by Puppeteer, Jan 14, 2021.

  1. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    7,166
    Location:
    Ohio
    You're probably right about this, but it seems a bit odd that you can't throw just any old file in a PCK bundle. Doesn't that limit Godot's flexibility a bit? Oh well, it's no more of a hack than renaming .exe files to .zzz files so you can GMail them to your friends (something I used to do all the time).

    Thoughts on the user:// and where to save natively supported files. It's a mess on Windows as to where games put their saves. Locations I have seen many games use include:

    - Their own save folder, within their install folder; Civ 3 is such a game. Officially discouraged by Microsoft, has been for decades, but never really caused any problems until Windows Vista, where it caused a lot of headaches. The obvious appeal was it was easy to find the saves if you knew where the game was installed.
    - C:\Users\<username>\AppData\Local - Cities: Skylines, Burnout Paradise, etc. use this location
    - C:\Users\<username>\AppData\LocalLow - A ton of games use this location. Offworld Trading Company (from Soren's company Mohawk Games) is one example
    - C:\Users\<username>\AppData\Roaming - Not as many games as the above, but still a good amount. Factorio being one example. Godot also uses this location for its local configuration.
    - My Documents - A whole bunch of games. Empire Earth II, all Paradox games since Victoria II, etc. IMO this is not a great option as it clutters up the My Documents folder which is intended to be used for storing the user's documents. IMO, a save game folder does not belong at the top level here. I no longer store my documents in the My Documents folder as it got overrun with game-specific folders.
    - My Documents\My Games - A bunch of games, such as Civ IV through Civ VI, Galactic Civilizations II, etc. IMO this is the best option, as it makes it easy to find the save games/mod files/etc., but doesn't take over the top-level My Documents folder if the user installs a bunch of games.
    - C:\Users\<username>\Documents, or C:\Users\<username>\Documents\My Games - A few games such as Van Helsing, plus a few tools such as Visual Studio 2010. These games don't use the system variable for referencing where My Documents lives, although they do for the user folder. Don't be one of those developers (looking at you, Microsoft!).
    - C:\Users\<username> - Really it's development tools and cross-platform, Linux-first programs such as GIMP that put their config here, i.e. C:\Users\<username>\.gimp-2.8 . IMO, while not "native", this is preferable to having to dig through the AppData folders. Few users know to check here, but fewer still know to check in AppData, let alone its various sub-folders.

    From a user standpoint, I have no idea what should be in Local/LocalLow/Roaming, and there seems to be no rhyme or reason as to what goes in each location. MyDocuments\My Games makes sense from a user standpoint.

    From a developer standpoint, I don't know what Local/LocalLow/Roaming should be used for either. I could probably look it up, but I don't think that would help our users figure out where their save files are.

    Does it show that I find AppData, and how it's used inconsistently by third-party (and sometimes even first-party) developers, to be one of the most annoying parts of the Windows platform?

    That can pretty much be merged at any time. I'd been keeping it open as I (slowly, due to moving plans coming together this afternoon and recovery from my booster shot's side effects on Monday) work on the finer points of mountainous terrain. But you're right, it's not so much a half-baked cake as a cake that doesn't have the icing yet, but would still taste fine and have a good texture.

    You are correct that there are no bytes for custom rules/map/player data, it's just based on whether those sections of the file exist. I thought those bytes existed too, but that's because I added flags for them in my editor code for ease of reference. I just read the first header, and if it's BLDG, there's custom rules. If it's not, I check if it's WCHR, etc.

    You're making me want to dive into SMAC. Which might endanger my progress on C7. Then again, I've also been pondering jumping into Call to Power 2 in recent days. Maybe C7 is making me think more about other 4X games.

    I don't have such friends, but I do suspect Civ III's availability on Steam at prices lower than the average game on GOG, couple with regular sales, has likely provided a significant boost to its user base over the past decade.

    Looking at the publishers, I see:

    - CivIII and later are Infogrames Interactive/Atari/2K, which all roles up to 2K today. 2K doesn't have its own game store, and puts its stuff on Steam, and a good chunk of its older stuff on GOG.
    - SMAC is published by Electronic Arts. They want you to buy on Origin, although they put some stuff on Steam as well. They also are pretty decent about putting older stuff on GOG, if it still runs okay on modern systems. They don't seem to put their own older stuff on Origin for some reason. Maybe GOG gives them a higher cut if it's only on GOG, and they figure their old-school game audience is more likely to be on GOG than Origin?
    - Civ II is published by Microprose, which I believe rolled up into Atari/2K. But on PS/2, it was published by Activision, so maybe it would up with Activision/Blizzard, which has their own store as well. AFAIK you can't legally buy it anywhere other than a disc on eBay/your local secondhand store.
    - Call to Power II is published by Activision, and is on GOG, which suggests Activision is not opposed to GOG as a venue for their old games. So where did Civ II fall through the cracks?

    Perhaps a relevant question is does the CD version of SMAC/CTP II work on modern systems as-is? If GOG did some work to iron out gremlins on modern systems, that could be why they are only on GOG. I know they do such work on some older games.

    But yes, it does seem reasonable that if Electronic Arts wanted to maximize their SMAC revenue, it would make sense to have it on Steam and Origin in addition to GOG. Whether that's enough of a drop to register on their bucket... I wouldn't guess so, but I'm not sure I would have guessed GOG sales would have either. Maybe the GOG release was part of their PR department's initiatives to make them seem less evil.

    This makes me ponder, what is the main point of having the save be in a human readable format? Is it for humans to manually fiddle with the save? Or for third party tools to be able to make use of it easily, and without relying on Apolyton documentation from 2004 that covers a binary format?

    I'm sure there's some people who will want to do it manually, and that could be really useful if you accidentally made the Medieval Infantry have an attack of 44 instead of 4 in your mod, and only realize it when one shows up at your border. But thinking about the times I've opened a Europa Universalis IV save and realized the enormity of the data, maybe the latter is at least as relevant? I think I did try to manually modify things once or twice (probably ending a permanent war in EU3, before automatic white peaces were implemented), but never with a lot of confidence.
     
  2. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Local and roaming Windows AppData folders behaved differently if you had a remote/roaming profile like some companies used to do before cloud drives became the standard. No idea why there is also a LocalLow; the previous is from ancient experience.

    The nice thing about user:// is that it would take platform detection out of our hands. Or the bad thing, depending on your point of view. If we're lucky enough to have people use the game, somebody will complain about where the saves are, and at least with user:// we can say "built into Godot that way" instead of defend our reasoning. :) On the other hand, having a Godot-dependent reference to save files ... oh yeah that's a problem since we're loading from C7Engine. Never mind.

    I'm beginning to understand why putting stuff in the PCK and referring to it is hard, and I think Unity is the same way based on some of the comments I've seen...maybe WW can confirm. In Godot, again we're approaching it "wrong". The Godot way is to load everything as a resource from res:// , and res:// is apparently the PCK file. And you don't just put arbitrary stuff there; it's supposed to be stuff that Godot knows what it's for and how to deal with it. So there aren't really levers to pull arbitrary data out of PCK or put it in.

    Not that I'm saying we're actually wrong; I just see why the PCK file is not easily accessible as a general-purpose file store. But I do think we need to understand (I'm slowly getting there) that many things that we would want to do in the PCK or with loading files from outside the PCK and putting them into Godot are going to be a bit harder than we think they should be.

    I don't think stuffing a JSON into the PCK is a good long-term strategy, but I like the idea of using it as our "new game" thing until we have map generation which doesn't seem like is anytime soon. And for that if I can get that reliably working on all three platforms by renaming it and working around Godot's expectations, I think it's a good (enough) idea.


    I presume SMAC and CTP do not work on modern systems. Anything with copy protection won't, or at least the SafeDisc & SecuROM stuff used back in the day. I didn't try it, though; I have a SMAC CD from way back when, but I just bought it on GoG.

    SMAC sucks! There are only seven factions, the xenofungus drives you crazy, and mind worms just bypass all your armor with psy attacks! :mischief: But seriously, I've been pressing pretty hard on C7 for a week and am turning at least partially to other things, too.

    As for human-readable save format, WildWeasel was really adamant about that. I wanted to play with protobuf or flat buffers as a save format, but I see the appeal of human-readable. As far as the json schema right now, I just needed a target into which to import a Civ3 save, and serializing GameData seemed to be the easiest way forward. But I've always imagined per-game data and rules/per-mod references being in separate hierarchies in the save...they kind of have to be to allow modding.
     
  3. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    7,166
    Location:
    Ohio
    Interesting, I've never had a remote or roaming profile. Just an LDAP-based one, and once a computer that I could log in to remotely while connected to a VPN (but still using the same account that I'd use in the office). But you're right, saving is (probably) going to be below the Godot level. Hmmm...

    You are probably right about the disc versions of SMAC and CTP, too. It's interesting to reflect that many late 90s/early 2000s games won't work due to SafeDisc/SecuROM, and I found out last year that many mid-90s games won't work due to 16-bit installers, or sometimes graphics API changes, or processors being too fast nowadays, or depending on something low-level that isn't accessible anymore. The ones that do work well, or have few/easily resolvable problems, tend to be the ones that are most likely to be on GOG or Steam.

    By comparison, I was able to run almost, though not quite, all of those mid-90s games on XP.

    I think I have a disc copy of SMAC, and my desktop is on 8.1 without the update that removes SafeDisc/SecuROM (have to be able to play Civ3 Vanilla, after all). After I move, I'll have to see if it works there. Probably more likely than on 10. If not, SMAX is on GOG. I don't know if I can handle just seven factions, though :).

    I agree that the save format should be human readable. I was just thinking that the intended use cases for that human readability could influence some decisions. Kind of like the guid vs integer vs key discussion - keys are more regular human friendly than integers, which are probably more so than guids. Comments can also be human friendly, but having non-standard JSON, while we could run it through a parser, would have the downside of making it harder for any other tools, written by other humans, to consume the data. Which had me wondering, if comments are "worth it", is there an alternative to JSON that would achieve that, and if so is that alternative "worth it"? E.g. XML would allow comments, but I wouldn't want to switch to XML over JSON for that, at least in most cases.
     
  4. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    I kinda hate xml. It's the worst of all worlds and doesn't generalizably translate to other formats. Apropos of nothing. Well, I guess XQuery is OK....

    SMACX actually doubles that I think. The GoG package has both. I've never played SMACX, though, not even since having the GoG version...I just don't have the hang of the basics, and the expansion seems even more complex with native aerial and "artillery" units.
     
  5. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Some thoughts while replying to the 20-year thread in C&C: I suppose if we just want to allow 32-bit graphics we could look for a PNG or WebP image before the PCX.

    I'm not sure how modders go about making graphics; if they're using a pixel art editor then this may not be of use, but if they're working in full color then making a palette at the end, they might want to just use the full color image in C7. So maybe they could just put the full color version alongside the PCX version, and C7 looks for PNG/WebP/whatever first?

    I see this as an intermediate step towards a future that has a complete C7 native graphics scheme.

    Since semi-accidentally getting interested in terrain generation, I've been more open to looking into it more and thinking about how to apply it to C7. I just rediscovered this link and am looking at it. It's a lot to read, and I can't see yet if there is any public code or utility to play with. (I'll probably end up reading it all, eventually.) Do you know if there is public code associated with this?

    Ok never mind, there is no public code. I Googled the heck out of this and found that even in recent months he's posted on social media and stated there is no public code when people show interest. Still, the blog looks to be great reading and probably applicable to future C7 terrain generation.
     
  6. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    7,166
    Location:
    Ohio
    I have to keep reminding myself that there are two broad categories of graphics:

    - Civ colored graphics (units, etc.)
    - Non-Civ-colored graphics (cities, terrains, resources, etc.)

    The first one is tougher because is depends on a palette to a fair degree to change the civ color. I'll ignore that for now.

    For the non-civ-colored ones, I tend to agree that supporting full color would be nice. Use whatever is available, PNG, WebP, 24/32-bit PCX, 8-bit PCX. I could even port my Java library of obsolete Atari/DOS/early Windows formats if we really want to go overkill on graphics format support for no compelling reason.

    I also agree that it is a nice intermediate, and one we could support incrementally. I think a really nice proof-of-concept would be to work with a graphic artist such as Kyriakos to add a high-res, high-color city set to C7. It's smaller than a whole terrain set, but enough to show a difference in detail, and non-civ-colored, so it bypasses potential tripping points there.

    I'm not sure exactly how modders go about making graphics either, although for the little programmer art I've done, I've largely followed a guide Kyriakos wrote around 2007 for using GIMP with Civ3 graphics (at the time, GIMP 2.4.7 with a Greek interface, IIRC... the problem now is the menus got rearranged in 2.6; the Greek wasn't much of a barrier). But I've read about other modders using PaintShopPro, and I'm sure there are other preferred programs, such as Blender for units; I also see tutorial threads for Poser and Bryce, though I'm not sure if those are still used often in the present day. Still, I suspect this is an area where there may be some enthusiasm from the artist community to showcase higher-res versions of their work in a proof-of-concept.

    (Edit: zo-zo wrote a similar guide in 2011 that shows how to do it in GIMP 2.6+. Also, alas, the images in Kyriakos's 2007 post no longer work)

    (Edit 2: Kyriakos's guide is preserved at the Wayback Machine, with images. I now have followed the most important step by playing some Metallica)

    One other consideration is alpha transparency. In theory this should be fairly easy in formats that support alpha natively, and nicer than reserving a couple slots of a PCX palette for that. But it's still worth a mention IMO.

    I may have to read up on Undiscovered Worlds as well.
     
    Last edited: Dec 9, 2021
  7. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    A discussion in Discord plus my recent work on the "save format" gave me a new framework for approaching the SAV (and BIC) files: read them in serially, but use the ASCII headers as a type discriminator.

    This almost has to be what Civ3 is doing as I've seen multiple examples of there being absolutely no forewarning whether a certain thing exists in the file or how many of them there are. And it would explain why the 4-char headers exist in the first place as that *never* made sense to me from a tech point of view. Unless they're a type discriminator.

    This is more or less how I started trying to read the file in way back, but I didn't really think of it that way. I looked at it as "Ok, CIV3 first, then BIC, then BLDG" or whatever, but sometimes things just aren't there or the count of them isn't where you might expect to find it.

    I'm not entirely sure if this new idea is useful, though. It might be if I wanted to actually write the SAV back out again so that Civ3 could read it, but that has never been an interest of mine so far. And it would still involve writing a bunch of extra code for that. The type discriminator framework might be more compute and/or memory efficient, but we're not doing enough with the save file to make that even worth the bother at this scale.

    Plus, I'm increasingly enamored with the convenience getter construct I've come up with, and it appears to be relatively memory efficient if I'm understanding how C# handles the data references.
     
  8. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    7,166
    Location:
    Ohio
    I still need to dig my Discord sign-in off of my laptop with a semi-broken screen. Which is better than my laptop with a broken screen in some ways, but less good in others (I have parts for fixing the one that's all-the-way broken)

    I'm 99% sure that how Civ works is:

    1. Read a header. Is it one that I can work with? Yes? Cool. No? Throw an error.
    2. Read the data length.
    3. Read however many bytes the data length says I should read. Process them according to the header.
    4. Repeat from step 1, if we haven't reached the end of the file.

    The reason is that one of the recurring errors I made in my editor early on was updating how much data a section had (e.g. adding another city name to the RACE section), and forgetting to update the data length for that section. My editor could open the resulting BIQ as it just read in whatever data the section should have, regardless of the data length parameter, but Civ3 would invariably blow up. This lead to being more disciplined around having setters/adders that integrated updating the data length variable, as well as having some integration tests to try to catch the issue, and it's been years since I introduced one of those bugs. But in the first few years, they were a recurring annoyance.

    The headers are still useful in this model as depending on the structure of the file, you may or may not have e.g. a BLDG section. If Civ3 sees a BLDG section, it knows there are custom rules. If it gets WCHR instead, default rules are used but there is a custom world map.
     
    Puppeteer likes this.
  9. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    I think we're on the same page. That means the header is a type discriminator. Not all sections have a length, though; some have a count, and some have neither. At least in the SAV. And then there is definitely data without headers here and there. But yeah, the reader must see the header and then expect a certain data format based on that.

    It might be fun to try to rearrange sections, but not actually fun, just tedious and informative, but possibly not worth the effort unless otherwise motivated.

    I recall pretty recently experimenting with adding arbitrary data to the end of either a BIQ or SAV, and Civ3 didn't crash or error. Must've been a BIQ because we can't rely on extra data propagating from one turn to the next in a SAV, but it would live long enough in a BIQ for Civ3X to do something with it.

    So I must conclude that for the BIQ at least at some point it just says "oh, that's all I need, I'll stop reading." Or maybe it does error internally then go "actually, I have all the data I could want, let's just ignore that and move on."


    Which reminds me, I had been speculating that we could inject arbitrary data into a SAV that we could persist across turns by stuffing it into "unused" TILE bytes as I believe there is vestigial space left over from vanilla and PTW, that successive patches and editions just added new data without reclaiming the space used for e.g. PTW terrain replaced by C3C terrain.

    Well, I haven't tried that, but I see possible evidence that Civ3 is already doing that! When using the heat map to examine tile bytes, there are a couple of offsets that only have a little data starting at the top left tile, lasting a few rows, and then everything else is zeroes. That sure as heck doesn't look like map data; it's what I think my idea would look like if I tried it. On the other hand, the data doesn't really look like any pattern I've seen before. When I get around to investigating more I'll have to see if its serialized length varies with turns in the game, size of the map, stuff like that.

    I don't recall if I mentioned that finding here...maybe I did.
     
  10. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    7,166
    Location:
    Ohio
    Interesting re: arbitrary data being acceptable at the end of a BIQ/SAV. I hadn't tried that yet, nor rearranging sections, since I agree that it would be tedious for no apparent benefit.

    It is possible to inject arbitrary data in unused parts of the BIQ, and have it persisted by Civ into SAV files, and to do the same with SAV files. I use this functionality in my editor to mark if the user has used the SAV modification functionality, so that if they do so on a Hall of Fame/GOTM save, the submission will be rejected. The PTW fields would likely be valid places for that; another one is after any null-terminated string. You can see this in my editor, where it shows the full data for null-terminated strings on the ERA tab, for scientist types by era. Civ never shows anything after the null, but the data sticks around. For most scenarios, the largest such area would be the Scenario Search Path, which is over 5200 characters long, but rarely if ever exceeds 200 characters in practice. You could fit a moderate sized event script in that area if you wanted.

    It makes sense when considering Civ's C++ lineage. Freeing memory doesn't mean zeroing out its data, strings end whenever there is a null, and you're reading everything into fixed-length buffers, so whenever there's extra space in that buffer, it can be used for something else. Of course that's also why we have limits on things such as city name length - put one too many characters in there, and you've got a buffer overflow, something I've tested a couple times in the hex editor just for fun.
     
    Puppeteer likes this.

Share This Page