1. Firaxis celebrates the "Asian American and Pacific Islander Heritage Month", and offers a give-away of a Civ6 anthology copy (5 in total)! For all the details, please check the thread here. .
    Dismiss Notice
  2. Old World has finally been released on GOG and Steam, besides also being available in the Epic store . Come to our Old World forum and discuss with us!
    Dismiss Notice

I Believe It's Critical To Integrate .Exe "Patches" With Quintillus' Editor

Discussion in 'Civ3 Future Development' started by Ozymandias, Jun 5, 2021.

  1. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    ... Like the thread title says, IF this path IS our way forward (which I'm 100% on board with, barring getting access to the Holy Grail - I mean, "Source Code") THEN (and - sorry for the Boolean humor - there ain't no "ELSE") -
    1. As @Flintlock is already doing, one proven stable .exe should be released, as such.
    2. Integrating any .exe options (Raze/NoRaze etc.) with - if he's in agreement - @Quintillus' editor provides us with:
      • A way to fully test the .exe's functions and true "plug compatibility" with The Game Itself ...
      • ... MEANING let @Quintillus, @Civinator, @Vuldacon REALLY beat this puppy up ( ... :hmm: - a terrible metaphor ... )
      • ... Thereby providing the equivalent of a full "User Acceptance Test" proof of stability & utility.
    3. This is "IT 101." We do it, and we get full community buy-in.
    What say ye?
     
  2. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    I would welcome integration of C3X with Quintillus' editor. I haven't pursued it yet for two reasons: (1) The mod configuration is still pretty basic. Most of the options are simple on/off switches for convenience features and bug fixes. The only things mod authors would want to change are city razing, railroad movement, and (in the next version) min city distance. On a related note, the INI config is a temporary thing, once the configuration becomes more complex than integers & booleans I'll switch it to a JSON file. (2) The patched EXE can't yet change its configuration for particular scenarios.

    There's a chicken-and-egg problem here where I don't want to spend time working on elaborate engine extensions that won't actually be used by modders but no modder is going to add C3X as a hard dependency unless it has some critical feature they need.
     
  3. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    1. Your "simple switches" can be the key to modders' acceptance & use. I agree with you about simply incorporating "NoRaze." Re: RR movement: I thought that you were making this a variable number (meaning: A way needs to be in place to modify this, and I hope that he could keep "infinite" as an option.) Ditto Y/N for changing minimum city distance.
    2. Using Quint's editor as an interface could (I believe) make this an interface for both pre-release testing, and then carry over directly into simultaneous new releases of C3X with Quintillus' editor.
    QED :)

    Lastly, a question: You mention JSON. Does this imply any chance for scripting?

    PS (added 6/7/21) Perhaps my point is better expressed in traditional IT terms: "Version Control" and, "Code Library." Lastly, I'm certain your ingenuity won't abandon you any time in the next century or so ( :D ) therefore I foresee many more Boolean choices ahead, ergo "ditto" my QED above.
     
    Last edited: Jun 7, 2021
  4. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    Railroad movement is configurable as an integer, I didn't mean to imply that was one of the switches. "Infinite", or in other words the vanilla behavior, is possible by setting the movement rate to zero. I'll do something similar for the min city distance.
    Scripting is possible but it's not really related to using JSON for config. The reason I'd choose JSON is because a few years ago I wrote some code that automatically generates parsers for JSON representations of arbitrary C structs, so it would be easy to drop that into this project and load complex JSON data with a minimum of effort. Whereas if I continued using INI files I'd have to find some other solution, the parser I'm using right now is just something very basic that I threw together in an hour or so. But about scripting, it's totally possible, at least in principle. I can inject any C code I want into the executable, including a Lua interpreter, and in fact I've already done it as a proof of concept (not included in any C3X release, on a separate branch). The hard part would be generating bindings for the scripts to interface with the rest of the EXE.
     
  5. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    ... So, getting back to qubits ( :joke: ) what's your view of "synchronized / integrated" release of your review, hypothetically, with @Quintillus' editor? At risk of repeating, IMHO it remains the the best and most facile way to :
    • Test any Booleans, both "locally," for good ol' fashioned "systems integration."
    • Ditto that ( :hammer: ) "plug compatibility.'
    • Help assure, "user confidence," that each release is backwards compatible.
    • Using a highly regarded like Q's editor would also be far easier (for even the mildly non-technical gamer) than potentially needing to make changes to a .ini file (WHAT'S THAT?? :run: )
    Similarly, I believe it would be the best way to (literally) keep us all on the same page. Example: "Wish Lists" -
    1. Just dotted down in whichever place/thread as is deemed appropriate (this would be for every other C&C "resident," aside rom those who are on The Team.
    2. Collating / compiling / prioritizing them/ whatever, thereby unveiling whichever we deemed best to next address.
    3. You and The Other Coders could keep a list of every other new Fine Discovery you've made, which can then be compared/contrasted/etc. with the "Plebian List."
    4. This could drive prioritizations and perhaps even a list of, "Whatever Excellent Features" might define (nd be announced as) "The next release, thereby building up and maintaining/increasing "audience awareness and participation."
    I spent 20 years or so either directly managing coders, or managing their managers. Almost all were exceptional, and many brilliant. And each and every one of them tended to loathe Specifications, Testing, and planning co-implementations, no matter what type of methodology was employed. But you don't need to involved in any rigorous testing. Just hand a stable .exe plug-compatible with Qs editor, and "we'll" take care of the rest, even inviting the entire community to join in on, "Open Beta Tests" - Or, whatever simply sounds good, and best balances regarding you and our ... :think: - Genius Spelunking Adventures ( :king: ) in that wild, open territory some simply call, ".Exe Patching." :cowboy:
     
  6. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    The big advantage of integration with Quintillus' editor would be that the mod configuration could then be edited with the assistance of a GUI. This is not a major advantage right now because there's so little configuration to do, and most of it comes down to player preferences. I don't see what you're getting at regarding testing or "plug compatibility". For example, the next version of C3X will have adjustable min city distance, autofill gold amount in the trade screen, and hopefully the ability to scroll between civs on the trade screen. Two of those things have nothing to do with an editor, and the other is a simple integer config. Overall I'm fine with integration I just don't think it's worth the trouble right now.
    "Like herding cats" is one of my favorite analogies and it applies very well to game modders. :lol:
     
  7. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    If a Boolean option you've created can, first, be integrated with Quintillus' editor; then, second, said function can be implemented directly into a .biq, then (a) a de facto integration test has been successfully completed (consider: if C3X is - in essence - "Our Way Forward For Now," one unforeseen integration/implementation glitch could cost us a lot of cred, especially if the .exe is not, in some unknowable way, not "backwards compatible" with any of the popular & complex mods.

    Also, consider that the phrase, "a simple integer config" is readily comprehensible to you and me, I assure you that it will make the great majority of eyes glaze over.

    That being said - any guess as to how many more brilliant little Boolean choices you might discover/implement? If (as is the case) we agree about the merits of (extending) Quintillus' editor as the de facto modding GUI, then ... ?

    And I recall the many years of joy getting programmers ( :old: ) to (of course, pre-"downstream" / code library / etc. methodologies) to do multiple layers of testing and documentation ... :whipped:
     
  8. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    I don't know if it's possible to pack additional data into a BIQ beyond what Firaxis intended. I've never had to look into the format. Currently C3X doesn't have problems with backwards compatibility because it doesn't affect the process of BIQ loading or game loading/saving.
    I don't know about future boolean config options. I don't have any concrete plans for what to do beyond R7. I intend to look into properly removing the city limit and fixing the phantom resource bug at some point but I don't know when I'd get around to those things or if they're even possible. I'd like to hear from @Quintillus about how difficult it would be to do this integration, i.e., how difficult it would be to add a new set of options to his editor and have those options be saved to & loaded from a JSON file. I could look into that myself, but I'm busy enough with my own mod and have very little experience with Java.
     
  9. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    Irrespective - you rock! :dance: :thanx: :cheers:
     
    Flintlock likes this.
  10. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    My initial thought was that such integration of Q's editor and C3X is nonsensical and/or impossible from a technical standpoint. I haven't strayed far from that starting point.

    Crazy idea #1 was that the editor could build the .exe, but that still leaves two files to manage and needing to run the correct .exe with the correct .biq. Also, including a build system in the editor...let's call this idea #1 a non-starter.

    Crazy idea #2 was trying to find unused/abandoned space in the .biq. But I was initially thinking of SAV files where I think it might be possible to hide new data in perhaps the histogram data, the map tile data, and perhaps elsewhere as my interpretation of the dumps is that vanilla and PTW data classes updated in C3C are still taking up space. In particular PTW tile terrain data is in a separate place than C3C's, and the PTW data seems to be there but unused. (All zeroes in newly-created C3C games but the original terrain data in PTW games converted to C3C, with the terrain data duplicated in the C3C location.)

    However, the SAV is not the biq. I don't know offhand of any hidden/wasted data structures in the biq data unless perhaps if a map is included in the biq, but that's wholly optional, and I don't know offhand in the tile structure in the biq is identical to the SAV.

    Idea #2 also has the problem of C3X needing to call different functions based on biq data, and I wouldn't presume that's trivial, and C3C's preferred error handling is crash-to-desktop which would make it that much more frustrating to troubleshoot.

    In either case, I'm not sure that such complex integration would be a higher priority than actually adding fixes and features.

    Afterthought: Oh, in the biq: what if there were a building or unit type that was never used in-game, call it "C3X-Config-Do-Not-Use"? And bitmap values in place of real stats for feature toggles. Is it reasonably possible to have a defined unit that nobody can ever build? Or perhaps this hidden "unit" would never be seen in the editor so the user can't mess it up. Still, there are quite a few issues to worry about for which limited effort might be spent elsewhere.
     
  11. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    The worst case scenario is that the editor has to communicate to the modded EXE through a separate file, for example, when saving some scenario the editor would produce Whatever.biq and Whatever.c3x_config.json then the modded EXE would have to know, when loading the scenario, to look for a config file of the same name. Packing the config info into the BIQ is really just a matter of convenience. Also it's not necessary to produce separate EXEs for different configurations, all config options are applied dynamically, there are really only two EXEs for any C3X version, one derived from the GOG executable and another from the Steam one. The problem of having to run different functions depending on the config is solved in the straight forward, unclever manner that the mod always intercepts the function call in question then the replacement code contains in if statement that either runs the original function or the modded one depending on the config. The harder part is when it's necessary to do machine code edits, like for most of the bug fixes, to make those configurable it's necessary to make the C3X EXE self-modifying.

    Hiding data in an unbuildable unit is an interesting possibility. It should be possible to create an unbuildable unit by clearing the "available to" list, or at least I would hope so, I haven't worked with the scenario editor much personally. The problem is that only gives limited room to work with, the UnitType object in game is only about 300 bytes. That would be enough for now but it could easily run out if the config becomes complex, for example a list of 100 unit type IDs, even at only two bytes each, would take up most of the space. The first thing I would try would be to just add some random bytes to the end of a BIQ file and see if the game will still load it.
     
    Puppeteer likes this.
  12. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Interesting. A further thought: Strings are fixed-reserved-length and null-terminated, so I imagine any string field could be hijacked to slip in arbitrary data after the null. That would offer more bytes to play with and more places to hide data.
     
    Flintlock likes this.
  13. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    @Flintlock I've only been distantly following the C3X progress. Are *all* the optional settings game-level settings?

    In other words, there are no per-unit, per-building, per-civ or any other context-conditional "C3X settings", right?

    I suggested in another thread that an after-editor utility might be run to inject the C3X config options, but I had assumed all settings were per-BIQ.
     
  14. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    That's right. I want to leave the option open for more fine grained config settings but there are none right now.

    On a related note, I was thinking of splitting the config options into player preferences and scenario settings, with the idea that player prefs would be set in-game and the rest would be set in an editor or special utility, etc. I think that would be the ideal, but an in-game configuration is not happening any time soon.
     
    Puppeteer likes this.
  15. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Oh cool, that's useful to know. The BIQ and SAV are similar enough where it shouldn't be particularly difficult to have separate BIQ- and SAV-based C3X config data, especially knowing that desire from the start.

    CLICK.

    Oh, if we're to have a SAV-based C3X config area, that means end users might desire to change settings. If we could allow the average user to turn e.g. sub bug, no raze, etc. on and off per-game, it might drive adoption of C3X independent of modders.

    1. Create new game
    2. Run C3X config injector
    3. Resume game C3X and get only the desired fixes/features
    (Until demand and time is enough for in-game-UI C3X config.) This is slightly less convoluted that creating a custom BIQ to configure C3X and allows more user choice than running C3X with no levers to flip.

    But the biggie to me is that non-modders might proactively adopt using C3X. Make chickens and eggs at the same time! More users will have C3X, and modders then don't have to worry as much if C3X is a detriment to having others play their mod.


    All that said, I know so far I'm just tossing out ideas. It would make sense for Quintillus and/or I to experiment with nondestructively injecting and pulling arbitrary data from BIQs and SAVs before Flintlock puts effort into reading it for C3X use. One of us will get around to it sooner or later.
     
  16. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Good news: For the BIQ I was able to add arbitrary data to the end of a (decompressed) BIQ file, and it still works! Only a couple of tests, but C3C seems to slurp in what it expects to find and ignores trailing data.

    I'm 99.9% certain this won't work for the SAV, or at least it won't be propagated to any future turns or manual saves which makes it useless for our purposes.

    But actually I think hiding and retrieving data from the SAV will be fairly easy IF the game persists "dead"/old/overridden class instance data, and I think it will.

    My idea: If I can fine ONE BYE per tile that isn't used, a "60x60" tiny map would have 60x30 tiles or up to 1800 arbitrary bytes retrievable by a search and simple offset loop. I think this will persist in all future saves.

    Edit: On the other hand, if we just tack on data to the end of the BIQ, C3X has to go back separately and read the tail end of the file I think. Maybe it's still worthwhile to find places to "hide" data in the data C3C already slurps in. Maybe...the more I think about this, the more potential pitfalls I'm thinking of.

    Edit 2: I have no code yet; I'm just manually decompressing, concat'ing, and testing starting a game with the modified BIQ.
     
    Last edited: Jun 30, 2021
  17. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    It's good to hear that appending data to a BIQ file appears to work. I think that would be the best way to store scenario config data because it gives unlimited space, is backwards compatible, and doesn't make distributing scenarios any more difficult (unlike the two file approach).

    For save games, it will be necessary to modify the saving and loading processes in any case and I don't think it matters as much if we use a one-file or a two-file approach because save games aren't traded around as often. I've already touched the game saving code but only at a high level. There is a function in the game that does the saving, taking as arguments a file path and a couple of other parameters. So it would be easy to modify it to write out a companion config file or whatever. I'm guessing, though I haven't actually looked into it, that game loading could be similarly modified.

    Unfortunately I don't know much about the internals of the save format. Don't all saves include a BIQ internally? If so, the neatest approach would be to pack the config details into that BIQ, though that might not work if it's not at the end of the file. Also when you say you're "manually decompressing" BIQs is that with an off the shelf decompressor like 7-zip or a scenario editor?
     
  18. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Oh I think it matters more than that. Play-by-email, succession games, and sharing saves when asking for help is a thing, especially on this forum. But also if you have a crapload of save files and go to clean them up or organize them, keeping the C3X and SAV files together would be a huge pain.

    Well...maybe and probably not. Non-custom "epic" games don't have the BIQ in them, and C3C then loads the default Conquests.biq file. Games based on BIQs *do* include the BIQ, but I am going to presume can confirm that the trailing data we add on after the BIQ is not thrown into the SAV. (Confirmed because the uncompressed autosave of my last test is smaller than my megabyte-of-trailing-data test BIQ is.)

    Both the BIQ and SAV data I believe are serialized class instance data dumps of the in-memory objects. They are not a "format" or generic serialization that I can discern. I'm not sure everyone agrees with me that these files are near-raw representation of in-memory objects, but only you and Antal1987 really have dug into the in-memory data.

    If you look at the decompressed BIQ and SAV files with a hex editor or string parser you'll find what Civ3 format descriptions call "sections", a char[4] ASCII header like CIV3, GAME, TILE, WRLD, CITY, UNIT, LEAD. Many of these repeat at odd intervals or even seemingly random. (Although I finally figured out CITY has like 86 or so repetitions of the CITY header for each city in game, but even knowing that each CITY section varies by length based on ... something, I forget, maybe citizen count and/or other stuff.)

    I tend to think of these "sections" as actual representation of the class instance memory object largely because each map tile in C3C has 4 TILEs of lengths (from memory, might be wrong) of 128, 12, 32, and 4, and it seems clear that these represent a cascade of object inheritance as the terrain data for PTW saves is in a different TILE portion than terrain data for C3C saves.

    In any case, there is no complete public full understanding of the whole data structure, so both Quintillus and I approach it with searching for a (usually the first) occurrence of a particular header and then decoding data from that point. But there are pitfalls here as sometimes a BIQ includes a map which makes finding the sav map vs the scenario map a bit more involved, and C3C doesn't zero-out allocated memory, so stray instances of text strings often pop up in uninitialized data, particularly in LEAD sections, so finding the first legit CITY and UNIT in particular takes some care.

    Heh, that's the short version. Sorry.

    The main point being, I *am* quite sure that the BIQ embedded in the SAV is deserialized into memory then serialized into the SAV, so no tacked-on data would survive that process as it's never slurped in by the deserializer in the first place.

    Which actually gives me more hope that "hiding" data in unused/obsolete places in the existing data would survive the deserialization/serialization process without your having to jump into the file read/write routines.

    C3C uses the PKWare DCL (Data Compression Library) which seems to have been a proprietary-only streamable de/compression format (analogous to gzip), and pretty much NO standard utilities can handle it. I know one's mind jumps immediately to PKWare being equivalent to ZIP, but DCL is just not a format used in zip archives or utilities.

    And it's not openly licensed or commonly used, so nobody is really freely distributing binary de/compressor utilities.

    But it has been reverse-engineered, and the reference C code is known as "blast" (the PKWare internal name is "explode"): https://github.com/madler/zlib/blob/master/contrib/blast/blast.c . So you can find various implementations here and there, or grab the code yourself.

    For what it's worth, C3C does not use the Huffman-coded literals and always uses the same sliding dictionary size. I coded my own implementation in Go of a decoder-only with no Huffman logic, and it works fine on Civ3 files. https://github.com/myjimnelson/c3sat/tree/master/civ3decompress . All implementations are based on a reverse-engineering posted at https://groups.google.com/g/comp.compression/c/M5P064or93o/m/W1ca1-ad6kgJ (although note that the blast.c implementor says the example encoding is incorrect...blast.c is well-documented, and I think blast.c is right.)

    But, in your case, the compressor and decompressor are obviously compiled into the executable, so it would probably make as much sense for you to leverage than than any third-party utility or library.

    What I actually did just now was use a PowerShell script that calls a C# implementation of Blast to do the decompression. This is the implementation we're using so far in the new-game-from-scratch code. I had sort-of intended to port my Go decompressor code to C#, but there's not a lot of compulsion to do so.
     
  19. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    A couple of add-on thoughts to my last post:

    If you go digging into BIQ/SAV format descriptions, be aware the research dates back to the original Civ3, and successive versions and even some patch levels have definitely changed the file format as far as us being able to reverse-engineer understand the files. The last C3C patch was I think 2004 or 2005 (aside from the Steam patch which just replaced the network game browser), so format descriptors from before that may be misleading. The data is right, but later versions may place that data in a different location. (And again my pet theory is that this means a new child class was created to add new or override fields.)

    The other thing is to clarify that by "hiding" data in the existing data I'm looking at string fields after the null and that data I believe is obsolete parent object data overridden by a child class instance. And that raw hidden data should persist being read from the save and then written back into a save.

    But now that I think of it, if that's true then I'm not sure how you'd address this memory. I was thinking in terms of C where you'd just grab an offset from the struct pointer, but I'm not sure how straightforward that would be in C++ where the instance is referred to by object reference. Is it a valid thing to address instance data of a parent class from a child instance?

    Hmm, well, if/when I poke around more at the hidden data idea I'll let you know the results, but I'll probably have to lean on you to figure out how to reach the "hidden" data from the object instance.

    Edit: Thing 3 of 2: Note that autosaves are NOT compressed for some reason, but regular saves and BIQs always are as produced by C3C and its editor. That might be useful info if you're hunting the de/compression routines. Loading decompressed BIQs or SAVs works just fine, so the code auto detects.

    Not sure how C3C detects compression, but all SAV files being with a char[4] of "CIV3", and all scenario files start with "BIC" (with either a space, X, or Q as the 4th char I think). DCL compressed files with the sliding dictionary size Civ3 always uses begins with 0x00, 0x04, 0x05, and 0x06 . (For DCL in general the first byte may be 0 or 1, and the second byte may be 4, 5, or 6.) I don't recall how/why I know that bytes 3 and 4 are 5 and 6, but I have that hard-coded check in my decompressor.
     
    Last edited: Jul 1, 2021
  20. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Oh. I finally actually downloaded and looked at your patch/code. It's in C, so I don't have to worry about object references to retrieve any successfully hidden data.
     

Share This Page