1. Firaxis celebrates the "Asian American and Pacific Islander Heritage Month", and offers a give-away of a Civ6 anthology copy (5 in total)! For all the details, please check the thread here. .
    Dismiss Notice
  2. Old World has finally been released on GOG and Steam, besides also being available in the Epic store . Come to our Old World forum and discuss with us!
    Dismiss Notice

Analysis: EXE patching

Discussion in 'Civ3 Future Development' started by WildWeazel, Jan 14, 2021.

Tags:
  1. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    I have a pedantic analytic brain. For patching, my brain jumps to Antal1987's framework where it is my understanding that his executables replaced the Conquests executable. I think Flintlock is using memory patching from a wrapper executable which is distinct from Antal1987's framework.

    Is what I'm saying correct, or am I misunderstanding something? Or does the distinction even have meaning?
     
  2. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    I think we might need to use, " @Flintlock " rather than simply, "Flintlock" a few times to grab his rapper executable's attention :dance:

    ;)
     
  3. Civinator

    Civinator Blue Lion Supporter

    Joined:
    May 5, 2005
    Messages:
    7,626
    Gender:
    Male
    Last edited: Apr 15, 2021
    Puppeteer likes this.
  4. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    Civinator & @Flintlock - Excellent news!

    Nonetheless, I "gamed it out" (without being able to use the game itself) I have indeed worked out, by analogy, of eliciting genuinely sophisticated in the same way that bees organize colonies and are capable of such seemingly intelligent behavior (including , for example's sake ;) how to make AI artillery actually work) with the added benefit that it can be assembled by individual components, rather than needing to do it all at once, keeping our fingers crossed to avoid :spear:
     
  5. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    It's not like I was waiting for an opportunity to type out a big technical post about by mod, but regardless I'm glad it's here.

    The mod works based mostly on function interception/patching/replacement or whatever you want to call it, I haven't been consistent myself. I think people are intimidated by EXE patching because they imagine it involves writing machine code by hand, but I do almost none of that. Instead I write C code and inject it into the executable. The injection process, which is fully automated at this point, works by replacing functions with custom versions in a manner that preserves the original function so that it can be called by its replacement. I wrote about how it's implemented in respose to Tsubasanut here: https://forums.civfanatics.com/thre...es-in-exe-modding.666881/page-3#post-16049068. This technique is very useful, clearly it can intercept function calls and modify them, but in addition it can:
    - intercept function returns by replacing a function, calling the original first thing, then running some code afterward
    - track changes in state by replacing every function that potentially modifies some state and check its value after versus before calling the original (for example this is how the disorder warning code detects that the player has signaled an intention to end the turn)
    - pass info through a side channel (f.e. if you want function B to get some info from its caller A, intercept A and set a global variable then read it in B)
    - alter a single function call by intercepting the callee then running different code depending on the return address
    All of this depends on knowledge gained by reverse engineering the executable, naturally you can't replace a function without first figuring out what it does and you can't read or write the game's memory without knowing what's kept where and how.
    How difficult any change is to make depends on how the relevant original code was structured. For example, changing the way railroad movement works turned out to be easy because there's a single function that determines the cost of moving a given unit between two tiles and every other part of the code, like pathfinding, AI logic, etc, all relies on it, so I just adjusted its return value. Making a stack bombard button appear was easy because there's a single function that sets up the unit buttons and the process of setting up a button is simple, so I added some code that runs after the normal button setup to create one more. But making that button work was tricky because the handling of the state of the UI w/r/t whether the player is selecting a bombard target or not is scattered in several different functions, I had to intercept (IIRC) 4 in total just for that. Regarding limits: Removing the unit limit was easy, it was a matter of patching a single byte to bypass a single check in the spawn unit function. Removing the city limit is maybe possible because the problem is a single table, accessed in hopefully only a few places. Removing the 31 civ limit is impossible b/c the limit appears in thousands of places scattered throughout the code.

    To contribute to the mod, I think basic knowledge of C/C++ is necessary and probably sufficient. To understand the decompiled code you'd need to know C and it's decompiled from C++ so knowing that definitely helps. The mod itself, injected code and injector, is all C. It would also help to know the low level details of how these languages are implemented, like what the call instruction does exactly, how the stack is laid out, etc. But you could probably get by without that or learn as you go, I knew much less about that sort of thing when I started several months ago.

    If you're curious to have a look at its innards, it's easy because the mod is completely open source. If you've downloaded it, all the injected code is right there in injected_code.c, you can have a look at it, you could even modify it then re-run the installer and any changes you've made will take effect. The injector is there too in ep.c. I've made this pitch several times now but as far as I know only Tsubasanut has ever taken me up on it. (Sadly he said it was over his head.)
    Since Release 3 the mod can either run as a memory patcher or produce a replacement EXE (but it's up to the user to supply the original EXE for modification). The distinction only matters at a low level for the implementation, most of the mod works above a little abstraction layer that covers up the difference. This isn't even hard to do, it's only about 200 lines, since there are only four functions you need to use to do memory patching, ReadProcessMemory and WriteProcessMemory for obvious reasons, VirtualAllocEx to allocate memory, and VirtualProtectEx to change memory protection in some circumstances. Now imagine doing the same on an EXE file: Reading and writing memory is a matter of mapping virtual addresses to locations in the file, which is not difficult to do since an EXE is fundamentally just an image of a process. Allocating memory is a matter of appending a section to the EXE, this is only difficult because the EXE format is thorny and not thoroughly documented, but I got it working after a couple of days of trial & error (and the final piece of the puzzle I found through Google on some random blog by a French guy last updated 10 years ago, so these things go). Lastly changing memory protection isn't a problem, translating page protection flags to PE section flags is easy, and you don't have to worry about page protections preventing you from writing b/c they don't apply, you're just modifying some file.

    Just to clarify exactly what I inherited from Antal, I benefitted a lot from his reverse engineering work but I'm not using his patch framework. I am using and building on his work reverse engineering the executable, i.e. figuring out how the data is laid out and what the functions do. I downloaded Civ3Conquests.h from his GitHub, imported it into Ghidra, and it's also a part of my mod, after a reorganization and with some additions here and there.
    Overhauling or even completely replacing the unit AI would be possible, I think, even though it would be quite a lot of work. For reference, the artillery AI decompiles to only about 220 lines of code and it depends on a helper function to evaluate targets that's about 120 lines (it seems the reason the artillery AI is so bad is that they didn't even bother). The functions for offensive and defensive units both come in at about 1000 lines, I haven't inspected what helper functions they use. So just in broad terms of complexity, it's doable. I've already done things of comparable size, like stack bombard ended up being 500-700 lines IIRC, and in total C3X R5 injects 1250 lines.
     
    WildWeazel, Civinator and Puppeteer like this.
  6. Ozymandias

    Ozymandias Archivist, redux Supporter

    Joined:
    Nov 5, 2001
    Messages:
    10,133
    Gender:
    Male
    Location:
    The lone and level sands
    :clap: :clap: :clap:

    As an aside, your mention of machine code reminded me of how :old: I am. My first job in IT entailed some 8 bit chip programming, long before "coding" replaced that term! ( :wow: )
     
  7. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    @Flintlock , a couple of questions on new-game generation and your framework:

    1 - Can there be a post-generate hook / trigger to run other code afterward? Perhaps to make changes impossible to do from the BIQ, or to run a map validator of some sort? Ideally able to invoke an external program or DLL.

    2 - Can the new game generator be looped? I'm imagining perhaps generating lots of new saves and doing some comparisons, perhaps with different settings, or perhaps with similar settings. This could be handy in nailing down various parts of the SAV data or rolling several saves to compare for a new game start, manually or via a statistical validator.

    3 - Is it reasonably possible to run the map generator as a DLL call from another program? This one is kind of "out there".

    I'm not asking for you do do all this or any of it, just wondering how likely or hard it is.
     
    WildWeazel likes this.
  8. Flintlock

    Flintlock King

    Joined:
    Sep 25, 2004
    Messages:
    861
    1. I couldn't point you to the exact location where you'd insert such a hook but I'm sure it's possible. I've already hooked various things like BIQ data initialization, game saving, and turn ending. There must be a function call or return somewhere you could intercept to run code after the new game is set up. Invoking an external DLL is no problem since the injected code can do anything a normal program can, the only catch is that there is no normal linking phase so you have to manually load the DLL and its functions. I've even already done this as a proof of concept of getting Lua running from inside the EXE. Here it is if you want to take a look: https://github.com/maxpetul/C3X/blob/integrate_lua/injected_code.c#L440. That's on a 6 month old branch but it illustrates the point.

    2. Maybe. The problem is that the game has a lot of global variables that would need to be reset to create a fresh new game. Even Firaxis didn't get it right, if you create dozens of new games without exiting the program completely, you'll notice civs start spawning right on top of one another as the map generator fails to place starting locations properly.

    3. I remember hearing somewhere that there is a program that lets you put in some map conditions (like cows around capital, alone on an island, etc.) and then it finds a seed that satisfies them. Assuming I didn't dream that, it's definitely possible and in fact has already been done. I expect the easiest way to do it would be to hack the editor rather than the game executable because of what I mentioned above.
     
    Puppeteer likes this.
  9. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,687
    Location:
    Silverdale, WA, USA
    Thanks!

    Very cool on the Lua integration. I had no idea you were even thinking of attempting that.

    I guess I shouldn't be surprised the new game functionality isn't properly isolated/encapsulated. I didn't know it could mess up like that, although I have seen the really-close civ placement before.

    I've seen the start-reroller you've described, or its thread before. I have just been kind of assuming it doesn't work anymore as it's so old, but I'll go find it and see if it still works. I'm nearly certain it's older than Conquests and perhaps older than PTW.

    Edit: Moonsinger's HoF Map Finder. Last thread posts were in 2017, and it reportedly still worked. https://forums.civfanatics.com/threads/hof-map-finder-generator-utility.75992/page-52
     
    Flintlock likes this.
  10. WildWeazel

    WildWeazel Carthago Creanda Est

    Joined:
    Jul 14, 2003
    Messages:
    7,206
    Location:
    %CIV3%\Conquests\Scenarios\
    I see what you're getting at, and like the idea. Sounds like a good angle to analyze some of the lesser known logic.
     

Share This Page