Analysis: EXE patching

kind of things can be done or not done using the framework

I have a pedantic analytic brain. For patching, my brain jumps to Antal1987's framework where it is my understanding that his executables replaced the Conquests executable. I think Flintlock is using memory patching from a wrapper executable which is distinct from Antal1987's framework.

Is what I'm saying correct, or am I misunderstanding something? Or does the distinction even have meaning?
 
I have a pedantic analytic brain. For patching, my brain jumps to Antal1987's framework where it is my understanding that his executables replaced the Conquests executable. I think Flintlock is using memory patching from a wrapper executable which is distinct from Antal1987's framework.

Is what I'm saying correct, or am I misunderstanding something? Or does the distinction even have meaning?
I think we might need to use, " @Flintlock " rather than simply, "Flintlock" a few times to grab his rapper executable's attention :dance:

;)
 
Last edited:
One of Flintlock´s sources for his brilliant work was Antal1987´s framework. Here Flintlock explains more about his great work: https://forums.civfanatics.com/thre...es-in-exe-modding.666881/page-7#post-16066922

...and Ozy, today Flintlock posted that he knows what function is responsible for making the AI's build choices. :)

Civinator & @Flintlock - Excellent news!

Nonetheless, I "gamed it out" (without being able to use the game itself) I have indeed worked out, by analogy, of eliciting genuinely sophisticated in the same way that bees organize colonies and are capable of such seemingly intelligent behavior (including , for example's sake ;) how to make AI artillery actually work) with the added benefit that it can be assembled by individual components, rather than needing to do it all at once, keeping our fingers crossed to avoid :spear:
 
It's not like I was waiting for an opportunity to type out a big technical post about by mod, but regardless I'm glad it's here.

The mod works based mostly on function interception/patching/replacement or whatever you want to call it, I haven't been consistent myself. I think people are intimidated by EXE patching because they imagine it involves writing machine code by hand, but I do almost none of that. Instead I write C code and inject it into the executable. The injection process, which is fully automated at this point, works by replacing functions with custom versions in a manner that preserves the original function so that it can be called by its replacement. I wrote about how it's implemented in respose to Tsubasanut here: https://forums.civfanatics.com/thre...es-in-exe-modding.666881/page-3#post-16049068. This technique is very useful, clearly it can intercept function calls and modify them, but in addition it can:
- intercept function returns by replacing a function, calling the original first thing, then running some code afterward
- track changes in state by replacing every function that potentially modifies some state and check its value after versus before calling the original (for example this is how the disorder warning code detects that the player has signaled an intention to end the turn)
- pass info through a side channel (f.e. if you want function B to get some info from its caller A, intercept A and set a global variable then read it in B)
- alter a single function call by intercepting the callee then running different code depending on the return address
All of this depends on knowledge gained by reverse engineering the executable, naturally you can't replace a function without first figuring out what it does and you can't read or write the game's memory without knowing what's kept where and how.
Do you think you could generalize at some high level what kind of things can be done or not done using the framework and with roughly how much effort? For example, numeric limits in the BIQ format, or AI behaviors, or new types of prerequisite conditions. Or will it be a strictly case by case basis? What if we came up with a prioritized wishlist? Also, assuming someone is familiar with C++, how hard would it be to positively contribute?
How difficult any change is to make depends on how the relevant original code was structured. For example, changing the way railroad movement works turned out to be easy because there's a single function that determines the cost of moving a given unit between two tiles and every other part of the code, like pathfinding, AI logic, etc, all relies on it, so I just adjusted its return value. Making a stack bombard button appear was easy because there's a single function that sets up the unit buttons and the process of setting up a button is simple, so I added some code that runs after the normal button setup to create one more. But making that button work was tricky because the handling of the state of the UI w/r/t whether the player is selecting a bombard target or not is scattered in several different functions, I had to intercept (IIRC) 4 in total just for that. Regarding limits: Removing the unit limit was easy, it was a matter of patching a single byte to bypass a single check in the spawn unit function. Removing the city limit is maybe possible because the problem is a single table, accessed in hopefully only a few places. Removing the 31 civ limit is impossible b/c the limit appears in thousands of places scattered throughout the code.

To contribute to the mod, I think basic knowledge of C/C++ is necessary and probably sufficient. To understand the decompiled code you'd need to know C and it's decompiled from C++ so knowing that definitely helps. The mod itself, injected code and injector, is all C. It would also help to know the low level details of how these languages are implemented, like what the call instruction does exactly, how the stack is laid out, etc. But you could probably get by without that or learn as you go, I knew much less about that sort of thing when I started several months ago.

If you're curious to have a look at its innards, it's easy because the mod is completely open source. If you've downloaded it, all the injected code is right there in injected_code.c, you can have a look at it, you could even modify it then re-run the installer and any changes you've made will take effect. The injector is there too in ep.c. I've made this pitch several times now but as far as I know only Tsubasanut has ever taken me up on it. (Sadly he said it was over his head.)
I have a pedantic analytic brain. For patching, my brain jumps to Antal1987's framework where it is my understanding that his executables replaced the Conquests executable. I think Flintlock is using memory patching from a wrapper executable which is distinct from Antal1987's framework.

Is what I'm saying correct, or am I misunderstanding something? Or does the distinction even have meaning?
Since Release 3 the mod can either run as a memory patcher or produce a replacement EXE (but it's up to the user to supply the original EXE for modification). The distinction only matters at a low level for the implementation, most of the mod works above a little abstraction layer that covers up the difference. This isn't even hard to do, it's only about 200 lines, since there are only four functions you need to use to do memory patching, ReadProcessMemory and WriteProcessMemory for obvious reasons, VirtualAllocEx to allocate memory, and VirtualProtectEx to change memory protection in some circumstances. Now imagine doing the same on an EXE file: Reading and writing memory is a matter of mapping virtual addresses to locations in the file, which is not difficult to do since an EXE is fundamentally just an image of a process. Allocating memory is a matter of appending a section to the EXE, this is only difficult because the EXE format is thorny and not thoroughly documented, but I got it working after a couple of days of trial & error (and the final piece of the puzzle I found through Google on some random blog by a French guy last updated 10 years ago, so these things go). Lastly changing memory protection isn't a problem, translating page protection flags to PE section flags is easy, and you don't have to worry about page protections preventing you from writing b/c they don't apply, you're just modifying some file.

Just to clarify exactly what I inherited from Antal, I benefitted a lot from his reverse engineering work but I'm not using his patch framework. I am using and building on his work reverse engineering the executable, i.e. figuring out how the data is laid out and what the functions do. I downloaded Civ3Conquests.h from his GitHub, imported it into Ghidra, and it's also a part of my mod, after a reorganization and with some additions here and there.
Nonetheless, I "gamed it out" (without being able to use the game itself) I have indeed worked out, by analogy, of eliciting genuinely sophisticated in the same way that bees organize colonies and are capable of such seemingly intelligent behavior (including , for example's sake ;) how to make AI artillery actually work) with the added benefit that it can be assembled by individual components, rather than needing to do it all at once, keeping our fingers crossed to avoid :spear:
Overhauling or even completely replacing the unit AI would be possible, I think, even though it would be quite a lot of work. For reference, the artillery AI decompiles to only about 220 lines of code and it depends on a helper function to evaluate targets that's about 120 lines (it seems the reason the artillery AI is so bad is that they didn't even bother). The functions for offensive and defensive units both come in at about 1000 lines, I haven't inspected what helper functions they use. So just in broad terms of complexity, it's doable. I've already done things of comparable size, like stack bombard ended up being 500-700 lines IIRC, and in total C3X R5 injects 1250 lines.
 
@Flintlock , a couple of questions on new-game generation and your framework:

1 - Can there be a post-generate hook / trigger to run other code afterward? Perhaps to make changes impossible to do from the BIQ, or to run a map validator of some sort? Ideally able to invoke an external program or DLL.

2 - Can the new game generator be looped? I'm imagining perhaps generating lots of new saves and doing some comparisons, perhaps with different settings, or perhaps with similar settings. This could be handy in nailing down various parts of the SAV data or rolling several saves to compare for a new game start, manually or via a statistical validator.

3 - Is it reasonably possible to run the map generator as a DLL call from another program? This one is kind of "out there".

I'm not asking for you do do all this or any of it, just wondering how likely or hard it is.
 
1. I couldn't point you to the exact location where you'd insert such a hook but I'm sure it's possible. I've already hooked various things like BIQ data initialization, game saving, and turn ending. There must be a function call or return somewhere you could intercept to run code after the new game is set up. Invoking an external DLL is no problem since the injected code can do anything a normal program can, the only catch is that there is no normal linking phase so you have to manually load the DLL and its functions. I've even already done this as a proof of concept of getting Lua running from inside the EXE. Here it is if you want to take a look: https://github.com/maxpetul/C3X/blob/integrate_lua/injected_code.c#L440. That's on a 6 month old branch but it illustrates the point.

2. Maybe. The problem is that the game has a lot of global variables that would need to be reset to create a fresh new game. Even Firaxis didn't get it right, if you create dozens of new games without exiting the program completely, you'll notice civs start spawning right on top of one another as the map generator fails to place starting locations properly.

3. I remember hearing somewhere that there is a program that lets you put in some map conditions (like cows around capital, alone on an island, etc.) and then it finds a seed that satisfies them. Assuming I didn't dream that, it's definitely possible and in fact has already been done. I expect the easiest way to do it would be to hack the editor rather than the game executable because of what I mentioned above.
 
Thanks!

Very cool on the Lua integration. I had no idea you were even thinking of attempting that.

I guess I shouldn't be surprised the new game functionality isn't properly isolated/encapsulated. I didn't know it could mess up like that, although I have seen the really-close civ placement before.

I've seen the start-reroller you've described, or its thread before. I have just been kind of assuming it doesn't work anymore as it's so old, but I'll go find it and see if it still works. I'm nearly certain it's older than Conquests and perhaps older than PTW.

Edit: Moonsinger's HoF Map Finder. Last thread posts were in 2017, and it reportedly still worked. https://forums.civfanatics.com/threads/hof-map-finder-generator-utility.75992/page-52
 
Top Bottom