WildWeazel
Carthago Creanda Est
Discuss it here!
Summary TBA
Summary TBA
Last edited:
Analysis: EXE patching
- Thread starter WildWeazel
- Start date
-
- Tags
- technical
All right, so riffing off of my most recent post to the "New game engine" thread, I'll note the single biggest problem with this approach IMO:
We don't have anyone with expertise in this area. Thus, it would both be very difficult to start making progress, and easy to be discouraged and lose motivation or "steam".
The second-biggest problem is one that Ozymandias quoted Antal on in my editor thread recently:
Practically speaking, that sort of problem would be a blocker on some of the most-wished-for changes. For some function values, e.g. shouldTheCityBeRazed, if you can identify the function location, you can short-circuit it to always return 0 (false). But when there's a data format problem, the problem quickly becomes intractable.
These two problems are the reason that I don't think we should pursue this path, unless we can get one or (preferable) more people with that sort of experience involved.
We don't have anyone with expertise in this area. Thus, it would both be very difficult to start making progress, and easy to be discouraged and lose motivation or "steam".
The second-biggest problem is one that Ozymandias quoted Antal on in my editor thread recently:
Practically speaking, that sort of problem would be a blocker on some of the most-wished-for changes. For some function values, e.g. shouldTheCityBeRazed, if you can identify the function location, you can short-circuit it to always return 0 (false). But when there's a data format problem, the problem quickly becomes intractable.
These two problems are the reason that I don't think we should pursue this path, unless we can get one or (preferable) more people with that sort of experience involved.
Puppeteer
Emperor
I concur with Qunitillus' points, and I'll add that as far as I can tell, nobody seems particularly motivated to program in C++ or another systems language (C, Rust) which would seem to be needed to accomplish this, even if we had confidence in extending Antal1987's work.
And if you're patching 794 direct references plus indirect references through the mechanism Antal1987 used, you're almost making a new game, anyway. It's possibly even more complex than a new game. Even he didn't attempt to touch the 31-civ limit.
And if you're patching 794 direct references plus indirect references through the mechanism Antal1987 used, you're almost making a new game, anyway. It's possibly even more complex than a new game. Even he didn't attempt to touch the 31-civ limit.
- Joined
- May 5, 2005
- Messages
- 8,875
I agree, that nobody of us has expertise in this area. The knowledge for hexediting C3C is in Russia and in the Baltic states (tsubasanut, knuckles - both with friends -, Antal1987 and skyer2. If this should be the path to go, some of these civers must be found and added to the team or anybody of us has to volunteer to learn the art of hexediting. Even to find the Russian hexeditors could be difficult. The last one of them (tsubasanut) appeared at CFC in May 2020 for the last time.
If I remember well, tsubasanut posted in the CCM2 thread, that he will ask for the material to hexedit many parts of the C3C exe and he managed to combine the No-Raze patch with the Antal1987 v. 4 patch and the no-unit-limit patch, all working in C3C games and he explained, why the NCL patch isn´t working.
If I remember well, tsubasanut posted in the CCM2 thread, that he will ask for the material to hexedit many parts of the C3C exe and he managed to combine the No-Raze patch with the Antal1987 v. 4 patch and the no-unit-limit patch, all working in C3C games and he explained, why the NCL patch isn´t working.
WildWeazel
Carthago Creanda Est
So the main take away here is that there's simply nobody able or willing to take it on so far. Unless some of patchers return or someone of similar skill shows interest, this one is DOA.
I found my old code exploring this option, and have uploaded it here.
I haven't really had a chance to look over it yet, though at a very cursory glance it's rougher than I'd recalled. But it could provide a starting point, and perhaps illustrate the approach somewhat.
I might take a look at it and see if I can polish it up to make it a bit more readable. I haven't done any C++ in the five years since I last updated it, but I've done a fair amount of C which might allow some stylistic improvements (it's hardly modern C++ anyway).
Perhaps most useful will be the Documentation.txt file at the top level, which gives an overview both of what it can do, and what my vision was at the time.
I haven't really had a chance to look over it yet, though at a very cursory glance it's rougher than I'd recalled. But it could provide a starting point, and perhaps illustrate the approach somewhat.
I might take a look at it and see if I can polish it up to make it a bit more readable. I haven't done any C++ in the five years since I last updated it, but I've done a fair amount of C which might allow some stylistic improvements (it's hardly modern C++ anyway).
Perhaps most useful will be the Documentation.txt file at the top level, which gives an overview both of what it can do, and what my vision was at the time.
- Joined
- May 5, 2005
- Messages
- 8,875
Quintillus, what you found out here is a lot (so unfortunately not sufficient) !
There are two areas where we may find data: This has some parallels what Tsubasanut posted in the CCM2 thread about the NCL component:
The NCL hack itself consisted of two changes in 2 very separate places in code:
1. In a procedure to calculate city resources.
2. In a procedure to place a new city.
And official 1.22 patch actually implemented the first change, but not the second. So i'm rolling back the 2nd (and i'm pretty confident that it is the ACTUAL check), but leaving 1st, as it is official (and i have no idea what this change does).
There are two areas where we may find data: This has some parallels what Tsubasanut posted in the CCM2 thread about the NCL component:
The NCL hack itself consisted of two changes in 2 very separate places in code:
1. In a procedure to calculate city resources.
2. In a procedure to place a new city.
And official 1.22 patch actually implemented the first change, but not the second. So i'm rolling back the 2nd (and i'm pretty confident that it is the ACTUAL check), but leaving 1st, as it is official (and i have no idea what this change does).
Are you referring to this section of the Documentation.txt file?
I would be careful about reading much into that. There are many possible reasons that the string BLDG_Palace could appear more than once - once in the BIQ file, once in a Civilopedia structure in memory, or perhaps the user loaded a game from the in game menu, and the old BIQ's memory hasn't been deallocated, or was deallocated but has not been overwritten yet. Tsubasanut was also discussing updating the code, whereas I was solely looking at data.
I agree that there's definitely a lot more to discover along that approach, and it may be possible to combine EXE patching of code with in-memory patching of data, but the overlap of two places where Tsubasanut had updates and two places where I saw BLDG_Palace in memory appears to be coincidental to me.
I would be careful about reading much into that. There are many possible reasons that the string BLDG_Palace could appear more than once - once in the BIQ file, once in a Civilopedia structure in memory, or perhaps the user loaded a game from the in game menu, and the old BIQ's memory hasn't been deallocated, or was deallocated but has not been overwritten yet. Tsubasanut was also discussing updating the code, whereas I was solely looking at data.
I agree that there's definitely a lot more to discover along that approach, and it may be possible to combine EXE patching of code with in-memory patching of data, but the overlap of two places where Tsubasanut had updates and two places where I saw BLDG_Palace in memory appears to be coincidental to me.
Puppeteer
Emperor
I've done no memory exploration, but I'm convinced this happens a LOT as there is a lot of junk text in the SAV file that is clearly uninitialized / dirty data. It makes seeking to e.g. the 83rd instance of "CITY" quite challenging because CITY appears a lot in random places in an unpredictable fashion.
WildWeazel
Carthago Creanda Est
New discussion here: https://forums.civfanatics.com/threads/direct-memory-access-in-game.666798/
- Joined
- May 5, 2005
- Messages
- 8,875
Please have a look at this post: https://forums.civfanatics.com/threads/sub-bug-fix-and-other-adventures-in-exe-modding.666881/
The civer said, he fixed the submarine bug by reverse engineering and added a zip-file.
The civer said, he fixed the submarine bug by reverse engineering and added a zip-file.
Overall, I think it's a fine idea, with the following caveat: Enough people focusing on the editable, existing .exe might wind up with that approach becoming our "default" solution - and one which would both dramatically limit the scope of what we'd like to accomplish, as well as (and along the same lines) "vectoring" our Coders' attention into that particular area
My general thought (having only skimmed that thread so far; I'm a bit behind on posts currently) is yes.
Although I think Ozymandias has a good point too. With the emergence of Flintlock, I am increasingly thinking the right option is the pursuing more than one option. Simply put, we have community members with non-overlapping skills. Flintlock is accomplishing things via the EXE that would take longer using alternative approaches. In the long term, however, a "new foundation" approach may be necessary for some enhancements; such as the No City Limit item that Flintlock mentions in post #16. I can contribute on the "new foundation" and "editor/related utilities", but not on EXE patching. Multiple approaches would maximize the number of contributors, and thus likely the eventual possibilities opened up.
Although I think Ozymandias has a good point too. With the emergence of Flintlock, I am increasingly thinking the right option is the pursuing more than one option. Simply put, we have community members with non-overlapping skills. Flintlock is accomplishing things via the EXE that would take longer using alternative approaches. In the long term, however, a "new foundation" approach may be necessary for some enhancements; such as the No City Limit item that Flintlock mentions in post #16. I can contribute on the "new foundation" and "editor/related utilities", but not on EXE patching. Multiple approaches would maximize the number of contributors, and thus likely the eventual possibilities opened up.
WildWeazel
Carthago Creanda Est
I tend to agree. I've also thought some about multiple approaches, especially if one can feed into another. Eg. BIQ tools could provide code for loading them into a new game.
It's true that involving more people may lead to pulling in different directions, but since so little has happened anyway I think we can only benefit by having more irons in the fire.
Is there anyone else we should approach? On another front, there's that Civ 2 clone project...
It's true that involving more people may lead to pulling in different directions, but since so little has happened anyway I think we can only benefit by having more irons in the fire.
Is there anyone else we should approach? On another front, there's that Civ 2 clone project...
... Which, I suspect, would make the interest of anyone looking for existing Mod compatibility fall away, sharply.
- Joined
- May 5, 2005
- Messages
- 8,875
Flintlock uses a 'C script' for fixing (and adding) the C3C exe.
WildWeazel
Carthago Creanda Est
My thought there was that since they're using C# and I nudged them towards Godot, there may be some opportunity to share code. But that could just as easily constrain us both. If we both choose that path for our own reasons it's a conversation we could have later.
What, you'd keep them Waiting for Godot??
Also, presupposing that we knew that they know what they're doing - Also, hey might not have much interest in any sort of close collaboration.Also, along all of these lines, I think that we should be keeping closer track(s) of:
- Where each of of us are, "deep diving;" and
- To whatever it is we think we might wind up finding, and:
- Is it a standalone effort, or, if not -
- How can it be integrated with other approaches, whether as (e.g.) a partial, standalone component? - And/or
- Can any two (or more) particular approaches be (theoretically) integrated, without tripping over each other, because we know which code we are each touching, and leaving untouched?
- Is it a standalone effort, or, if not -
At last I, count think these are the avenues we're currently barreling down:
- Header .exe patching (Furthering antal1987's approach.)
- Runtime decompilation for "injecting" code.
- Runtime decompilation for gaining "access" to whatever degree is feasible.
- Evaluating other game platforms. ("Co-opting" and utilizing another, open-sourced as our new foundation.)
- Evaluating other game engines. (Which - for whatever unplumbed reason(s) - is starting to":feel" the most problematic to me.)
I believe that the lion's share of my efforts should be re-directed towards an OO game template (without calling it such, of course
)My own belief is that focusing on .exe patching, only, will immediately leave us with a feeling of ... Dissatisfaction.
My own instinct is that - with some degree ,or another, of realizing that we'll have to do some serious, new coding on our own - That we might as well have an OO template ready to go. I've already mentioned tat my first impression of the code<->game templates i to give the developers kudos: I see no reason why an agile enough OO game mod can't be "patched atop" another
AND that I have genuinely, truly, and thoroughly mapped out a way to accomplish this, "elegantly."
WildWeazel
Carthago Creanda Est
Paging @Flintlock 
Before your patching adventure we had all but written off this approach because none of us were particularly knowledgeable or interested in it. I've been impressed by what you've been able to do in a short time. It seems this is a viable option with someone such as yourself leading the way, but I think we still don't have a good understanding of its scope. Do you think you could generalize at some high level what kind of things can be done or not done using the framework and with roughly how much effort? For example, numeric limits in the BIQ format, or AI behaviors, or new types of prerequisite conditions. Or will it be a strictly case by case basis? What if we came up with a prioritized wishlist? Also, assuming someone is familiar with C++, how hard would it be to positively contribute?
Before your patching adventure we had all but written off this approach because none of us were particularly knowledgeable or interested in it. I've been impressed by what you've been able to do in a short time. It seems this is a viable option with someone such as yourself leading the way, but I think we still don't have a good understanding of its scope. Do you think you could generalize at some high level what kind of things can be done or not done using the framework and with roughly how much effort? For example, numeric limits in the BIQ format, or AI behaviors, or new types of prerequisite conditions. Or will it be a strictly case by case basis? What if we came up with a prioritized wishlist? Also, assuming someone is familiar with C++, how hard would it be to positively contribute?
