Shrinking resource bubbles by modifying the EXE

[f1rpo]

@Ramkhamhaeng: Well, I'm glad I asked, Cheat Engine with its value scan feature seems like a valuable hint. I had indeed, at one point, been inspecting registers in the VS debugger in search of my screen dimensions. The note about virus scanners is also interesting to me; I see that your code has already been in use for more than 5 years. Not sure if I understand the addLeaderheadGFC example correctly. E.g. the Dawn-of-Man screen calls that function on the CyGInterfaceScreen object. To identify the corresponding C++ function in the EXE, I guess we could anticipate that it'll call CvGlobals::getLeaderHeadInfo and maybe some functions CvLeaderHeadInfo; so breakpoints in those DLL functions could help. But those would be EXE-to-DLL calls, not calls via the Cy... wrappers in the DLL, so I don't think that's what you're suggesting. Regarding the diplo screen, I assume that it's entirely implemented in C++, so I don't see how the Python wrappers in the DLL (or in the EXE) would be involved at all.

I'm using this [EBP] to distinct if a DLL-function was called by the exe or not.

This could also be accomplished by introducing separate functions for internal and external calls through the .def file. But your method also allows checking for a specific call location in the EXE, which could be helpful. Your workaround for the lost network packet – shortening the leader and civ descriptions – reminds me of a crash that occurs in large single-player scenarios when the total length of the civ and leader descriptions exceeds a limit. This might involve a net message, but no actual network packet would be sent in single-player mode, so I guess this is essentially an unrelated bug – though the workaround, curiously, is the same.

• Width of info overlays (like f1rpo).

Not sure what else I might want to investigate in the foreseeable future. Faster combat animations (for the late game) would be neat, but probably not worth the trouble even with those new techniques. Come to think of it, if someone were to implement a new Custom Game screen, rather than simulating a mouse or keyboard input to launch the game (as I had proposed), one could probably just figure out which function in the EXE gets called upon launch and call that function directly. In any case, thanks a lot for explaining your methods!

 

[Ramkhamhaeng]

@Ramkhamhaeng Not sure if I understand the addLeaderheadGFC example correctly. E.g. the Dawn-of-Man screen calls that function on the CyGInterfaceScreen object. To identify the corresponding C++ function in the EXE, I guess we could anticipate that it'll call CvGlobals::getLeaderHeadInfo and maybe some functions CvLeaderHeadInfo; so breakpoints in those DLL functions could help. But those would be EXE-to-DLL calls, not calls via the Cy... wrappers in the DLL, so I don't think that's what you're suggesting. Regarding the diplo screen, I assume that it's entirely implemented in C++, so I don't see how the Python wrappers in the DLL (or in the EXE) would be involved at all.

Your approach, starting debugging with a breakpoint in

CvGlobals::getLeaderHeadInfo

at take a look who called it, is also possible. Probably I used this for the diplo menu searching, too.

My approach was the inverse variant and could also be used to find the desired target function. We're starting with a breakpoint before the EXE function is call. This is useful for calls of CyGInterfaceScreen (not in SDK).
The CyGInterfaceScreen functions are accessible by Python:
Python function -> unknown Boost Cy...Interface-Connector in EXE(*) -> EXE function we're seraching(**).

You can search for the string of the python interface function name, searching for the usage of this string in the EXE to find (*).
Like the Cy-Functions in the DLL this is just a wrapper and which processed/converts the Python arguments and then calling (**).

And the position of (**) it our target.

 

[f1rpo]

Ah, I see, thx again. I had not realized that the Python wrappers in the EXE are searchable. I suppose the "searching for the usage" of the string will take some thought and effort, but, since you've already gone through the trouble of locating several dozen important UI functions, I probably shouldn't worry about the details too much unless I ever need to identify another Python-exposed function.

 

[sqwishy]

Hi I was looking at implementing this in Realism Invictus. I was hoping of just adding this as a C extension in Python. Just a simple Python importable DLL compiled from C that is a simple wrapper around modifying the executable memory.

I am planning on doing something using `ReadProcessMemory(GetCurrentProcess(), address, buffer, size, NULL)` because it seems pretty approachable and won't segfault if you give it an invalid address presumably? I figure, if the self-modifying stuff could be in this C extension and that would remove the need for modifying CvGameCoreDLL. Maybe mod authors can use this functionality more easily by dropping the C extension in and calling it from a Python API rather than recompiling their CvGameCoreDLL.

I attached a proof of concept. Address modified is from the steam version I think? And requires toggling the pips. Should be able to run Makefile with nmake passing `-D PY24=Path\To\CvGameCoreDLL\Python24` and if you make the install target then `-D INSTALL=Path\To\Assets\Python`. I installed VS2022 and ran nmake from the x64 to x86 cross compilation environment. I included the compiled DLL if you want to try it without building it. I don't really know anything about developing on Windows so I assume/hope that it will _just work_? Like when I installed VS2022 it asked about something called Windows Kits and I picked 11 because it's a big number and I'm running Windows 11 and hopefully that means it works for other versions of Windows but I don't really know.

I haven't quite read through all of what Ramkhamhaeng did with minhook to see if that approach is better, but yeah I thought I'd mention the C extension thing because it seems like a nice way to bootstrap some of the runtime modifications. If you have any thoughts or reasons why this is a dumb idea let me know. (Like I didn't have to use `VirtualProtect` for this and I'm not sure why, might be a thing it works on my machine but not others, I don't know.)

Anyway, also came by to ask two things.

1. Importing the DLL doesn't work with the import statement (I think because of some shenanigans to make importing Python modules inside of Mods work implicitly). Using `imp.load_dynamic()` from the `imp` package does but I need to give it the full path relative to the current working directory/exe? That includes the mod name. I figured, I can import a .py file and it has a `__file__` variable that should have the path I want, but it doesn't. Again the funny import they do just lies about the module path. Like what I want to see is `Mods\Realism Invictus\Assets\Python\MyModule.py` and what I get is `Assets\Python\MyModule.py`. Is there a good way to get the path to the .py file including the mod name? Otherwise the mod name written in the .py file that loads the DLL and it's not the end of the world; mod authors can update it when they distribute or use it.

2. When I modify a file inside of Assets/Python the game crashes, I think using the built in reload() function also causes this and I think something is trying to reload modules when files change on disk. I haven't figure out what's doing this. Is there a way to turn this off? It always crashes the game and sometimes that's a little annoying.

Bonus Meme: If the C extension exposes `Read|WriteProcessMemory` over bytes generally, I think it's pretty easy to combine with the struct module to make an interface that takes python values and writes them appropriately or reads out python values. So a Python function that writes memory takes an address, struct format string like "fih", and values like (42, -1, 3) and uses the struct package to get a Python string of bytes like '\x00\x00(B\xff\xff\xff\xff\x03\x00' and passes that to WriteProcessMemory. Similarly, reading would take an address and struct format string, call struct.calcsize, allocate a buffer, read to buffer from ReadProcess memory, then use that buffer with struct.unpack to return a tuple that matches the format given in the argument. One thing that might not be great is that padding is automatically added between struct fields sometimes. So it's not quite always what you see is what you get. Also the extra allocation and indirection is worse for performance but if it's not a high performance scenario it might not matter. I thought I'd mention it in case this would be useful for making it easier to build out these sort of techniques in Python. Or I can totally see it being more pleasant in C or C++ compared to a twenty year old version of Python. Just thought I'd ask to see if this interests anyone.

 

[f1rpo]

Oh, right, those runtime code changes don't need to be applied by the GameCore DLL necessarily. I guess it does help to apply them from the BtS process, but of course Python scripts do run within that process. This hadn't occurred to me and I also had no idea whether and how Python can deal with pointers (virtual memory addresses). So, only through a C extension, I see.

If the C extension exposes `Read|WriteProcessMemory` over bytes generally, I think it's pretty easy to combine with the struct module to make an interface that takes python values and writes them appropriately or reads out python values. [...] I thought I'd mention it in case this would be useful for making it easier to build out these sort of techniques in Python. Or I can totally see it being more pleasant in C or C++ compared to a twenty year old version of Python. Just thought I'd ask to see if this interests anyone.

20y-old Python, 20y-old C++ or modern-day C? For me, MSVC03 is the devil I know, but that can't just be dropped into other mods without re-compilation. I'd prefer C over Python (any version) still, but this, again, requires a compiler. On this note, I was going to at least give your proof of concept a try on my Windows system, but the Steam address isn't going to work for my disc version and I don't have a present-day MSVC compiler installed. It really should work with my old compiler too – it's C ... I should try it.

[...] And requires toggling the pips.

I guess this won't be a problem if the code is either modified only once shortly after loading the mod nor if it's tied to a BUG option. Having re-examined my own code, it seems that setting the GlobeLayer dirty bit will take care of redrawing the pips.

[...] I didn't have to use `VirtualProtect` for this and I'm not sure why, might be a thing it works on my machine but not others, I don't know.

In the GameCore DLL, I was getting a segmentation fault until I added the VirtualProtect calls. In Microsoft's documentation, it doesn't sound like WriteProcessMemory takes care of that – though I guess it might. Probably a good idea, in any case, to use WriteProcessMemory instead of dereferencing the raw address pointer. I suppose this will avoid a crash when the protections haven't been properly lifted,

[...] Is there a good way to get the path to the .py file including the mod name? Otherwise the mod name written in the .py file that loads the DLL and it's not the end of the world; mod authors can update it when they distribute or use it.

The BUG mod already stores its own name in a single place, namely CvModName.py with a higher-level accessor function in BugPath.py. And the initModFolder function in that module tries to obtain the mod folder from CyReplayInfo.getModName. Not the nicest of solutions – if they'll even work for your purposes.

[...] I think something is trying to reload modules when files change on disk. I haven't figure out what's doing this. Is there a way to turn this off? It always crashes the game and sometimes that's a little annoying.

No idea how this works. With just BtS, the reloading doesn't crash (for me); with BUG, it shouldn't crash to desktop either, but there'll be Python exceptions that will pretty much necessitate a restart. (latest thread on that subject)

 

[sqwishy]

On this note, I was going to at least give your proof of concept a try on my Windows system, but the Steam address isn't going to work for my disc version and I don't have a present-day MSVC compiler installed. It really should work with my old compiler too – it's C ... I should try it.

It seems all ABI compatible. I think the python.dll shipped with Civ is the same dll you get from downloading Python 2.4.1 from python.org compiled forever ago. But the C extension compiled with a recent version of msvc still works with it. I only mentioned it in case there is something I'm not aware of or if it doesn't compile because I accidentally used a feature not supported in older compilers. I'm not a big Microsoft person and their compiler is a bit unfamiliar to me.

I was getting a segmentation fault until I added the VirtualProtect calls. In Microsoft's documentation, it doesn't sound like WriteProcessMemory takes care of that – though I guess it might.

That was my thinking. The documentation doesn't sound like I don't need VirtualProtect, in fact it says I still need PROCESS_VM_WRITE permission. I even checked running the game under steam and it works; whereas typically I can't use a debugger unless I use un-DRM the exe with Steamless as Nightinggale pointed it out in my thread the other week. So I dunno.

I think one issue with the mod name information from BUG or CvReplayInfo is that I _think_ loading the DLL needs an absolute path or relative to the program's current working directory. And IIRC mods can sometimes live in My Documents somewhere, but maybe you can't necessarily tell that from CvReplayInfo.getModName()? I haven't tested that so I could be mistaken.

It seems the Python modules are loaded with the help of some goofy function exported to Python through boost under the name `loadImportModule` but I can't follow it well in a debugger to get anything useful out of it. It seems to look for a file, read it, and return the file contents. Later, another boost exported function named `getModulePathName` is called to return the file path under "assets", but I want the full path including where that assets folder is.

I attached a newer version that checks two addresses for the plot picker size; I think it will work for most non-steam executables? Also did the struct thing I mentioned so users can do something like `PlotPinScale.getValues(0x553eba, '=Bi')` and it returns `(185, 353)` a tuple from an uint8 and int32 corresponding to the struct module docs [1]. There's also a set of functions for reading or writing numbers. Again, not sure having this API in Python is that helpful; but, at least for me, for this one plot picker scaling thing, this seems less fussy that recompiling CvGameCoreDLL.

[1] https://docs.python.org/release/2.4.4/lib/module-struct.html

 

[Ramkhamhaeng]

I think one issue with the mod name information from BUG or CvReplayInfo is that I _think_ loading the DLL needs an absolute path or relative to the program's current working directory. And IIRC mods can sometimes live in My Documents somewhere, but maybe you can't necessarily tell that from CvReplayInfo.getModName()? I haven't tested that so I could be mistaken.

It seems the Python modules are loaded with the help of some goofy function exported to Python through boost under the name `loadImportModule` but I can't follow it well in a debugger to get anything useful out of it. It seems to look for a file, read it, and return the file contents. Later, another boost exported function named `getModulePathName` is called to return the file path under "assets", but I want the full path including where that assets folder is.

I don't think there is a Python-only way to find out the absolute path of a Python file loaded by Civ4. The modified module loader of Civ4 searches on multiple paths for modules, but give the imported module always the same (default) path information.

My way to solve the problem was requesting the absolute path of the loaded DLL CvGameCoreDLL.dll (Not in Python but C++ part). This file is placed inside of the loaded mod.

 

[sqwishy]

My way to solve the problem was requesting the absolute path of the loaded DLL CvGameCoreDLL.dll (Not in Python but C++ part). This file is placed inside of the loaded mod.

Right, I had seen something called BUFFY/Buffy.py that calls `gc.getGame().getDLLPath()` which is presumably implemented by some mods in CvGameCoreDLL (but not in Realism Invictus)? But if it's not commonly available then I'm not sure I can rely on it. It's a bummer that ctypes isn't in Python 2.4.

 

[Matsuda123]

It seems all ABI compatible. I think the python.dll shipped with Civ is the same dll you get from downloading Python 2.4.1 from python.org compiled forever ago. But the C extension compiled with a recent version of msvc still works with it. I only mentioned it in case there is something I'm not aware of or if it doesn't compile because I accidentally used a feature not supported in older compilers. I'm not a big Microsoft person and their compiler is a bit unfamiliar to me.



That was my thinking. The documentation doesn't sound like I don't need VirtualProtect, in fact it says I still need PROCESS_VM_WRITE permission. I even checked running the game under steam and it works; whereas typically I can't use a debugger unless I use un-DRM the exe with Steamless as Nightinggale pointed it out in my thread the other week. So I dunno.

I think one issue with the mod name information from BUG or CvReplayInfo is that I _think_ loading the DLL needs an absolute path or relative to the program's current working directory. And IIRC mods can sometimes live in My Documents somewhere, but maybe you can't necessarily tell that from CvReplayInfo.getModName()? I haven't tested that so I could be mistaken.

It seems the Python modules are loaded with the help of some goofy function exported to Python through boost under the name `loadImportModule` but I can't follow it well in a debugger to get anything useful out of it. It seems to look for a file, read it, and return the file contents. Later, another boost exported function named `getModulePathName` is called to return the file path under "assets", but I want the full path including where that assets folder is.

I attached a newer version that checks two addresses for the plot picker size; I think it will work for most non-steam executables? Also did the struct thing I mentioned so users can do something like `PlotPinScale.getValues(0x553eba, '=Bi')` and it returns `(185, 353)` a tuple from an uint8 and int32 corresponding to the struct module docs [1]. There's also a set of functions for reading or writing numbers. Again, not sure having this API in Python is that helpful; but, at least for me, for this one plot picker scaling thing, this seems less fussy that recompiling CvGameCoreDLL.

[1] https://docs.python.org/release/2.4.4/lib/module-struct.html

I'm sorry to necro post, but what terrain pack are you using? It looks so good, and I want to have those farms / grassland especially.