Shrinking resource bubbles by modifying the EXE

Discussion in 'Civ4 - SDK/Python' started by f1rpo, May 8, 2022.

  1. f1rpo

    f1rpo plastics

    Joined:
    May 22, 2014
    Messages:
    1,197
    Location:
    Germany
    I've found the four places in the code section of the EXE where the size of the plot indicators ("bubbles") is hardcoded. Those values aren't unreasonable, the real problem is that the EXE increases them based on the screen height. I've located some code in that vein in the debugger, but I'm not at all sure that that's really it. (Also not sure if the width doesn't also have some impact.) But we can just decrease the base values to compensate for the adjustment.

    For a static approach, one can use a hex editor [edit #3: see instructions in this post] and simply replace all occurrences of the floating-point numbers 42 and 68 (FP32 encoding) in Civ4BeyondSword.exe with a smaller number. The 68 is for the bubble shown when the selected unit is off-screen. Well, this is not totally simple because one needs to get the encoding right. [Edit #2: Moreover, there are actually four occurrences of 42 in the EXE and seven occurrences of 68. Only the first three of the former and the first one of the latter ought to be replaced as far as I know.] I can imagine a little tool that let's the user choose the size and perhaps also offers the large-address awareness (4 GB) patch – but I don't imagine anyone actually implementing that.

    What I've implemented, tentatively, is a mod component that overwrites the offending values at runtime in the main memory and that could be merged into other mods with little effort (beyond recompiling the DLL). Here's the mod component with its source code:
    https://github.com/f1rpo/Civ4_SMC_PoC (download)
    The name is short for "self-modifying code - proof of concept". (I've written a little bit of a framework.)
    After unzipping, the mod folder should be renamed to match the name of the INI file inside. Caveat: The DLL has assertions enabled; hopefully none will fail, but, if they do, one may have to Alt+Tab out of fullscreen mode to unfreeze the game. Edit #1: And when using 1024x768 resolution, the mod will have almost no effect.

    Works nicely on my end (screenshot; original size is at the bottom), and, for my AdvCiv mod, I've already implemented options on the BUG menu for either canceling out the height adjustment or setting a particular size or adjusting to the field-of-view value. I've only been able to test this on Win 8.1 and Win XP and only with my Complete Edition (CE) from DVD. I did try it with and without the 4 GB patch; no problem there. The virtual memory layout of the EXE should not depend on the OS and hardware, but the OS could still matter because the mod needs to temporarily lift write protections on a few pages of the code segment. This behavior might also cause trouble with virus scanners. Letting VirusTotal.com scan my DLL has not resulted in any detections; however, there could possibly also be detections at runtime based on memory accesses. Another concern are localized versions of the game. I don't know if those use the exact same build of the EXE as the CE (not really sure about the Steam "beta" and GoG version either). I suppose the differences in the memory layout can't be that great. My code already performs a sanity test, and, if that fails, will search for a particular sequence of binary instructions to determine an address offset that aligns the relevant part of the present EXE with the CE EXE. Edit #4: This thread makes me hopeful that localized editions don't actually use a different build of the EXE. Edit #5: Also heard back from a Linux user who didn't seem to have trouble.

    So – do let me know if you encounter or foresee problems or if you have a better idea how to go about this.

    Edit: This view shows the DLL changes relevant for merging this mod pretty well.
     
    Last edited: Jun 23, 2022 at 6:41 PM
  2. raystuttgart

    raystuttgart Civ4Col Modder Supporter

    Joined:
    Jan 24, 2011
    Messages:
    8,502
    Gender:
    Male
    Location:
    Stuttgart, Germany
    @f1rpo
    I like it. :thumbsup:

    I always felt that the Icons were a bit too large and obtrusive.
    Still I liked to activate them from time to time for better strategic overview.

    ------------

    Questions:

    1. Does it affect performance?
    2. Is it compatible to Civ4Col EXE?

    ------------

    Edit:

    I was considering to merge this into WTP or ask one of the WTP team colleagues if they would like to do it.
    But I checked the amount and complexity of changes required and it simply too much for my own skills as a programmer.

    Also I am afraid to introduce bugs that we would later not be able to trace back.
    I consider changing data in memory during runtime extremely dangerous.

    And last but not least I am afraid that it may cause compatibility issues.
    (e.g. with other localizations and versions of the EXE that I do not have and thus would never test.)

    ------------

    So please do not misunderstand me:

    What you have done here needs extremely high skills as a programmer and the quality of your code is great as well.
    But that is also my problem actually. Code on such a high level of skill comes with extreme liabilities.

    Most likely my WTP team members like @Nightinggale or @devolution may maintain such code because they are better programmers than myself.
    But each modder should only merge / implement code he wants to take the burden of liability to fix or maintain on himself.

    ------------

    Summary:

    What you did is great and showed your outstanding skill as a programmer.
    But I do not feel comfortable to integrate code like this into WTP myself.
    (Because as I said, I could not maintain or adapt it in the future either.)
     
    Last edited: May 26, 2022
  3. f1rpo

    f1rpo plastics

    Joined:
    May 22, 2014
    Messages:
    1,197
    Location:
    Germany
    In case that anyone (other than ray :)) ever wants to try porting this to Civ4Col:
    Spoiler :
    Although CvPlayer::getGlobeLayerColors is not part of the Col DLL, it should still be possible to locate the hardcoded base size values in the debugger by placing an assertion in CvArtFileMgr::getInterfaceArtInfo that fails when INTERFACE_PLOTPICKER_OVERLAY is accessed; then investigate the caller (and the caller's callers) in the EXE in the various relevant contexts: resource display toggled on, unit selected, interface message displayed. However, it'll probably suffice to just search the Col EXE with a hex editor.

    I've just done that, and the first four occurrences of 42.0f in the Col EXE are spaced apart similarly enough to the first three in the BtS EXE to suspect that they serve the same purpose: 5F372, 5FDFC, 8D5C0, ADCB7. The first two are quite close together; no such pair exists in BtS, so maybe something was added there in Col. Could change those four on disk through the hex editor for confirmation. Then, to change them at runtime, it's probably sufficient to add 0x400000; this appears to be the "base address" of most executables. I.e. just prepend 0x004 to the addresses found in the hex editor. (Those addresses would already include the "OperandOffsets" in my code.) Could do a manual test upfront through the Memory window of the Visual Studio debugger.

    For the off-screen size 68.0f, I suspect that the Col address is (0x004)8FA7D.
    And in case that someone wants to experiment with changing the size in their Civ4BeyondSword.exe (or maybe Colonization.exe; see above) on disk, here are some instructions using Frhed, an open-source hex editor:
    Spoiler :
    Make a backup of the EXE, start Frhed, open the original EXE (can use drag-and-drop for that), open the Replace dialog (Ctrl+H), searching for <fl:42> (that's a lower-case L) and replacing with e.g. <fl:27>:
    frhed-replace.jpg
    Since we only want to replace the first three occurrences (Colonization: four?), hit
    "Find Next" - "Replace" - "Find Next" - "Replace" - "Find Next" - "Replace"
    Close the Replace window, move the cursor back to the top of the file (Ctrl+Home), open the Replace window again, change the search pattern to <fl:68> and the replacement to e.g. <fl:38>, hit "Find Next" once then "Replace" once.
    Close the Replace window again, save the EXE (Ctrl+S), close Frhed, done.

    To figure out what size to use, my best bet is to calculate
    (42 * 768) / screen_height
    That's ca. 27 when the height is 1200 pixels. I suppose one can also use a fractional value, e.g. <fl:26.88>. It seems that a large field of view will result in bigger plot indicators. If that's a concern (e.g. when using the BUG field-of-view slider), taking the result from above times
    square_root(42 / FoV)
    should mostly cancel that out. (Another small factor is camera distance; plot indicators, strangely, seem to get a bit bigger when zooming out.) E.g. when using FoV=60, the 27 would become ca. 23.
    For the offscreen unit icon size, 1.4 times the base plot indicator size works well for me, e.g. 32 in the example above.

    And you may want to decrease the plot indicator size in Globe view a little bit too. That's a humble XML change.
    Hard to imagine that there's any performance penalty. The brief lifting and reinstating of memory protections might be a somewhat slow operation (no idea really), but, normally, that happens only once upon loading the main map. (If a mod allows players to change the camera's field of view, the EXE will get patched again after such a change.) The EXE should run just as fast or slow as before with those few replaced constants.

    As for compatibility, sure, it would seem prudent to get confirmation from users of various operating systems and builds of the EXE before including such a hack in a release. That said, defensive programming also goes some way. My code already checks whether the relevant code exists at the proper address and will take no action otherwise. Failed VirtualProtect calls (to suspend memory protections) should also be recoverable.

    This would be a more attractive proposition if some other limitations imposed by the EXE could be worked around using the same technique. The speed of combat animations comes to mind (I'd like them a bit faster, no way to do that through the DLL afaik), a wider help text area (WtP and RI have run into that issue), or bypassing the mod name check when loading savegames. But, arguably, it's too laborious to reverse-engineer the EXE for such minor nuisances, even if the modification at runtime were a nonissue.
     
    Last edited: May 26, 2022
  4. raystuttgart

    raystuttgart Civ4Col Modder Supporter

    Joined:
    Jan 24, 2011
    Messages:
    8,502
    Gender:
    Male
    Location:
    Stuttgart, Germany
    I was really annoyed by the fact that I could not change the width of help texts.
    At that point of time I also did not understand why. So thanks for explaining now. :thumbsup:
     

Share This Page