1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Idea for making C3C patch framework

Discussion in 'Civ3 - Creation & Customization' started by Antal1987, May 22, 2014.

  1. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    Any serious game patch requires changing of asm instructions and adding new code. The problem is that there are only about 0x200 (512) bytes of available (unused) space in exe-file, where patch code can be placed. It is inside PE-header allocation space)

    After some experiments with dll injecting and investigating C3C I concluded that it's possible to design all patches in a separate DLL, which can be loaded and unloaded during runtime without code violations.

    There is a DLL in Civ3 installation package, it's called jgl.dll
    Not everyone knows, that this is the most valuable library for the game, because it controls game windows (1 window actually), system events and game graphics.

    That dll is loading inside the following function
    address: 0x006306E0
    Code:
    HMODULE __cdecl f_Load_JGL_Graphics(LPCSTR lpLibFileName)
    {
      HMODULE result; // eax@1
      class_JGL_Graphics *(*p_Get_Graphics)(void); // eax@3
      class_JGL_Graphics *_Graphics; // esi@5
    
      result = lpLibFileName;
      if ( lpLibFileName )
      {
        result = LoadLibraryA(lpLibFileName);
        JGL_Lib_Module = result;
        if ( result )
        {
          p_Get_Graphics = GetProcAddress(result, "get_graphsy_object_ptr");
          if ( p_Get_Graphics )
          {
            _Graphics = p_Get_Graphics();
            if ( !_Graphics )
              JGL_Lib_Free();
            JGL_Graphics = _Graphics;
            result = _Graphics;
          }
          else
          {
            JGL_Lib_Free();
            result = 0;
          }
        }
      }
      return result;
    }
    
    JGL provides many useful methods but the game imports only 1 of them: "get_graphsy_object_ptr". The other methods are getting accesed "somehow".

    Here is how it works.
    JGL in get_graphsy_object_ptr creates and returns pointer to an object of class:
    Code:
    #pragma pack(push, 1)
    struct class_JGL_Graphics
    {
      struct_JGL_Graphics_vtable *vtable;
      int field_4[74];
      int Screen_Width;
      int Screen_Height;
      int field_134;
      HDC DC;
      int field_13C;
      HMODULE Main_Module;
      int hWnd;
      class_PCX_Image *PCX;
    };
    #pragma pack(pop)
    
    General functionality provided by JGL is exporting in vtable (virtual functions table) of that class object.

    The game after receiving the object pointer saves it along with library's module handle.
    JGL_Lib_Module is actually a pointer to a place where the JGL is located. That value is used when the game finalizes:
    address: 0x006306A0
    Code:
    void __cdecl JGL_Lib_Free()
    {
      if ( JGL_Graphics )
        (JGL_Graphics->vtable->m03)();
      if ( JGL_Lib_Module )
      {
        FreeLibrary(JGL_Lib_Module);
        JGL_Lib_Module = 0;
      }
      JGL_Graphics = 0;
    }
    
    Patch framework can be made similarly.

    Few worlds about virtual tables. When a simple class creates new instance, it's 1-st field has offset = 0. But if a real class inherits from an abstract virtual class (interface) it's objects have the virtual table pointer at offset = 0.
    Virtual table consist of pointers to virtual methods with predefined signatures.

    The compiler places virtual methods into vitrual table in the order corresponding to order of those methods in the class. For example, the table for class:
    Code:
    class iOpenable
    {
        public:
        virtual ~iOpenable(){}
     
        virtual void open()=0;
        virtual void close()=0;
    };
    
    would be:
    0x00 - destructor (~iOpenable)
    0x04 - open
    0x08 - close.

    So, lets make a special class for storing patch functions:
    Code:
    class IC3CPatchCollection
    {
        public:
        virtual ~IC3CPatchCollection(){}
     
        virtual void patch_1()=0;
        virtual void patch_2()=0;
        ...
        virtual void patch_N()=0;
    };
    
    class C3CPatchCollection: public IC3CPatchCollection
    {
        public:
        C3CPatchCollection(){}
        virtual ~C3CPatchCollection(){}
     
        virtual void patch_1(){}
        virtual void patch_2(){}
        ...
        virtual void patch_N(){};
    };
    
    IC3CPatchCollection is just an interface, it cannot have instances. But C3CPatchCollection is a real class. It may have instances.

    Functions patch_1, patch_2, ... patch_N may have various signatures. It's order would be fixed. Therefore C3CPatchCollection class object would have a virtual table (offset: 0) with the following structure:
    0x00 - patch_1;
    0x04 - patch_2;
    ...
    4 * (N-1) - patch_N;

    Now lets define an export function which creates an instance of C3CPatchCollection and returns it.

    Code:
    #define PATCH_FRAMEWORK_API __declspec(dllexport)
    
    PATCH_FRAMEWORK_API C3CPatchCollection * init_patch_collection(void)
    {
    	return new C3CPatchCollection();
    }
    
    Now we have to make some initial patch inside C3C exe.
    1) allocate 2 int values in free space for storing:
    - HModule for out patch library
    - pointer to C3CPatchCollection object
    2) write inside free space an asm function which would load the library and call it's export function (Init_Function)
    3) write inside free space an asm function to call C3CPatchCollection object's destructor and free library (Dispose_Function)
    4) make somewhere in f_Load_JGL_Graphics an injection to calling Init_Function
    5) make somewhere in JGL_Lib_Free an injection to call Free_Function

    Now, how to use patch methods.
    Lets assume we already define a pointer to C3CPatchCollection object:
    C3CPatchCollection * Patch_Framework

    the call in C++ would be this:
    Code:
    Patch_Framework->patch_1()
    That call in asm would be like this:
    Code:
    mov ecx, DS:Patch_Framework
    mov ecx, [ecx]
    call [ecx+patch_1_offset]
    In summary: that approach provides a lot of possibilities:
    1) It can be overwritten any difficult function or even function group which represents an entire behaviour.
    2) There is no need writting complicated code in asm. C++ can be used instead. Asm is required only for calls injection.
     
  2. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,132
    Location:
    DFW, Texas USA
    :worship: :w00t: :popcorn: :yeah:

    Wow, being able to hook into the game like that. Very cool. Opens the door to ideas possibly not even dreamed of yet.

    If this gets traction someone will have to figure out how to keep track of the patches used for any given save file.
     
  3. PiotrG

    PiotrG Chieftain

    Joined:
    Aug 22, 2012
    Messages:
    54
    It's very interesting. Opens up new possibilities. Good luck in further trials
     
  4. T-mun

    T-mun King Numa

    Joined:
    Aug 13, 2004
    Messages:
    2,300
    Location:
    France
    I don't understand most of what you're talking about, but anything you can do to make this game better is welcome! :goodjob:
     
  5. Puppeteer

    Puppeteer Emperor

    Joined:
    Oct 4, 2003
    Messages:
    1,132
    Location:
    DFW, Texas USA
    Dude, this would be the next best thing to having the source code released as open source. He's talking about a framework for arbitrary new/replacement game code.

    Some reverse engineering is required, but a lot is already known/inferred about game mechanics, and clearly Antal-1987 has the ability to work out in-memory data structures and program code. He's already made a couple of amazing patches; if he can generalize a framework I bet a few other people can whip up some code to do neat things within the framework.

    The possibilities are mind-boggling. Perhaps making it so it can run on 720-high or 600-high displays, perhaps being able to add actual 3d models for display or play via web browser. Launching arbitrary content (videos!) for wonder completion, age cut-scenes and such. Game rule changes. For those who care, a "city view" screen that has all the new wonders and/or modded wonders. We know there are some unfinished stub features in the game; those could be fleshed-out. Trade route calculation could be looked at and perhaps sped up. Who knows what we can and can't do?

    I don't recall offhand what the state of hardware was in 2001-2005 when the game was released, but I bet we have more RAM now than we had hard drive space back then. And we pretty much all have multi-core PCs now. I just haven't begun to think of what can be done here.

    Edit: Oh, and since he's talking about intercepting calls from a dll, I think this means we aren't limited to C++; we should be able to write patches in any language we want as long as the dll interface is the same. CivAssist and MapStat do very cool things based off of save file updates; with this framework I think we would have access to the same information live in-memory.

    Edit 2: Here are some ideas: a mod to save and load save games, existing mods, .ini files and any changeable data from a user folder instead of Program Files (to avoid the VirtualStore problems). Perhaps a menu to dynamically load graphics mods of choice and/or pick sets of mods; I want this BIQ, this tileset and that collection of modded civs/leaderheads.

    Edit 3: Holy crap! We could probably fix things like worker and city automation to be not quite so brain-dead. OMFG that means we could make the AI more effective. :eek: :eekdance: :wow: Seriously, if the AI simply had more effective workers, what would that do to your games? If they did something besides irrigate grassland in Despotism and spend two movement turns to build a pointless disconnected road then two more turns moving to the next tile? I assume that the automation code is the same for AI units and cities as it is for the player using automation; if I'm correct then fixing it once helps players using automation *and* makes the AI a bigger challenge.
     
  6. general-jcl

    general-jcl King

    Joined:
    Nov 5, 2005
    Messages:
    617
    Location:
    France
    Does it means that the modding of the game has no boundaries anymore and that we can now consider everything including exceeding the limit of 256 buildings/wonders?
     
  7. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    Absolutely, but the costs differ sharply.
    For example, Civilizations have been coded in a static structure array:
    class_Leader Civilizations[32].
    To overpass civ limit of 32 it requires to make this array dynamic.
    But there are at least 794 direct references and a lot of another refs to it's offsets. And all of this references have to be patched.
     
  8. Civinator

    Civinator Blue Lion Supporter

    Joined:
    May 5, 2005
    Messages:
    5,927
    Here a revolution is going on! That´s brilliant! :clap::worship:
     
  9. general-jcl

    general-jcl King

    Joined:
    Nov 5, 2005
    Messages:
    617
    Location:
    France
  10. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    Of course differenent languages can be used. But there are some constraints/conditions have to be followed. Dll interface is the same, but framework initial patch can afford only one function to be imported inside the game exe, because of free space lack.
    Any patch functions can be called only using interface mechanics.
    That requires from the language and it's compilier to make a class object predictable. That means the functions' order and it's offsets have to be constant.
     
  11. Virote_Considon

    Virote_Considon The Great Dictator

    Joined:
    Jul 7, 2004
    Messages:
    9,331
    Location:
    Skaville UK Reputation: 1
    So would it be possible to do things like add terrain types too? Or perhaps have different graphics for hills and mountains based on the underlying terrain?

    Either way, exciting stuff!
     
  12. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    As I said, the costs differ sharply. It would be hard work to provide new graphics, but it would be certainly easier than fixing civ count limit :)

    After all, I almost comletely studied map rendering process.
     
  13. Gray Wolf

    Gray Wolf Winter Wolf

    Joined:
    Jun 2, 2012
    Messages:
    482
    Location:
    Ravenswood
    I'm not too interested in making the game have more than 32 civilizations. However, I'm thinking that we may be able to write a function that reads information from custom xml files and make our own screens that uses the custom xml. But I guess that would require tons of reference patches also... :(

    Anyway, if we could create xml nodes (say one for CivName=Civilization Name), we could have as many civilizations as we want. Also, we could use it to read information about units, leader names for different era's (which is something I've always wanted), and possibly different culture groups so we can have unique city graphics for each civilization.

    I'm just dreaming of possible solutions involving the function you were talking about before that can work via loading a dll. And having custom xml file could help memory issues and limited array type issues, right?

    Anyway, what are your thoughts on this?
     
  14. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    Well, I already know how the game renders graphics. So I can even obtain temporary imaging results like this:

    or view resources array in memory directly, like this:


    I know which functions maitain loading resources for most of general forms and controls, which of them renders it and where the forms store it.

    So, it's pretty possible to inject into renderers. The advantage is that graphic resources are refered only by few functions. And these functions are related to rendering process and nothing else.
     
  15. WildWeazel

    WildWeazel -- Turns to Completion

    Joined:
    Jul 14, 2003
    Messages:
    6,838
    Location:
    %CIV3%\Conquests\Scenarios\
    Well that changes everything. And here I was about to start rewriting it from scratch.
     
  16. gwendoline

    gwendoline Prince

    Joined:
    Dec 9, 2010
    Messages:
    336
    Location:
    Forgotten in Siberian woods
    I believed nothing is impossible here! You're the truly Civ Fan ;)

    My opinion
    Spoiler :
    If such traditional and major 'limits' as 32 civs, 256 buildings, 5 culture groups etc. will be changed, it could become a new game, a Civ3 Reborn or something. For our original Civ3 (Conquests) it would be enough to fix some annoying bugs and change\add a little features as usual patch does.
     
  17. warmwaffles

    warmwaffles Programmer

    Joined:
    Jan 15, 2004
    Messages:
    2,255
    Location:
    Texas
    Hats off to you for reverse engineering this. I can't even fathom having to reverse engineer stuff with almost a non existent tool chain in windows. :goodjob: I look forward to seeing what you can do.
     
  18. Steph

    Steph Multi Many Tasks man Retired Moderator

    Joined:
    Sep 1, 2002
    Messages:
    18,162
    Gender:
    Male
    Location:
    Pont de l'Arn, FRANCE
    I have to ask here again, as retired moderator.
    This seems a very good idea from a modder/player point of view.
    But from the forum point of view, we need to be sure it doesn't violate the anti piracy rule.

    Can you please explain how you handle this problem?

    I believe that if it is something than "hooks" into the game to expand functionalities, or fix it, it can be very interesing. And I'd gladly contributes with ideas.

    If you remember, years ago I compiled a list of things to "fix" and possibly to add to the game, if we had the source files. I can probably find it again and propose it for discussion.

    However, can this be done without altering the exe itself and removing any protection to the game?
    I mean, if requires player to own the game, and just put something in the install folder, and then it works, I think it should be fine.
    But if it means a new exe than could allow playing the game without buying it, then I'm afraid it's against the forum rules.

    Can you explain a little more how you are doing it?
     
  19. Steph

    Steph Multi Many Tasks man Retired Moderator

    Joined:
    Sep 1, 2002
    Messages:
    18,162
    Gender:
    Male
    Location:
    Pont de l'Arn, FRANCE
    Actually, this is already possible with my "Expanded editor concept". The problem I had was it relied only on saved game, to change the "rules" in the saveame.
    So when I detected, with the autosave, that the player has changed an era, I could generate a new savegame, with a different leader name. Then you had to load, and you had a name/gender/graphics different for each era!

    But you had to change the autosave manually. Here, there could be several possibilities:
    - It could be changed directly into memory.
    - Or if Antal1987 can manage loading /saving, may be it's possible to have an hybrid approach: my expanded editor system automatically analyze the autosave, generate a new save automatically, and loads it. The player doesn't have the choice, doesn't have it to do it, it just get a kind of patience window like "Please wait while the game updates".

    Many possibilities.
     
  20. Antal1987

    Antal1987 Warlord

    Joined:
    Sep 4, 2013
    Messages:
    160
    Location:
    Karelia, Russia
    It's not creating a "pirate" version. Patches still require legal version. And I do not publish "pirated material, telling people how to avoid copy-protection"

    So I think it doesn't violate anti piracy rules.
     

Share This Page