This post is primarily aimed at modders, or anyone else with a software background. Anyone is welcome to chip in though, or to ask questions.
Outline
I was thinking about a possible alternative to the start-again approach of AXXXE to extending the lifetime of C2C. The following is an outline:
Suppose, instead of trying to start from scratch (AXXXE approach), we were to stick with the Civ4 engine, and seek to restructure the DLL as follows (this is a recipe of fairly [haha] mechanical steps):
1) Introduce proxy classes for all entities visible to the game engine (in much the same way as we did for maps when adding viewports). These proxies would contain no intelligence and no data apart from that required by the interface to the game engine (so basically things that are 'visible' like a unit's strength, but not internals like all its combat modifiers). Any state-modifying methods, or request for data that cannot be satisfied without calculations involving hidden variables are passed to the underlying real entity class. All external APIs go to the proxies. Python APIs also have to talk through the proxies (why will become apparent shortly).
2) Move all the data in the proxies out of those classes into flat structs with a 'dirty' flag. On read the proxy consults the dirty flag, if not set reads the data from the struct, else calls an 'update' method on the underlying class, which responds by updating the struct and clearing the dirty flag.
3) Split the DLL apart so that the actual DLL contains only the proxies, and the remainder moves into it's own EXE (which is auto-launched by the DLL). The flat structs move into shared memory. The direct calls that remain move into some sort of IPC (probably a simple command/response over a duplex named pipe, or something).
4) Change the build settings on the new EXE - multithreaded runtime, 64 bit, optimization flags set to make appropriate use of more modern CPU architectures
5) Multithread the stuff in the EXE as we see fit
Pros and cons:
Relative to the current situation:
Pros:
Relative to the AXXXE approach:
Pros:
Cons:
It certainly wouldn't be a small amount of work (or one that would be certain to yield a sufficiently preformant result, so it is somewhat speculative), but I think it would be doable in something like the timeframe of 1-2 release cycles.
Outline
I was thinking about a possible alternative to the start-again approach of AXXXE to extending the lifetime of C2C. The following is an outline:
Suppose, instead of trying to start from scratch (AXXXE approach), we were to stick with the Civ4 engine, and seek to restructure the DLL as follows (this is a recipe of fairly [haha] mechanical steps):
1) Introduce proxy classes for all entities visible to the game engine (in much the same way as we did for maps when adding viewports). These proxies would contain no intelligence and no data apart from that required by the interface to the game engine (so basically things that are 'visible' like a unit's strength, but not internals like all its combat modifiers). Any state-modifying methods, or request for data that cannot be satisfied without calculations involving hidden variables are passed to the underlying real entity class. All external APIs go to the proxies. Python APIs also have to talk through the proxies (why will become apparent shortly).
2) Move all the data in the proxies out of those classes into flat structs with a 'dirty' flag. On read the proxy consults the dirty flag, if not set reads the data from the struct, else calls an 'update' method on the underlying class, which responds by updating the struct and clearing the dirty flag.
3) Split the DLL apart so that the actual DLL contains only the proxies, and the remainder moves into it's own EXE (which is auto-launched by the DLL). The flat structs move into shared memory. The direct calls that remain move into some sort of IPC (probably a simple command/response over a duplex named pipe, or something).
4) Change the build settings on the new EXE - multithreaded runtime, 64 bit, optimization flags set to make appropriate use of more modern CPU architectures
5) Multithread the stuff in the EXE as we see fit
Pros and cons:
Relative to the current situation:
Pros:
- Most non-graphical memory usage is moved out of the Civ4 engine process, giving it more room to work for its graphics - less MAFs
- Multi-threading is a bit easier because we have more complete control over the flow control
- 64 bit process means we can scale a lot better (at least for non-graphical aspects)
- By caching the shared data it should be possible to page in (to the Civ4 engine process) only what it needs at the time (again better memory scalability)
- Freedom to optimize without having to worry about the runtime constraints the Civ4 EXE imparts (so can move to a modern compiler for the EXE with modern code optimization)
- Potential for performance issues with Python since it has to funnel everything across the IPC boundary via the proxies, so some patterns of use will potentially slow down
- XML loading might have to stay in the DLL (certainly it will if we want to continue to use the provided parser), which implies the need for some mechanism to pass the parsed data across the IPC boundary (actually the cached version that AIAndy added might serve the purpose, so writing it is always DLL-side, but reading it is EXE side)
- No benefit (and indeed extra overhead) for people with 32 bit OS's or less than 4G RAM
Relative to the AXXXE approach:
Pros:
- No grey areas in regard to copyright - we are clearly still only operable under a bought CiV4
- Full compatibility with existing assets and asset-generation techniques
- Much easier
Cons:
- Stuck with C++ (well, in principle you could replace the EXE part with anything, provided it could expose the right shared memory structures, but practically...)
- Stuck with current codebase, with all its warts, and non-generic solutions (but that's also why this approach is way easier of course)
- Stuck with limitations of the Civ4 engine for graphics - that limits scalability due to the memory-hungryness of it, and still doesn't help us with multimaps except via the viewport mechanism as we are intending now
It certainly wouldn't be a small amount of work (or one that would be certain to yield a sufficiently preformant result, so it is somewhat speculative), but I think it would be doable in something like the timeframe of 1-2 release cycles.
