(Warning: The post that follows is lengthy)
Howdy all! I got some outside-of-work Internet today, so I was able to stop by... sounds interesting. Reading through the first couple pages, I thought, "This reminds me a lot of Steph's expanded game", and alas, I was pretty sure that the extra era effort was not going to be successful.
Post 40 is pretty much on track, in particular as to what's possible and what probably isn't (although I don't yet see how different shields/commerce/food per terrain per civ is possible - the idea in post 47 would work, and is a rather ingenious way to do it, but you'd still be limited to two variations max unless I'm missing something. Maybe 4 if you count the Agricultural flood plains/desert bonus as distinct). But there's one sentence in particular that I think is relevant:
I do guess that such a scripting tool will be possible only if save files can be edited and if the game reloads a temporary cache between turns.
There's two basic ways to go about this. Reloading the game to get the changes, and not reloading the game to get the changes. Steph's Expanded Editor goes for the former, which is also the easier one to go for (and probably the less-limited one as well). It does indeed require editing the save file. However, it also would require reloading the game, which is time consuming. I don't know of any way to get Civ to automatically load up changes, particularly in a timely fashion (involved scenarios can take quite awhile to load, which can really interrupt the flow of the game). With changes once per era, that's acceptable. But if you want events to happen more often than that, you'll have to be pretty careful to avoid requiring reloads so often that players decide it's not worth it. The example of switching terrains to landmark and back, as in post 47, wouldn't really work in this model - at least, I know I wouldn't be willing to reload the game every turn to keep that up to date.
I do think this approach could be expanded beyond what Steph's expanded editor currently allows (although I haven't played with it enough to know exactly what the capabilities and limitations are, so perhaps it allows more than per-era switches and I'm just not aware of it). It also would allow, for example, adding in new cities. But if the reloads became too frequent, I think it would become a significant disadvantage for the scenario.
The other way, and the one that I think is more exciting (at least, if you don't do both) is making changes without having to reload the game, and it's that one that I've been pondering off and on this fall. Creating a new .exe that could handle scripts would be one way to do that, but at that point you might as well just build a whole new game. Perhaps you could build a game that both expanded modder's possibilities, and was capable with Civ3 scenarios. It could load and play BIQs, but also have new features. However, you'd also have to have AI, graphics support, etc... it would be nice to have better AI and higher-resolution/color depth graphics, but it would take a really long time to do it all.
So what I was thinking was, keep using the existing .exe, but add a helper program that could modify what Civ3 saw while it was running. In other words, what
WildWeazel said - access the game's memory at run time (and which specific utilities do that? I'm curious what they currently do, but I'm drawing a blank on which ones currently do that). So, essentially, you'd be able to set up triggers (such as population of world above x, year is at least y, or number of gunpowder units in existance at least z). Then, this helper program would (either by examining memory or by examining autosave files - at this point I'm favoring autosave files since it would be easier and all the information you'd need would be there anyway) see what triggers had happened, and do whatever the effect was, such as setting two civilizations to be at war, reducing population (due to a plague, perhaps), increasing how many shields mines produced, or even changing certain tiles' terrain. These changes would happen directly to the game, with no reload required, and would be saved in the next autosave. No reload would be necessary. It probably would be desirable to have some sort of notification of this happening, however. I'm thinking an overlay, such as what CivAssistII uses, might be a good starting place for that (although I have no experience in how to do that).
There would be some limitations to this method. For example, I don't know of a good way (and less yet, a reliable one) to add memory to the game. So adding units and cities wouldn't be possible. You'd still have to reload to do that (and thus a hybrid approach between reloading and automatic changes might be best). In general, adding or deleting anything wouldn't be possible - just changing what's already there (although I may be wrong about the deleting part). But I think having automatic changes primarily, and a reload to add/remove stuff on rare occurrences (perhaps every x turns, or for
major events, such as an American landing in a WWII North African scenario) could provide a lot of new options.
Technically, you would still have a new .exe, but you'd run both the Civ3 one, and the new one. The new one would present itself as a debugger, and would thus technically be debugging Civ3 (although in reality, adding new features to it). Right now, I'm leaning towards the idea that the modder would use an enchanced scenario editor that allowed events (probably starting with a limited set of triggers and results). They'd then save the scenario, which would result in a regular .BIQ file, and some new event file. Then, to play the scenario, you'd load it up like any old scenario, but you'd also start the new .exe, and give it the event file. It would then monitor the autosave folder, like CivAssist II currently does, and whenever it saw a new autosave, it would check and see if any triggers applied. If they did, it would change the memory of the Civ program, and the changes would apply in game. It could also provide a pop-up for the event, if desired. My current thought was that I'd use my editor as a base for the event editor (since you'd need to be able to edit the standard stuff anyway), and its I/O code for monitoring the autosaves (this would need to be enhanced, however - CivAssist II would provide a better base in that regard. I haven't really added any SAV support - the Input From SAV just extracts the BIQ part of a SAV and doesn't do anything with SAV-specific data). So, that would all be in Java (or Java + what CivAssist is in - C++ or C#?). Then, there'd be another program that handled the actual updating of Civ. If the autosave monitor detected that an event happened, it would create a binary file describing what needed to be updated, which the Civ-updater program would read in, and use to update Civ. That program would likely be in C++, since it would need to be able to access the Windows APIs needed to act as a debugger (and the equivalent may be possible on OSX, too, but let's worry about getting one thing working first).
Would it work? I think so. Having a program update Civ3's memory with a desired value is possible, and it works on Windows 8.1, so I'm confident that getting it to work on XP through 8.1 won't be too difficult (maybe 2K, too). The two things that I'm concerned about are whether it would be fast enough, and the possibility of something else being updated while it's updating. So far, I haven't found a good way to predict where in memory a certain section of information about the game is. That means searching for the information that needs to be updated, which takes time. It's predictable enough that I don't have to search the whole 4 GB address space (which even in memory takes awhile), but it might take several seconds to make a large set of changes. It might also be necessary to add in a delay so that the game isn't doing anything (i.e. is the game ready for input and not doing anything when the autosave is created, or does it keep doing stuff for a second or two after the autosave is ready? I'd have to test). Assuming all this worked, it might be possible to look at mid-turn updates, but updates during the AI turn would be off-limits due to the unpredictable nature of what would happen.
So far, this is very much theoretical. I've tested updating a Civ3 memory while it's running, and while I can update the value I want to update, that's it so far. There's a lot left to go - both for support for making scenarios for it, and to know if it really will scale up enough to work in practice. I hadn't planned to discuss it on CFC until I was more confident that it really would work, so as not to unduly set expectations, but it seems like this is as good a time as any to mention it.
I hadn't heard of Angel Script before, and I'll have to look at it to see if it might be a better way to go about some of this than what I've thought of up to this point. I've downloaded it, its documentation, and a bit of info about Jazz Jackrabbit 2's use of it, but it's something that will take some time to investigate.
Definitely some interesting ideas here - I'll be sure to check back, and likely give an update on how feasible some of this looks as I investigate a bit more. It's definitely at the "intriguing ideas" phase and not the "definitely possible" phase, however - it would be great if some of these limits can be broken, but I don't want to set the expectation that it is possible when it may not be.
It probably will be a few days to a week and a half before I'm back, too - I'll still be reachable by e-mail (and that's still a valid way to reach me), but if I don't show up in the thread for awhile, don't be too surprised.
