WildWeazel
Carthago Creanda Est
Discuss it here!
Summary TBA
Summary TBA
Last edited:
Analysis: BIQ/SAV Tools
- Thread starter WildWeazel
- Start date
WildWeazel
Carthago Creanda Est
Quoting generously from @Quintillus in an email thread:
----
On Cost-Benefit of SAV File Modification Utilities
The tl;dr is probably that if the cost were low, you'd see more of this already in my editor, and what you do already see (including in the most recent update from last month) is among the low-hanging fruit in this area.
The main problems are:
- Inflexibility in the SAV format. If the format doesn't support more than 31 civs, you're not going to be able to support more than 31 civs.
- Game engine limitations. The famous 512-city limit could be bypassed by adding more to a SAV file, but as the mixed results with Antal1987's modifications have shown (IIRC even with his patch it becomes flaky around the 550-city mark), it's going to be difficult to teach the old game engine new tricks without its source code.
- Incomplete knowledge of the SAV format. We know it less well than the BIQ format.
The above three all limit what we can do. We could have the best idea in the world, and it might conceptually sound simple, but if one of the above affects it, the cost to implement it via any sort of SAV modification utility would be extremely high if not impossible. This is where a separate engine is necessary.
In addition, there are some further practical limits:
- Ability to quickly apply changes.
- Ability to accurately apply changes when trying to apply them quickly
This is the "have to reload the SAV" problem, basically. No one wants to have to reload their game every turn, or even multiple times a turn, to reload changes. This leads to two options:
On Applying SAV Updates Via Loading the Game
In terms of implementing changes via SAV files, this is the "safe" option, the one with an achievable cost-benefit ratio, and the one already present to some degree in my editor (including some updates in December. Kirejara and I have discussed this in the editor thread recently, and Kirejara has been making practical use of it.
So far I've gone for the low-hanging fruit in this area, specifically updating the embedded BIQ rules in the SAV, and I could expand this out to updating any of the BIQ rules (with a SAV reload) fairly easily; this is the best cost-to-benefit ratio available. More difficult is updating SAV-specific structures (e.g. things on the map in the SAV, which are not the same as the map sections of the BIQ), due primarily to the lesser knowledge of the SAV format (and to some extent the already-existing code for handling the BIQ).
More difficult still is automatically watching SAVs and dynamically alerting the user based on a scriptable set of conditions. While I've done some preliminary proof-of-concept work to show this can be done, having it work universally is a lot of work, and realistically it would be necessary to know what the most in-demand functionality in that area would be and focus on that rather than try to allow events based on every condition imaginable. This also counts as an area where collaboration (and a plan to use set of this technology in a mod) would likely increase progress, and so far that hasn't really existed in any sustained fashion beyond the conversations with Kirejara on the lower-hanging fruit.
On Applying SAV Updates Without Loading the Game
It's possible in at least some cases to apply updates without reloading. However, this adds several additional hazards:
- It requires knowledge of the structure of the game in memory, which does not correspond 1:1 with the structure of the SAV file (although there is some overlap).
- There is a risk of applying updates based on what you think the current structure is, but finding out the game has modified the location of data.
- There is the risk of being "too slow", and applying changes while the AI is making updates, or after the player has made an update, and causing havoc, i.e. potentially an invalid game state, undoing an AI/player move, or crashing the game.
- You still can't do any of the things mentioned in the "main problems" section above
- You also cannot do things like add a city or unit (which you can when updating via loading an updated SAV), unless you could also figure out how to update the in-memory data structures Civ uses so the game engine knows about it. This ties into the next problem...
- To really do this accurately and reliably, it requires advanced Windows-specific knowledge and likely knowledge about compiled C++ programs and their structure that I don't have, and am realistically not going to be an expert in any time soon. Antal1987 is the only forum member I know of who I might consider an expert in this area.
- The way to do this is to connect to the Civ3 game using Windows-specific APIs that are intended to be used by debuggers. While this works, it's definitely a bit of a hack, and I have some concerns that anti-virus software may find software that makes heavy use of these APIs to be suspicious, and not without reason. To extend game functionality with these APIs is innocuous; to examine the in-memory contents of a password manager with could well be nefarious.
Short of having a new game engine, I think the benefit of this approach combined with the "SAV Updates Via Loading the Game" is the highest possible; a hybrid of real-time updates and reloading a SAV 5-10 times per campaign for larger overhauls, as well as an event-based SAV monitor (perhaps complemented by a limited memory-based monitor), could achieve essentially as much as is possible with the current game engine.
Final Cost-Benefit Analysis
But the cost becomes exponentially higher as you go farther down this road, and you'd still have non-negligible limitations. IMO, not too far down this road you hit the point at which the cost has risen far more than the benefit, and pursuing the C7 idea would be a better option (particularly as you could mop up any remaining low-hanging fruit found along this path in parallel).
I'd probably put the point of inflection as where you can update any BIQ-based rules via reloading the SAV, and perhaps have a very limited and targeted-to-what-is-needed-for-specific-scenarios SAV monitor. Some low-hanging fruit in the SAV structure could be tossed on, particularly if an interesting new scenario could make good use of it, but much beyond that would rapidly be costlier than the benefit that would be derived from it.
----
On Cost-Benefit of SAV File Modification Utilities
The tl;dr is probably that if the cost were low, you'd see more of this already in my editor, and what you do already see (including in the most recent update from last month) is among the low-hanging fruit in this area.
The main problems are:
- Inflexibility in the SAV format. If the format doesn't support more than 31 civs, you're not going to be able to support more than 31 civs.
- Game engine limitations. The famous 512-city limit could be bypassed by adding more to a SAV file, but as the mixed results with Antal1987's modifications have shown (IIRC even with his patch it becomes flaky around the 550-city mark), it's going to be difficult to teach the old game engine new tricks without its source code.
- Incomplete knowledge of the SAV format. We know it less well than the BIQ format.
The above three all limit what we can do. We could have the best idea in the world, and it might conceptually sound simple, but if one of the above affects it, the cost to implement it via any sort of SAV modification utility would be extremely high if not impossible. This is where a separate engine is necessary.
In addition, there are some further practical limits:
- Ability to quickly apply changes.
- Ability to accurately apply changes when trying to apply them quickly
This is the "have to reload the SAV" problem, basically. No one wants to have to reload their game every turn, or even multiple times a turn, to reload changes. This leads to two options:
On Applying SAV Updates Via Loading the Game
In terms of implementing changes via SAV files, this is the "safe" option, the one with an achievable cost-benefit ratio, and the one already present to some degree in my editor (including some updates in December. Kirejara and I have discussed this in the editor thread recently, and Kirejara has been making practical use of it.
So far I've gone for the low-hanging fruit in this area, specifically updating the embedded BIQ rules in the SAV, and I could expand this out to updating any of the BIQ rules (with a SAV reload) fairly easily; this is the best cost-to-benefit ratio available. More difficult is updating SAV-specific structures (e.g. things on the map in the SAV, which are not the same as the map sections of the BIQ), due primarily to the lesser knowledge of the SAV format (and to some extent the already-existing code for handling the BIQ).
More difficult still is automatically watching SAVs and dynamically alerting the user based on a scriptable set of conditions. While I've done some preliminary proof-of-concept work to show this can be done, having it work universally is a lot of work, and realistically it would be necessary to know what the most in-demand functionality in that area would be and focus on that rather than try to allow events based on every condition imaginable. This also counts as an area where collaboration (and a plan to use set of this technology in a mod) would likely increase progress, and so far that hasn't really existed in any sustained fashion beyond the conversations with Kirejara on the lower-hanging fruit.
On Applying SAV Updates Without Loading the Game
It's possible in at least some cases to apply updates without reloading. However, this adds several additional hazards:
- It requires knowledge of the structure of the game in memory, which does not correspond 1:1 with the structure of the SAV file (although there is some overlap).
- There is a risk of applying updates based on what you think the current structure is, but finding out the game has modified the location of data.
- There is the risk of being "too slow", and applying changes while the AI is making updates, or after the player has made an update, and causing havoc, i.e. potentially an invalid game state, undoing an AI/player move, or crashing the game.
- You still can't do any of the things mentioned in the "main problems" section above
- You also cannot do things like add a city or unit (which you can when updating via loading an updated SAV), unless you could also figure out how to update the in-memory data structures Civ uses so the game engine knows about it. This ties into the next problem...
- To really do this accurately and reliably, it requires advanced Windows-specific knowledge and likely knowledge about compiled C++ programs and their structure that I don't have, and am realistically not going to be an expert in any time soon. Antal1987 is the only forum member I know of who I might consider an expert in this area.
- The way to do this is to connect to the Civ3 game using Windows-specific APIs that are intended to be used by debuggers. While this works, it's definitely a bit of a hack, and I have some concerns that anti-virus software may find software that makes heavy use of these APIs to be suspicious, and not without reason. To extend game functionality with these APIs is innocuous; to examine the in-memory contents of a password manager with could well be nefarious.
Short of having a new game engine, I think the benefit of this approach combined with the "SAV Updates Via Loading the Game" is the highest possible; a hybrid of real-time updates and reloading a SAV 5-10 times per campaign for larger overhauls, as well as an event-based SAV monitor (perhaps complemented by a limited memory-based monitor), could achieve essentially as much as is possible with the current game engine.
Final Cost-Benefit Analysis
But the cost becomes exponentially higher as you go farther down this road, and you'd still have non-negligible limitations. IMO, not too far down this road you hit the point at which the cost has risen far more than the benefit, and pursuing the C7 idea would be a better option (particularly as you could mop up any remaining low-hanging fruit found along this path in parallel).
I'd probably put the point of inflection as where you can update any BIQ-based rules via reloading the SAV, and perhaps have a very limited and targeted-to-what-is-needed-for-specific-scenarios SAV monitor. Some low-hanging fruit in the SAV structure could be tossed on, particularly if an interesting new scenario could make good use of it, but much beyond that would rapidly be costlier than the benefit that would be derived from it.
Similar threads
