Pazyryk
Deity
- Joined
- Jun 13, 2008
- Messages
- 3,584
Contrary to the title of the post, this is really a tutorial about how to deal with the potential ill effects of deleting. I'm assuming a basic knowledge of XML and SQL sufficient to actually delete an item.
I've seen this statement in various forms from some of the best modders here. The same modders then identify the specific issues related to deleting (e.g., as Spatz does in the post I'm quoting). I'll go one step further here and tell you how to address these issues.
Further, I'm a contrarian so I'll say that (depending on your mod) deleting things is Good. If it's a total conversion mod, do it. If you are removing a whole era, I'd say do it but others will disagree. If you just want to remove one thing (e.g., Giant Death Robot) it is much easier to disable (e.g., make cost = -1) rather than delete.
Note that everything I say below applies to deleting items from the "main" tables -- i.e., anything that has a Type and ID. You can delete rows from "subtables" without worrying about this stuff at all. Also, note that the trend in patches has been to make the game more and more robust to these issues (especially #2 below). But this is probably a curse rather than a cure: an outright crash is much preferable than an insidious and inconsistent error.
1. Find references to deleted items in both XML and Lua (then remove or decide that deletion is not a good idea after all)
Depending on the table, you may have to have some sophistication in Lua programming and/or Civ5 UI to be able to do this. The first step is to identify all occurrences of your deleted item in base game XML or Lua files. Your friend here is a good application for searching text in files. I use Windows Grep (whoward69 has another that he recommends frequently in his posts). There is a more sophisticated DB-ish way to do this but you don't need that. These cross-references are mostly in Gameplay and UI folders but they can be elsewhere (e.g., DLC Babylon Lua references TECH_WRITING). You may want to do these searches first before you decide whether deletion is the right way to go: some things are used a lot in base Lua (e.g., ResourceTypes and TerrainTypes in AssignStartingPlots.lua) or UI (e.g., YieldTypes, but I doubt you are going to want to delete any of those), others never.
Where you find these in XML (non-UI), just delete that row (if it is a "subtable") or the column entry (if it is a main table item that you don't want to delete). If it is in Lua, you are going to have to do some Lua re-coding, which you will have to figure out yourself. If it is in a particular UI XML/Lua system, then you will have to do some UI re-coding (reading whoward69's UI tutorial is a good start).
2. Re-sequence table IDs
When you delete an item, if you do nothing else you will have "gaps" in a table where the IDs are now noncontinuous. Depending on the particular table, noncontinuous IDs will: 1) crash the game outright, 2) cause nearly silent nonsensical errors that are hard to track, or 3) have no effect. Rather than try to figure out which table fits in which category, it is easiest to re-sequence IDs for any table for which you may have introduced noncontinuous IDs (i.e., you deleted something). The full SQL code for this was posted by lemmy101 here. The basic code is the same for all tables except Technologies. For all but technologies, it looks like this (just replace "Buildings" with your table name):
The only difference for Technologies is that it has to be ordered by its GridX position (I think this is needed for proper pipe drawing):
Obviously, the above code has to run after you have deleted/added items. lemmy101 originally wrote one giant SQL file that does this for all tables in the game. I prefer to add the code above only for the specific tables where I have deleted things.
[Edit 8/24/2012] If there is any possibility of additional items being added to the same table later, for example by another mod, then you need to add the following code after the block above (code from Thalassicus; example given is for UnitPromotions). This will ensure that additions to the table start with the next available ID (seq is the "counter" for this and it got screwed up with your deletions/additions/reIDing above).
3. Check/fix references to intermediate tables
In unmodded Civ5, these three lines evaluate to the same thing:
GameInfo.Improvements.IMPROVEMENT_FARM.ID
GameInfoTypes.IMPROVEMENT_FARM
ImprovementTypes.IMPROVEMENT_FARM
However, if you change IDs for the Improvement table, then the last line will not update to reflect your mod's changes and may now give an incorrect value. The 1st and 2nd lines will always correctly reflect your mod's Impvovements table. These "____Types" are a set of "intermediate tables" that you can find in the wiki under "Lua Enums" (although the list is incomplete and not all shown are used by the game). One of these that's used extensively in base game UI is YieldTypes. But I doubt you are deleting yield types so that shouldn't matter. AssignStartingPlots.lua uses these intermediate tables for resources and terrain types. Other than these, I believe that the base Civ5 Lua almost always uses the direct table reference (1st line above) rather than the intermediate table refence (2nd line above). Use your text search program (search "Types.") to find any references to these for tables that you are deleting from, and change them to the 2nd longer "direct table reference" shown above. I've avoided this problem in my own Lua by always writing out the direct table references and never using intermediate tables (well, except YieldTypes...).
Caveat/disclaimer
The one potential problem with this is for scenarios. These add units and other items by ID so things get messy if you reorder these IDs after the scenario loads (see discussion here). Obviously it's a bad idea to load "base game" scenarios in a mod that has deleted base elements. I believe that scenarios created from your mod will not have this problem. But I don't really know since I haven't tried this yet.
Feel free to post questions or other issues that I may have missed.
Deleting things is BAD.
I've seen this statement in various forms from some of the best modders here. The same modders then identify the specific issues related to deleting (e.g., as Spatz does in the post I'm quoting). I'll go one step further here and tell you how to address these issues.
Further, I'm a contrarian so I'll say that (depending on your mod) deleting things is Good. If it's a total conversion mod, do it. If you are removing a whole era, I'd say do it but others will disagree. If you just want to remove one thing (e.g., Giant Death Robot) it is much easier to disable (e.g., make cost = -1) rather than delete.
Note that everything I say below applies to deleting items from the "main" tables -- i.e., anything that has a Type and ID. You can delete rows from "subtables" without worrying about this stuff at all. Also, note that the trend in patches has been to make the game more and more robust to these issues (especially #2 below). But this is probably a curse rather than a cure: an outright crash is much preferable than an insidious and inconsistent error.
1. Find references to deleted items in both XML and Lua (then remove or decide that deletion is not a good idea after all)
Depending on the table, you may have to have some sophistication in Lua programming and/or Civ5 UI to be able to do this. The first step is to identify all occurrences of your deleted item in base game XML or Lua files. Your friend here is a good application for searching text in files. I use Windows Grep (whoward69 has another that he recommends frequently in his posts). There is a more sophisticated DB-ish way to do this but you don't need that. These cross-references are mostly in Gameplay and UI folders but they can be elsewhere (e.g., DLC Babylon Lua references TECH_WRITING). You may want to do these searches first before you decide whether deletion is the right way to go: some things are used a lot in base Lua (e.g., ResourceTypes and TerrainTypes in AssignStartingPlots.lua) or UI (e.g., YieldTypes, but I doubt you are going to want to delete any of those), others never.
Where you find these in XML (non-UI), just delete that row (if it is a "subtable") or the column entry (if it is a main table item that you don't want to delete). If it is in Lua, you are going to have to do some Lua re-coding, which you will have to figure out yourself. If it is in a particular UI XML/Lua system, then you will have to do some UI re-coding (reading whoward69's UI tutorial is a good start).
2. Re-sequence table IDs
When you delete an item, if you do nothing else you will have "gaps" in a table where the IDs are now noncontinuous. Depending on the particular table, noncontinuous IDs will: 1) crash the game outright, 2) cause nearly silent nonsensical errors that are hard to track, or 3) have no effect. Rather than try to figure out which table fits in which category, it is easiest to re-sequence IDs for any table for which you may have introduced noncontinuous IDs (i.e., you deleted something). The full SQL code for this was posted by lemmy101 here. The basic code is the same for all tables except Technologies. For all but technologies, it looks like this (just replace "Buildings" with your table name):
Code:
CREATE TABLE IDRemapper ( id INTEGER PRIMARY KEY AUTOINCREMENT, Type TEXT );
INSERT INTO IDRemapper (Type) SELECT Type FROM Buildings ORDER BY ID;
UPDATE Buildings SET ID = ( SELECT IDRemapper.id-1 FROM IDRemapper WHERE Buildings.Type = IDRemapper.Type);
DROP TABLE IDRemapper;
Code:
CREATE TABLE IDRemapper ( id INTEGER PRIMARY KEY AUTOINCREMENT, Type TEXT );
INSERT INTO IDRemapper (Type) SELECT Type FROM Technologies ORDER BY GridX ASC;
UPDATE Technologies SET ID = ( SELECT IDRemapper.id-1 FROM IDRemapper WHERE Technologies.Type = IDRemapper.Type);
DROP TABLE IDRemapper;
[Edit 8/24/2012] If there is any possibility of additional items being added to the same table later, for example by another mod, then you need to add the following code after the block above (code from Thalassicus; example given is for UnitPromotions). This will ensure that additions to the table start with the next available ID (seq is the "counter" for this and it got screwed up with your deletions/additions/reIDing above).
Code:
UPDATE sqlite_sequence
SET seq = (SELECT COUNT(ID) FROM UnitPromotions)-1
WHERE name = 'UnitPromotions';
3. Check/fix references to intermediate tables
In unmodded Civ5, these three lines evaluate to the same thing:
GameInfo.Improvements.IMPROVEMENT_FARM.ID
GameInfoTypes.IMPROVEMENT_FARM
ImprovementTypes.IMPROVEMENT_FARM
However, if you change IDs for the Improvement table, then the last line will not update to reflect your mod's changes and may now give an incorrect value. The 1st and 2nd lines will always correctly reflect your mod's Impvovements table. These "____Types" are a set of "intermediate tables" that you can find in the wiki under "Lua Enums" (although the list is incomplete and not all shown are used by the game). One of these that's used extensively in base game UI is YieldTypes. But I doubt you are deleting yield types so that shouldn't matter. AssignStartingPlots.lua uses these intermediate tables for resources and terrain types. Other than these, I believe that the base Civ5 Lua almost always uses the direct table reference (1st line above) rather than the intermediate table refence (2nd line above). Use your text search program (search "Types.") to find any references to these for tables that you are deleting from, and change them to the 2nd longer "direct table reference" shown above. I've avoided this problem in my own Lua by always writing out the direct table references and never using intermediate tables (well, except YieldTypes...).
Caveat/disclaimer
The one potential problem with this is for scenarios. These add units and other items by ID so things get messy if you reorder these IDs after the scenario loads (see discussion here). Obviously it's a bad idea to load "base game" scenarios in a mod that has deleted base elements. I believe that scenarios created from your mod will not have this problem. But I don't really know since I haven't tried this yet.
Feel free to post questions or other issues that I may have missed.