Making a "Mostly Gutted Mod": removing most elements for a complete rebuild

Pazyryk · Oct 7, 2010

Xadie,

It's an interesting alternative. One advantage is that it is more "modular," possibly working with other mods (like AI improvements) that might reference specific units. The disadvantages is that there are 100s (or 1000s?) of unused items just sitting in tables. Those are all memory hogs (I think) and slow down table lookups (I think, at least for the many tables that are not indexed).

Xadie · Oct 7, 2010

Well, that is something I'm absolute at ease. Even if it goes in the thousands of unused entries it will only take quite some time when building the SQL database. Once that is done, the actual in-game lookups are so lightning fast, that few hundred extra entries doesn't matter much IMO. As long as the player doesn't change mods the SQL stays unaltered.

But, as mentioned, I'm more concerned about possible hard coded limits on the size of the tables. Much more likely to become a bottle neck.

Szpilman · Oct 7, 2010

Xadie said:
EDIT: ...thinking about it: with that approach we could hit some hardcoded limits, we're not yet aware of, easily. For example I've read that the maximum number of normal working promotions is about 60!!! (http://wiki.2kgames.com/civ5/index.php/Known_Issues) ... oh well >_>

Jeez! That must be fixed asap! 60 is a ridiculous number, I'm planning a mod to have at least 500 and hope it gets fixed pretty soon..

Gedrin · Oct 8, 2010

Pazyryk said:
Gedrin,

That looks like a great way to do it. One question: How to you make sure your sql code runs before any XML files that re-add these elements? Is it just the order of OnModActivated list? That would make sense but I have no real confidence that it works that way.

I'll try some of this over the weekend. It's pretty slow going for me due to lack of error feedback and silent errors (where the game runs but didn't load file due to error -- this is far worse than a crash for troubleshooting). Both problems are plaguing me for XML modding. I'm guessing that SQL modding doesn't get around these particular problems either (or does it?).

As for order the files are run... I bet it is as you supposed "the order of OnModActivated list" but I too have no real evidence of that.

as for immediate feedback.

I tested that the list DELETEs for the units would work by opening the SQLite Manager add-on for firefox...
Then with Database>Connect Database open the .../cache/Civ5DebugDatabase.db
then in the right hand pane you will see a tab "Execute SQL"
first exec the command: PRAGMA foreign_keys=true;
This is because foreign_keys are not enforced via the tool by default [dunno why].

Then run all the DELETEs and if you have caught all the FKs it runs without an error.
If you missed any you know immediately where and why from the popup.

Pazyryk · Oct 8, 2010

Thanks Gedrin! It's actually proceeding quite well now with your help. I've gotten through units, promos, builds, civs, traits, leaders, policies, improvements (testing after each step, it's still a pain). For each main table (at the top of each core XML file), most FKs pointing to it are from the subtables below. I've found that I can delete contents of these subtables entirely without worrying about subsequent FKs pointing to the subtables (so far) -- which saves a lot of time (I only have to worry about FKs for 20 tables rather than 200 tables). For now I'm putting back each of the specific game elements that are called on from Lua code (a couple units, techs, barb civ; all resources) but without any FKs. So all of the little pieces left after I'm done should be safely and easily removable. Should have it up and running soon.

Pazyryk · Oct 9, 2010

Sadly, I've come to the conclusion that this is impossible. It doesn't matter if you identify and deal with all of the foreign keys and Lua references. Some tables like policies, technologies, and buildings appear to be hardwired in. For these, you can't even delete a single item and replace it with an identical item copied from core xml file (try it, you'll see). Other things seem to be deletable and replaceable in whole but this causes weird, seemingly unrelated glitches. For example, if you delete units the game seems to work perfectly except you loose the civ-specific 2D art at game start (don't ask me why, I don't see any relationship). So, I see only two options for compete rebuild mods:

1. Disable (don't delete) units, buildings, techs. Add your own elements and cross your fingers that you don't hit some hard-coded limit on number of table items, or a nasty slowdown because your mod has to accommodate 2x units, buildings, techs compared to vanilla civ5.

2. Use vanilla units, buildings, techs but rename (via txt key) and modify them as needed. This is a very ugly solution because you have to remember that TECH_RAILROADS really means something else in your mod. But it avoids having to double the number of elements in the game.

Hope this can be fixed when we get the dll.

Hamil · Oct 9, 2010

Dale said:
OH I REMEMBER NOW!!!!!

Thou shalt not delete iron!

OMG i had such a laugh at this. My girlfriend actually came running from the other room to see if i have gone crazy...
I immediately saw you DALE as Moses coming down a mountain caring a big stone table and shouting "Thou shalt not delete iron!".
Sorry, but i just HAD to share this. :lol:

Szpilman · Oct 9, 2010

Pazyryk said:
Sadly, I've come to the conclusion that this is impossible. It doesn't matter if you identify and deal with all of the foreign keys and Lua references. Some tables like policies, technologies, and buildings appear to be hardwired in. For these, you can't even delete a single item and replace it with an identical item copied from core xml file (try it, you'll see). Other things seem to be deletable and replaceable in whole but this causes weird, seemingly unrelated glitches. For example, if you delete units the game seems to work perfectly except you loose the civ-specific 2D art at game start (don't ask me why, I don't see any relationship). So, I see only two options for compete rebuild mods:

1. Disable (don't delete) units, buildings, techs. Add your own elements and cross your fingers that you don't hit some hard-coded limit on number of table items, or a nasty slowdown because your mod has to accommodate 2x units, buildings, techs compared to vanilla civ5.

2. Use vanilla units, buildings, techs but rename (via txt key) and modify them as needed. This is a very ugly solution because you have to remember that TECH_RAILROADS really means something else in your mod. But it avoids having to double the number of elements in the game.

Hope this can be fixed when we get the dll.

I'm still hoping you'll put the mod up anyway, as gutted as is possible at this time

Paasky · Oct 10, 2010

Me too, could really use it

xxhe · Oct 23, 2010

Any progress on this topic?

Or we have to wait till some day that 2K fixes this problem?

Putmalk · Oct 23, 2010

How come when I deleted units I had no problem with the 2D art (or does that occur when you only have no units to choose from)?

Ahwaric · Oct 27, 2010

xxhe said:
Any progress on this topic?
Or we have to wait till some day that 2K fixes this problem?

I guess we still have to wait. For firaxis to fix it, or at least to release dll to let us d it.
If not, there is not much future for Civ5 mods. Just how much civs can you add to your game and still enjoy it?
This problem is #1 on the know issues list in Wiki: http://wiki.2kgames.com/civ5/index.php/Known_Issues

I have exactly the same problem with Orbis 2. For now, I decided to simply edit base game files. When modding will be possible, I will just transform my results into a mod.
I have hit a few more problems though, i.e. only one possible upgrade for a unit, so no altarnate upgrade paths for now. Frustrating

Pazyryk · Oct 27, 2010

I made some progress on this, although it's been slow. I worked on this for a while, then took a couple weeks off, now back at it. I thought I'd post an update based on where I am now (I'll make a tutorial eventually, but I feel like I'm only halfway there at this point). Please feel free to take and use anything you want.

Lesson 1: With a few possible exceptions, disable rather than delete the main table contents (by "main" table I mean the first in each xml file; the subtables below can be safely emptied out). Some things just can't be deleted at all (techs, buildings) without instant game crash (even if you replace with exact duplicates). Other things appear to be deletable (units) but then weird and seemingly unconnected bugs crop up later. The only exceptions I've found so far are civilizations and leaders, which seem to be deletable without bad effects (but I'm not 100% sure). Most things can be disabled in some way. Techs, units and buildings can all be removed from the pedia by making tech (or prereq tech for units or buildings) have Era = NULL [edit: this might be causing me some problems, try setting these to Future Era for now, which at least puts them out of the way in the pedia]. Techs can be removed from tech window by setting GridX=-1.

Lesson 2: Learn SQL. You can do a lot with it that just can't be done with xml (which is converted to sql anyway but with a lot of limitations). It's also a lot easier to write without error and to read.

So in my total conversion mod I have one file called DECONSTRUCTION.sql which disables/deletes and then I have separate XML files to update/add items (and the DECONSTRUCTION.sql comes before the xml adds/updates in the actions list). Read notes in sql file below for more details ("--" is used for comment lines in SQL or to disable a line for troubleshooting). This is "working" now. That is, if you use this and re-add a couple civs, units, techs, and so on, it loads and you can start a game. (You can see I'm not finished yet... I'm taking it one part at a time.)

DECONSTRUCTION.sql

Code:

--Notes:
--Deleting Techs, Policies and Buildings appears to be impossilbe (even when replaced by exact duplicates)
--Deleting Units caused an (apparently) unrelated glitch where civ start 2D art was missing
--Deleting Civs and Leaders appears to be harmless (???????)

-------------------------Technologies-------------------------
UPDATE Technologies SET Era='ERA_ANCIENT';
--Disable all but these (Era='NULL' removes tech and tech-associated units & buildings from pedia):
UPDATE Technologies SET Disable='true', GridX='-1', Era='ERA_FUTURE' WHERE Type NOT IN
('TECH_SAILING','TECH_OPTICS','TECH_COMPASS','TECH_ASTRONOMY',
'TECH_AGRICULTURE','TECH_CALENDAR','TECH_ANIMAL_HUSBANDRY','TECH_ARCHERY',
'TECH_WRITING','TECH_MATHEMATICS','TECH_EDUCATION','TECH_PHYSICS','TECH_MACHINERY',
'TECH_MINING', 'TECH_BRONZE_WORKING','TECH_IRON_WORKING','TECH_METAL_CASTING',
'TECH_MASONRY','TECH_CONSTRUCTION','TECH_ENGINEERING','TECH_FUTURE_TECH');
--Unchanged: Technology_Flavors
--Delete all other tech sub-table contents:
DELETE FROM Technology_DomainExtraMoves;
DELETE FROM Technology_ORPrereqTechs;
DELETE FROM Technology_PrereqTechs;

-------------------------Resources-------------------------
--Note: need to deal with this later in Lua. For now, just make modern ones un-revealable.
--NULL a few fields (must be reset in xml for re-used resources):
UPDATE Resources SET TechCityTrade='NULL';
UPDATE Resources SET TechReveal='NULL';
--Disable following resources by assigning TechReveal to unresearchable tech:
UPDATE Resources SET TechReveal='TECH_NANOTECHNOLOGY',TechCityTrade='TECH_NANOTECHNOLOGY' WHERE Type IN
('RESOURCE_COAL','RESOURCE_OIL','RESOURCE_ALUMINUM','RESOURCE_URANIUM');

------------------------- Units -------------------------

--NULL a few fields (must be reset in xml for re-used units):
UPDATE Units SET ObsoleteTech='NULL';
UPDATE Units SET GoodyHutUpgradeUnitClass='NULL';
--Disable all Units but these:
UPDATE Units SET PrereqTech='TECH_ROBOTICS', Cost='-1' WHERE Type NOT IN
('UNIT_SETTLER','UNIT_WORKER','UNIT_WORKBOAT','UNIT_ARTIST','UNIT_SCIENTIST','UNIT_MERCHANT',
'UNIT_ENGINEER','UNIT_GREAT_GENERAL','UNIT_TRIREME','UNIT_CANNON','UNIT_CROSSBOWMAN',
'UNIT_CATAPULT','UNIT_TREBUCHET','UNIT_KNIGHT','UNIT_HORSEMAN','UNIT_ARCHER','UNIT_SCOUT','UNIT_WARRIOR',
'UNIT_BARBARIAN_WARRIOR','UNIT_BARBARIAN_ARCHER','UNIT_GALLEY','UNIT_CARAVEL');
--Unchanged: Unit_Builds, Unit_AITypes, Unit_Flavors, Unit_GreatPersons, Unit_UniqueNames
--Below sub-tables all emptied; need to rebuild any entries for re-used and new units
DELETE FROM Unit_Buildings;
DELETE FROM Unit_BuildingClassRequireds;
DELETE FROM Unit_ProductionModifierBuildings;
DELETE FROM Unit_ClassUpgrades;
DELETE FROM Unit_FreePromotions;
DELETE FROM Unit_ResourceQuantityRequirements;

------------------------- Buildings & Projects -------------------------

UPDATE Buildings SET PrereqTech='TECH_NANOTECHNOLOGY', Cost='-1' WHERE Type NOT IN
('BUILDING_COURTHOUSE','BUILDING_STABLE','BUILDING_FORGE','BUILDING_MONASTERY',					--Special
'BUILDING_LIGHTHOUSE','BUILDING_HARBOR','BUILDING_COLOSSEUM','BUILDING_THEATRE',
'BUILDING_MONUMENT','BUILDING_BARRACKS','BUILDING_ARMORY','BUILDING_MILITARY_ACADEMY',
'BUILDING_WALLS','BUILDING_CASTLE','BUILDING_GRANARY','BUILDING_MARKET','BUILDING_BANK',
'BUILDING_LIBRARY','BUILDING_UNIVERSITY','BUILDING_LABORATORY',									--Normal
'BUILDING_PALACE','BUILDING_HEROIC_EPIC','BUILDING_NATIONAL_EPIC','BUILDING_IRONWORKS',			--National Wonders
'BUILDING_GREAT_LIGHTHOUSE','BUILDING_GREAT_LIBRARY','BUILDING_PYRAMID','BUILDING_COLOSSUS',
'BUILDING_ORACLE','BUILDING_HANGING_GARDEN','BUILDING_GREAT_WALL','BUILDING_PORCELAIN_TOWER');	--Wonders
--UPDATE Projects SET PrereqTech='TECH_NANOTECHNOLOGY', Cost='-1';

------------------------- Civilizations -------------------------
--Delete everything except barbs and minor and rebuild
--Update: Deleting seemed OK, but not sure now. Use disable strategy instead.
DELETE FROM Civilization_CityNames;
DELETE FROM Civilization_DisableTechs;
DELETE FROM Civilization_FreeBuildingClasses WHERE CivilizationType NOT IN
('CIVILIZATION_BARBARIAN','CIVILIZATION_MINOR');
DELETE FROM Civilization_FreeTechs;
DELETE FROM Civilization_FreeUnits;
DELETE FROM Civilization_Leaders WHERE CivilizationType NOT IN
('CIVILIZATION_BARBARIAN','CIVILIZATION_MINOR');
DELETE FROM Civilization_UnitClassOverrides WHERE CivilizationType NOT IN
('CIVILIZATION_BARBARIAN','CIVILIZATION_MINOR');
DELETE FROM Civilization_UnitClassOverrides WHERE UnitClassType IN
('UNITCLASS_SPEARMAN','UNITCLASS_SWORDSMAN');
DELETE FROM Civilization_Start_Along_Ocean;
DELETE FROM Civilization_Start_Along_River;
DELETE FROM Civilization_Start_Region_Priority;
DELETE FROM Civilization_Start_Region_Avoid;
--DELETE FROM Civilizations WHERE Type NOT IN ('CIVILIZATION_BARBARIAN','CIVILIZATION_MINOR');
UPDATE Civilizations SET Playable="false", AIPlayable="false";

Here's a snippet from my new XXXTechnologies.xml file (no, it's not a porn mod; I'm just keeping the name secret for now). In the snippet below you can see an added tech (fishing) and a revised tech (sailing). (Note that Technology_PrereqTechs was completely emptied out in the .sql file above. So there is no need for any "delete" command here and any "old" prereqs would have to be re-added.)

Code:

	<!-- These all are kept from base:
	TECH_SAILING,TECH_OPTICS,TECH_COMPASS,TECH_ASTRONOMY,
	TECH_AGRICULTURE,TECH_CALENDAR,TECH_ANIMAL_HUSBANDRY,TECH_ARCHERY,
	TECH_WRITING,TECH_MATHEMATICS,TECH_EDUCATION,TECH_PHYSICS,TECH_MACHINERY,
	TECH_MINING, TECH_BRONZE_WORKING,TECH_IRON_WORKING,TECH_METAL_CASTING,
	TECH_MASONRY,TECH_CONSTRUCTION,TECH_ENGINEERING,TECH_FUTURE_TECH-->
	<Technologies>
		<Row>
			<Type>TECH_FISHING</Type>
			<Cost>100</Cost>
			<Description>TXT_KEY_TECH_FISHING_TITLE</Description>
			<Civilopedia>TXT_KEY_TECH_FISHING_DESC</Civilopedia>
			<Help>TXT_KEY_TECH_FISHING_HELP</Help>
			<Era>ERA_ANCIENT</Era>
			<Trade>true</Trade>
			<GoodyTech>true</GoodyTech>
			<GridX>0</GridX>
			<GridY>0</GridY>
			<Quote>TXT_KEY_TECH_SAILING_QUOTE</Quote>
			<PortraitIndex>13</PortraitIndex>
			<IconAtlas>TECH_ATLAS_1</IconAtlas>
			<AudioIntro>AS2D_TECH_SAILING</AudioIntro>
			<AudioIntroHeader>AS2D_HEADING_TECH_SAILING</AudioIntroHeader>
		</Row>
		<Update>
			<Where Type="TECH_SAILING" />
			<Set Cost="300" GridX="1" GridY="0"/>
		</Update>
...
	</Technologies>

	<Technology_PrereqTechs>
		<Row>
			<TechType>TECH_SAILING</TechType>
			<PrereqTech>TECH_FISHING</PrereqTech>
		</Row>
...
	<Technology_PrereqTechs>

primordial stew · Oct 28, 2010

Here is my current one, it's all XML. YMMV, but this gets rid of most of what I want to. Don't know if the game will bomb, but WB can load the map, and my civs, units, and CS are there now. Next up is deleting techs, and then lots and lots of adding.. or maybe just write a conversion script to convert the civ4 xml into basic civ5 xml.

Added deleting buildings. ERROR: gc.getNumBuildingInfos <= 0

Techs are in too. The game CTDs on successful compilation

Probably need to add some things back in now!

Spoiler :

joehbright · Oct 28, 2010

How do you get rid of the DLC civs individually? I can't find any XML reference to Mongolia, except in the DLC textinfo file.

xxhe · Oct 28, 2010

joehbright said:
How do you get rid of the DLC civs individually? I can't find any XML reference to Mongolia, except in the DLC textinfo file.

The same process as what you did on the initial civs.

The only difference is that the DLC civs use other XML file at other places (DLC folder), which doesn't influence the modding.

Seven05 · Oct 29, 2010

Pazyryk said:
I made some progress on this, although it's been slow. I worked on this for a while, then took a couple weeks off, now back at it. I thought I'd post an update based on where I am now (I'll make a tutorial eventually, but I feel like I'm only halfway there at this point). Please feel free to take and use anything you want.

I think you're vastly over-complicating things for yourself, but kudos for plugging away as long as you have

The first thing to remember is that you DO NOT need to either delete or disable a ton of things in the game. The only reason for doing so is to keep the interface (pedia mainly) clean but there are ways around that.

Anyway consider this basic element of the game: Resource, Building or Unit X requires Technology Y. Therefore, if Technology Y is disabled, or more precisely if Tech Y is unobtainable because either it or all of its prerequisites have been disabled or are unobtainable themsleves for the same reason then Technology Y is essentially disabled and any unit, building, wonder or resource that requires Technology Y is aslo effectively disabled.

So if you disable all technologies that have no prereq(s) and make sure that all unwanted buildings, units and resource are dependant on a now unobtainable technology you have your 'blank' game. Resources will require the extra step of making sure they aren't placed, even if they'll never be revealed, as that could complicate placing your own resources.

Since you no longer have to work by modifying existing XML files you don't have to worry as much about starting with a blank slate. There are a few limits for table sizes but you're going to run into those regardless of how you decide to do it. Data residing in those tables isn't going to have a significant impact on the game because 1) they've already been loaded before your mod was so there is no savings in load times, and 2) SQL databases are substantially quicker than the simplified "read file and stick it in memory" data storage methods of most games so as long as you aren't polling the entire database every second and performing some complex calculation on it the presence of even thousands of unused entries will be irrelevant.

TW_Honorius · Oct 29, 2010

How about you truncate the table, its faster and more efficent than deleting the table.

primordial stew · Oct 31, 2010

Seven05 said:
The first thing to remember is that you DO NOT need to either delete or disable a ton of things in the game. The only reason for doing so is to keep the interface (pedia mainly) clean but there are ways around that.

Purging all the useless stuff cleans up the lists in WB too. No need to scroll through all those irrelevant units/buildings/techs/policies. Not a biggie, but is nice.

Ahwaric · Nov 1, 2010

Thanks guys for all the hard work you put into researching this topic. I hope someone from firaxis reads this and we will get a fix soon.
For now, it seems as it was not designed to edit whole tech tree

Seven05 said:
I think you're vastly over-complicating things for yourself, but kudos for plugging away as long as you have

The first thing to remember is that you DO NOT need to either delete or disable a ton of things in the game. The only reason for doing so is to keep the interface (pedia mainly) clean but there are ways around that.

I disagree. The fact is, you can't delete most of the things in game, and even disabling is not easy.

But another fact is, that you need to delete unused stuff. True, you might get it working without deleting. But you never know when "disabled" item starts to interfere with added stuff. From my experience, it is always better to start clear. Using items that are simply renamed by the TXT_KEY leads to a mess and makes modding harder - same as keeping all base game items and trying to modify them.

And now some of my own research. I have been trying to put my planned mod together, but so far only by modifying the base game files. This works, though I still have some bugs, like black marshes (I mean black holes

, and accursed UnitGameplay2DScripts (can't remove or even find this code...). I plan to try to turn it into a mod now that I have most of the things working. I have reworked policies today, and some problems occured.
Namely, be aware that the display is coded in SocialPolicyPopup.xml and SocialPolicyPopup.lua. I mean, you need to edit references to every single policy branch to make it display. If a policy branch gets renamed and you do not edit these files accordingly, the policies screen is gone. I think it will make problems even if you simply want to add/modify the policies. Need to add code in lua creees as well.

Also, the order of policy branches DOES matter. The prerequired era is not linked (in SocialPolicyPopup.xml) to the branch name, but rather to its ID number. So, make sure to edit that as well, or use the default order:
upper row IDs: 0,1,2,3,4
lower row IDs: 8,9,7,6,5
Do not ask me why :crazyeye:

Someone might already found this, but I decided to post it anyway, as I couldn't find any mention of it.

Making a "Mostly Gutted Mod": removing most elements for a complete rebuild

Deity

Chieftain

Void Co. Chairman

Chieftain

Deity

Deity

Chieftain

Void Co. Chairman

Good News Everyone!

Prince

Deity

Shrubbery-hugger

Deity

Emperor

Chieftain

Prince

Warmonger

Warlord

Emperor

Shrubbery-hugger

Similar threads