XML investigation

[Rwn]

Now that most of the merging is done, I'm looking into the files to make them easier to tweak, starting with Civ4BuildingInfos.xml (the largest one).

First, before discussing them more into details, I'd like to know how useful you'd rate the following things ("useful", "not really useful but not problematic either", "should not be done"):

- Check possible errors on a case-by-case basis
I've stumbled upon probable mistakes; for example the following buildings:
BUILDING_INFO_NET
BUILDING_PARADISE_GARDEN
BUILDING_GLOBE_THEATRE
have two <iPrereqPopulation> (twice 13 for the first two, 13 and 6 for the latter).
I can check every tag, look if there are values that seems off, and report them here.

- Merging similar entries
Probably the result of merging files, I guess that the entries can now be merged now that everything is in the same file. Here's the list of duplicates inside Civ4BuildingInfos.xml:

Spoiler :

BUILDING_ARCHERY_RANGE
BUILDING_BAMBOOWORM
BUILDING_EQ_PLUS1
BUILDING_EQ_PLUS10
BUILDING_EQ_PLUS2
BUILDING_EQ_PLUS3
BUILDING_EQ_PLUS4
BUILDING_EQ_PLUS5
BUILDING_EQ_PLUS6
BUILDING_EQ_PLUS7
BUILDING_EQ_PLUS8
BUILDING_EQ_PLUS9
BUILDING_FARM_SUPPLY
BUILDING_PALACE
BUILDING_PALACE
BUILDING_UN_MISSION

- Put the tags in each entry in the same order
Could be any order as long as it's the same everywhere, or the order defined in the schema, or a specific order (see next item).
The idea here is to make it easier to avoid potential errors such as putting the same tag twice as above or to find the value a specific tag for an entry.

- Change the order of each tag
Extending on the last item, the idea here would be to discuss a specific order for the tags, ideally putting together tags in the same "area of influence" (for example <PrereqTech>, <TechTypes><PrereqTech> and <ObsoleteTech>) and putting the most important/frequent tags early (so you don't have to look for <iCost> beyond much less useful tags such as <bAreaCleanPower> or <bAllowsNukes>).
I don't know whether changing the order in the schema would be necessary or useful or impossible by the way.

- Remove unused tags
Some tags never ever have any value for any building, for example <HotKey>. Could be removed altogether to improve readability.

- Harmonize default tags
Extending a bit on the last item, rarely effectively used tags could be removed to improve readability, for example <bAltDown> never has any other value than 0 (and I guess that's the default value). Or buildings that only require one tech should not have anything in <TechTypes><PrereqTech> (instead of <TechTypes><PrereqTech>NONE</PrereqTech><PrereqTech>NONE</PrereqTech><PrereqTech>NONE</PrereqTech><PrereqTech>NONE</PrereqTech></TechTypes> - cf. BUILDING_LIQUORSTORE for an example).
This could also mean agreeing to put again empty or default tags when that improves readability, for example buildings that do not obsolete currently either have a <ObsoleteTech>NONE<ObsoleteTech> instead of <ObsoleteTech><ObsoleteTech> or nothing at all; everything could be set to <ObsoleteTech>NONE<ObsoleteTech>.

- Splitting the BuildingInfo file into different files
Could be for example Great Wonders, National Wonders, ordinary buildings and "pseudobuildings" (diseases, myths...).

- Sorting the entries inside the BuildingInfo file(s)
Could be alphabetical, or chronological by unlocking tech, or anything else.

- Maybe other things I haven't thought of?
Do not hesitate if you have other ideas of things that could be convenient, I'll tell if I think I can do it in a reasonable amount of time.


A little warning, I don't see any risk with the first two items, but the others require going through the table2xml tool I did, which *should* be bugfree by now (I've done many tests with various files) but I can't guarantee it 100% as an entry may have an odd formatting that I might not have taken into account. So going through it will require some playtest afterwards to confirm that nothing went wrong in the process.


Also, attached is the (current) full table of building properties from Civ4BuildingInfos.xml, in csv format, in case it is of help to anyone

View attachment Full building table svn 8478.7z

 

[Faustmouse]

Just a thought I had some time ago: to reduce loading times and MAFs, we could sort all buildings, units and probably techs by era and then only load everything for all the currently reached eras in a game +1. So lets say if you are medieval and your opps are classical, the game should only load everything up to the last renaissance technology. Computers, Modern Tanks, Space Ports... all of that would be unloaded and safe a lot of memory. And you can still plan ahead as you can see the next era still.

That was just an idea and seemed to hard to achieve as everything needed to be sorted, but it would be one application for your work.

 

[Rwn]

Just a thought I had some time ago: to reduce loading times and MAFs, we could sort all buildings, units and probably techs by era and then only load everything for all the currently reached eras in a game +1. So lets say if you are medieval and your opps are classical, the game should only load everything up to the last renaissance technology. Computers, Modern Tanks, Space Ports... all of that would be unloaded and safe a lot of memory. And you can still plan ahead as you can see the next era still.

That was just an idea and seemed to hard to achieve as everything needed to be sorted, but it would be one application for your work.

I don't have the knowledge to get the game load different files, but at least making separate files for each era sure is feasible and easy (the attached .csv file already has the buildings grouped by era).

Though I wonder how it'd work for the tech tree display or the pedia, wouldn't every building need to be loaded for them to work properly? And would "unloading" early era buildings be problematic as there are probably buildings that last through the whole game?

Edit : now I think about it, another possibility would be to have a different file for each era including all relevant buildings for this era, removing all those that were obsoleted by a tech unlocking that era and including all those unlocked in this era (and the next one?). Meaning there would probably be a lot of buildings duplicated across the files (as long as they are not obsolete), but if only one file is loaded at a time it could be helpful?

 

[alberts8]

I wrote a little tool to batch edit all those xml files. Fully cleaning them without breaking anything is a matter of seconds so don't waste time with ordering or cleaning.

But merging those duplicates is a good idea

 

[Rwn]

I wrote a little tool to batch edit all those xml files. Fully cleaning them without breaking anything is a matter of seconds so don't waste time with ordering or cleaning.

I wasn't considering manually reordering/cleaning each entry, I already have a tool that can do it (cf. the table2xml topic), though I'm interested to know more about yours

 

[Dancing Hoskuld]

I would not reorder the unit file. The order it is in is good for WorldBuilder as it has the units in groups eg all the spearmen together, naval units from earliest to latest. This is a useful to players.

I had no problems with finding stuff as a modder, even before the merge, as I use WinGrep to find what I are looking for.

 

[Sparth]

I would not reorder the unit file. The order it is in is good for WorldBuilder as it has the units in groups eg all the spearmen together, naval units from earliest to latest. This is a useful to players.

I had no problems with finding stuff as a modder, even before the merge, as I use WinGrep to find what I are looking for.

+1 for this. Now its very helpful to find correct units in WorldBuilder. IMO What can be alphabeticaly sorted is:
- Cultural Units
- Heroes
- Animals

What do you think DH?

 

[Sparth]

RWN, if you bored and eager to get new task with XML here is the list with all dds files from C2C.fpk pack file
https://docs.google.com/spreadsheets/d/1pXTnw2g81UVDi6TlMLwBxwMobZVcxRymqLVvGypEq2A/edit#gid=0

You can check XML files and tag every file which is not used in mod. Its help to reduce size of the mod and quantity of files.

 

[Dancing Hoskuld]

+1 for this. Now its very helpful to find correct units in WorldBuilder. IMO What can be alphabeticaly sorted is:
- Cultural Units
- Heroes
- Animals

What do you think DH?

Yes to Cultural Units and Heroes.

As I work on the animals they will become more ordered but for some reason known only to Hydro the class name of the unit does not always match the text name eg UNITCLASS_ASIANBEAR is the Moon Bear. This means the alphabetic order in the file is not the alphabetic order in WorldBuilder.

There will remain 3 animal unit infos files (animals, SubduedAnimals and TamedAnimals) for a while. Since the button for the wild and subdued animal is the same it is good to keep the two sets apart in the WorldBuuilder.

 

[Faustmouse]

I don't have the knowledge to get the game load different files, but at least making separate files for each era sure is feasible and easy (the attached .csv file already has the buildings grouped by era).

Though I wonder how it'd work for the tech tree display or the pedia, wouldn't every building need to be loaded for them to work properly? And would "unloading" early era buildings be problematic as there are probably buildings that last through the whole game?

Edit : now I think about it, another possibility would be to have a different file for each era including all relevant buildings for this era, removing all those that were obsoleted by a tech unlocking that era and including all those unlocked in this era (and the next one?). Meaning there would probably be a lot of buildings duplicated across the files (as long as they are not obsolete), but if only one file is loaded at a time it could be helpful?

It was just an idea. You wouldn't unload earlier eras, so the more you progress, the more memory is used, up until you reach the TH era were everything is loaded, like it is now. This would only help the game before someone reach the TH era. But from what I know, only very few people even get to the modern era, mostly because of turn times.

I also thought of having each era in a seperate folder. I don't know how much effort it would be to check after each game how advanced the tech leader is and then edit the loading MLF to only load the "required" files. Of course, this only works if you don't play several games at the same time, so it should be able to turn off.

The unit structure for the WB looks great though!

 

[Rwn]

RWN, if you bored and eager to get new task with XML here is the list with all dds files from C2C.fpk pack file
https://docs.google.com/spreadsheets/d/1pXTnw2g81UVDi6TlMLwBxwMobZVcxRymqLVvGypEq2A/edit#gid=0

You can check XML files and tag every file which is not used in mod. Its help to reduce size of the mod and quantity of files.

Unfortunately checking whether each one isn't used in any xml file is hell, at least I haven't found an easy way to do it.

What I've done is the opposite, I wrote in your sheet the art name and file path for each one I've been able to find, either in a Button, Nif or Path tag.
Assuming the reverse is true might be a little risky, I'm not completely sure that each one that is noted as #N/A is indeed never used so if you delete all of them, there might be a few that slipped between the cracks. Still I've tried to be very thorough so I don't think there are many, if at all.

I hope that was useful anyway!

 

[Dancing Hoskuld]

I thought it would just take two lists which you then compare. One list you make of all the dds in the mod with their paths. The other from the XML of all dds and paths used.

Mind you there are some dds in the mod that wont show up in the XML because that project was put on hold. EG the merge of ideas from WLBO mod.

 

[Rwn]

I thought it would just take two lists which you then compare. One list you make of all the dds in the mod with their paths. The other from the XML of all dds and paths used.

It's just that is difficult to get a thorough list of all dds used: I don't know whether there are .dds hidden in non-xml files, or behind non-obvious tags, or if I haven't inadvertantly skipped one among the 300 files that contained a .dds... This is why I'd rather say that I've found a location for many of them and that the others are *likely* not used anywhere...

 

[Rwn]

I would not reorder the unit file. The order it is in is good for WorldBuilder as it has the units in groups eg all the spearmen together, naval units from earliest to latest. This is a useful to players.

I'm not with the unit file yet, that's the kind of things that's interesting to know.

It's definitely possible to sort any file not by their <Type> but by their in-game name (I just have to trace the TXT_KEY to the name), I guess it'd be possible to sort your animal files this way (though it'd probably be more efficient to do it once you've mostly finished working on it?).

Apart from that, do you have other comments on the things suggested in the first post? To be clear, I'm not intending to do anything manually (except maybe fixing some mistakes), so it's more a question of avoiding useless or harmful changes.

 

[Dancing Hoskuld]

It's just that is difficult to get a thorough list of all dds used: I don't know whether there are .dds hidden in non-xml files, or behind non-obvious tags, or if I haven't inadvertantly skipped one among the 300 files that contained a .dds... This is why I'd rather say that I've found a location for many of them and that the others are *likely* not used anywhere...

Most of the dds are in the XML. However there are some in the Python and I suspect some in the dll and exe.

Apart from that, do you have other comments on the things suggested in the first post?

I did not comment on the others because I saw no harm in them.

In normal BtS almost all tags need to be present and in the order described in the schema file. In C2C order and existence is not as important, though some tags are still compulsory.

 

[Rwn]

Most of the dds are in the XML. However there are some in the Python and I suspect some in the dll and exe.

So I guess the way to find out is to remove all those noted #N/A and see what's missing in-game

 

[Sparth]

So I guess the way to find out is to remove all those noted #N/A and see what's missing in-game

Good job with these DDSes.
Ill try to remove most of them and we will see what happened

 

[Rwn]

OK so let's start with some things in CIV4BuildingInfos.xml!

<Advisor>

The following entries have a FLAVOR_something in their <Advisor> tag instead of the usual ADVISOR_something :
BUILDING_AESOP_CRIME
BUILDING_OSTRICH_FARM
BUILDING_SAFETY_ACHIEVEMENT_EFFECT
BUILDING_MEDICAL_ACHIEVEMENT_EFFECT
BUILDING_WATERCLOCK
BUILDING_ENVIRONMENTAL_ACHIEVEMENT_EFFECT
BUILDING_TOURISM_ACHIEVEMENT_EFFECT


<PrereqGameOption>

This tag is only used once, for Bee Bombs, which thus requires GAMEOPTION_HEART_OF_WAR. This option is invisible and deactivated by default and only appears (among all xml files) for this building.
Bee Bombs thus do not appear in the Pedia, though they show as a possible building in the Apiculture tech.

I'd suggest removing altogether both the Bee Bombs building and GAMEOPTION_HEART_OF_WAR inside GameOptions.xml.


<FreeBonus>

The following buildings have a non-0 number set as their number of free bonuses, but the free bonus is defined as NONE so it has no effect:
BUILDING_HUNTERS_CAMP
BUILDING_RICHARDSON
BUILDING_PEDRA_FURADA
BUILDING_MOVIE_STUDIO

I suggest putting 0 in the number of free bonuses for consistency's sake.


Also the following buildings have "-1" set as their number of free bonuses:

BUILDING_BROADWAY
BUILDING_HOLLYWOOD
BUILDING_ROCKNROLL
BUILDING_RESEARCH_HOSPITAL
BUILDING_MOVIE_STUDIO

It seems that "-1" is interpreted as "6" in-game (the vanilla entries for Broadway, Hollywood and Rock'n Roll already had this "-1") so I guess it works (and should be let as is), though that's misleading...


<GreatPeopleUnitClass>

The following buildings have a <GreatPeopleUnitClass> defined but their <iGreatPeopleRateChange>
is set to 0, meaning they don't really contribute to great peoples:

BUILDING_NATIONAL_LIBRARY_SERVICE
BUILDING_CORPORATION_BURGERWORLD_RESTAURANT
BUILDING_CORPORATION_MCDOWELLS_RESTAURANT
BUILDING_CORPORATION_MOBBY_STORE
BUILDING_CORPORATION_BULLSEYE_STORE
BUILDING_CORPORATION_EWTRADE_STORE
BUILDING_CORPORATION_MALLWART_STORE
BUILDING_CORPORATION_MOONBEAN_SHOP
BUILDING_CORPORATION_WONKA_SHOP
BUILDING_CORPORATION_ULTSOLDIER_TRAINER
BUILDING_CORPORATION_ADVENTURE_FRANCHISE
BUILDING_CORPORATION_MILEHIGH_FRANCHISE
BUILDING_CORPORATION_SAFARI_GUIDE
BUILDING_WIKIPEDIA

I suggest replacing what's in <GreatPeopleUnitClass> by NONE for them.


<bPower>

This tag means that the building provides power, I guess it was confused with "requires power" for the following buildings:
BUILDING_WIKIPEDIA
BUILDING_SRC (Species recovery center)

I suggest putting 0 in <bPower> for those.



<iHurryCostModifier>

I assume this tag alters the hurry cost for the building. Usually it's something like +50%, +100%, +300% etc. but for BUILDING_GLOBAL_STOCK_EXCHANGE it's set to an odd +1%.

I suggest replacing it by 100.



<iYield>

The following buildings have less than 3 <SeaPlotYieldChanges><iYield> defined (for ). I assume the game considers those not filled as 0, but it would be clearer to actually fill them.
BUILDING_MOAI_STATUES
BUILDING_LIGHTHOUSE
BUILDING_NETHERLANDS_DIKE

Same as above for <RiverPlotYieldChanges>.
BUILDING_IRRIGATION_CANALS
BUILDING_LEVEE
BUILDING_NETHERLANDS_DIKE

Same as above for <YieldChanges>.
BUILDING_ARTESIAN_WELL
BUILDING_CANNERY


<iCommerce>

Same as above for <CommerceChanges>, which should be present 4 times science:, , , ), <ObsoleteSafeCommerceChanges>, <CommerceModifiers>, etc. (I won't list them all).


<BonusHealthChanges><BonusHealthChange><BonusType>

BUILDING_TOMB has 3 <BonusHealthChanges><BonusHealthChange><BonusType> defined (Corn, Rice and Wheat), but the associated <iHealthChange> is 0 for each, making them effectively unused.
The corresponding tags could be removed.


<BonusProductionModifiers><BonusProductionModifier><BonusType>

BUILDING_FRENCH_SALON has NONE defined as <BonusType>, but 100 defined as <iProductonModifier> (tag name is oddly missing a "i" by the way).
The corresponding tags could be removed.


<UnitCombatFreeExperiences><UnitCombatFreeExperience><UnitCombatType>

BUILDING_FLAVIAN has <UnitCombatType> defined with UNITCOMBAT_MELEE but with a <iExperience> of 0.
The corresponding tags could be removed.


<SpecialistYieldChanges><SpecialistYieldChange><SpecialistType>

BUILDING_FIRE_BRIGADE and BUILDING_FIRE_DOCK have a science specialist defined in <SpecialistType>, but always 0 in <YieldChanges><iYield>.
The corresponding tags could be removed.



<Flavors><Flavor><FlavorType>

Many buildings have a flavor defined in <FlavorType>, but with 0 in <iFlavor>.
The corresponding Flavors could be removed.


<iPrereqPopulation>

The following buildings have a <iPrereqPopulation> defined several times:
BUILDING_GLOBE_THEATRE
BUILDING_INFO_NET
BUILDING_PARADISE_GARDEN

I suggest only keeping the last one.


<PrereqOrFeature><PrereqFeature><FeatureType>

BUILDING_STRATOSPHERE has NONE defined as <FeatureType> but 1 as <bPrereqFeature>.
This could be removed.


<AndDependencyTypes><DependencyType>

Some <DependencyType> refer to buildings or features that are already in core:

Spoiler :

RELIGION_SHAMAN
RELIGION_TENGRIISM
BUILDINGCLASS_STRATEGIC_GRAIN_RESERVE
ART_DEF_BUILDING_STRATEGIC_GRAIN_RESERVE
TECH_POTTERY
TRAIT_ORGANIZED
BUILDINGCLASS_GRANARY
UNITCLASS_ENGINEER
FLAVOR_GROWTH
BUILDINGCLASS_ARK
ART_DEF_BUILDING_ARK
ART_DEF_MOVIE_ARK
ERA_CLASSICAL
TECH_PRIESTHOOD
RELIGION_JUDAISM
FLAVOR_RELIGION
RELIGION_MESO
RELIGION_MESO
BUILDINGCLASS_NATIONAL_LIBRARY_SERVICE
ART_DEF_BUILDING_NATIONAL_LIBRARY_SERVICE
TECH_LIBERALISM
BUILDINGCLASS_LIBRARY
UNITCLASS_SCIENTIST
FLAVOR_SCIENCE
TECH_EXTREME_SPORTS
TECH_LONG_RANGE_FORECASTING
BUILDINGCLASS_PALACE
BUILDING_PALACE

All those dependencies could be removed, except those refering to religion (since they are defined in Modules/Custom_Religions which may not be loaded).





Tags that are never used (or only with a default value)

Though they may appear in modules files, they could probably be removed when they appear in the core building file, making entries slightly easier to read...

<PowerBonus>
<bMapCentering>
<bNoHolyCity>
<bDirtyPower>
<bAreaCleanPower>
<iAIWeight>
<iGlobalMaintenanceModifier>
<iAreaMaintenanceModifier>
<iOtherAreaMaintenanceModifier>
<iDistanceMaintenanceModifier>
<iNumCitiesMaintenanceModifier>
<iCoastalDistanceMaintenanceModifier>
<iConnectedCityMaintenanceModifier>
<iRevIdxDistanceModifier>
<CommerceFlexibles>
<HotKey>
<bAltDown>
<bShiftDown>
<bCtrlDown>
<iHotKeyPriority>
<FreeTradeRegionBuilding>
<bBuildOnlyOnPeaks>
<iHappinessPercentPerPopulation>

 

[alberts8]

Good job with these DDSes.
Ill try to remove most of them and we will see what happened

Aren't DDS files also used in NIF or KFM files? I can't remember which of them.

 

[Sparth]

Aren't DDS files also used in NIF or KFM files? I can't remember which of them.

Yep NIF uses them. Ill check every DDS in Photoshop before removing them.
All textures will stay.