Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
-
Civilization 7 has been announced. For more info please check the forum here .
Fun with xml extraction
- Thread starter Rwn
- Start date
Dancing Hoskuld
Deity
Today.
Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
Hmm... that might be slightly to few time to put these in before then...
Reallygood, ty a lot!However, it would be nice to see the actual name of a building as well, since the tags are not always matching. Or are they in this files and i was just to blind to see them?
Here you go. I also added the names for Units and Techs while I was on it.
There are some names missing (which I just replaced by the Type), likely those added recently (I haven't updated my gametext file recently).
Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
mh.... is it just me or are the entries misplaced? like sometimes "path" is in "Description" etc?
That's odd, I don't see anythink problematic when I open them. In which file do you have this issue and with which software do you open the .csv?
What I'd guess is that the software you use somehow removes empty entries.
Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
I only tested the building2 file and used open office calc. On the first building file I didn't have the problem. Maybe it is because the names have spaces in them so counting as multiple entries?
That's probably the reason. I'm not familiar with OpenOffice, but you should specify somewhere that you're opening a .csv file so the software knows what the cell delimiter is.
Anyway, here's attached the same files in tab-separated txt, should be easier to open with OpenOffice (do not use notepad++
). I thought .csv was better, but I'll stay with this format from now on.Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
I would definetly support this. Most buildings are in Hydro's or my module and they are so deep in the building structure that you can't really toggle of a module without causing a lot of XML errors. So they can be merged into core. I think the main reason why this hasn't happened is that neither Hydro nor I have time for this at the moment
I've started playing a bit with this. Instead of doing it manually with Notepad++, I'm using the table2xml macro, which allows to merge entries much faster in any way you want once I know which entry should go where.
As a first test, I merged all BuildingInfo files located in Modules/Hydro/ into one file. The order the files are merged is the one in Modules/Hydro/MLF_CIV4ModularLoadingControls.xml (I don't think there are duplicates, but just in case). It's possible (and easy) to change this order or even to change the order of entries inside a given file (or across several files).
The result is a single file named Hydro_CIV4BuildingInfos.xml (attached) that I've placed in one of the Modules/Hydro subfolders while I removed all other BuildingInfos in Modules/Hydro/. The xml structure is valid, the mod loads correctly and (at first sight) the buildings appear to work correctly.
Also, during the merging process, a few things changed:
- All entries have their tags in the same order. I can change the order if needed.
- All empty tags (such as <whatever/> or <whatever></whatever>, but not <whatever>NONE</whatever> for example, though this is doable if useful) are removed.
Before moving further (first by discussing what could be merged where exactly), I'd like you opinion on:
- whether this is useful
- whether I "step on anyone's toes" by doing that
- when would be the best moment to merge (before v35, after v35, never?)
- if you have any other comment!
Thunderbrd
C2C War Dog
Alberts would probably appreciate some of what's being done in that!
Hydro would need to comment on how he feels about it of course.
I'll be doing this (merging to the core) with my promos at some point probably soon but I don't think it will require what you had to do because it's already all in one document.
Hydro would need to comment on how he feels about it of course.
I'll be doing this (merging to the core) with my promos at some point probably soon but I don't think it will require what you had to do because it's already all in one document.
Dancing Hoskuld
Deity
@Rwn Doing the merge for all Vokarya's wonders (alphabetic order) into one set of files in folder Vokarya/Wonders would be useful. Unless we are going to remove all those that don't do anything first.
Similarly in the Resources folder if we could reduce it does two three folders Culture, Manufactured and TradeGoods would also be useful. You would need a bit of guidance there.
Similarly in the Resources folder if we could reduce it does two three folders Culture, Manufactured and TradeGoods would also be useful. You would need a bit of guidance there.
I'll start with resources, first with BonusInfo files from SVN 7442 (I'll do the ArtDefines_Bonus and Gametext located in the same folders later on if everything goes well).
I assume all of them are located in files matching .*CIV4BonusInfo.xml :
My suggestion :
- Keep the ones in Alt_Timelines inchanged
- Keep the Culture file inchanged (it contains all culture bonus except the alt_timelines ones ; I counted 215).
- Merge everything else (383 entries) into a single file. I can split them into two files (TradeGoods and Manufactured), but I'm not sure how one knows whether a resource is either. I assume the first is for resources "gathered" on the maps while the second are for those generated by buildings - but can't a resource be both? From the look of it, I'd say that TradeGoods are the one located in XML\Terrain\CIV4BonusInfos.xml and Manufactured all the ones in Modules, but I'm not really sure...
Anyway, if you want me to split this file, let me know which bonus should go where (below is the full list; the ones from XML\Terrain\CIV4BonusInfos.xml are from BONUS_BARLEY to the end). Also, I can take this opportunity to sort all entries in whatever order (for example alphabetical), tell me if you want a specific order.
Quick other comments:
- Not all files have the same x-schema defined; they start with either:
<Civ4BonusInfos xmlns="x-schema:C2C_CIV4TerrainSchema.xml">
<Civ4BonusInfos xmlns="x-schema:Tools_CIV4TerrainSchema.xml">
<Civ4BonusInfos xmlns="x-schema:Hydro_CIV4TerrainSchema.xml">
etc. depending on where they are located. The content of those xml schemas do not seem identical.
Is one of them a "master file" which I can use for the merged file? Or will I have to merge them somehow?
- Some tags are nearly unused:
* <Help> is only used for BONUS_PERSONAL_REPLICATORS (it contains TXT_KEY_BONUS_PERSONAL_REPLICATORS_HELP)
* <bForceOverwrite> is only used (with value 1) for BONUS_WOOL, which has only one instance and is located in core (XML\Terrain\CIV4BonusInfos.xml)
* In the following tag that appears for nearly every bonus, the 4th subtag (<iRandApp4>0</iRandApp4>) never contains anything else than 0:
<Rands>
<iRandApp1>25</iRandApp1>
<iRandApp2>25</iRandApp2>
<iRandApp3>0</iRandApp3>
<iRandApp4>0</iRandApp4>
</Rands>
Could those tags be removed?
- More broadly, as I remember you said that reading each tag took time, would it be useful to remove all tags which contain default values? If so, where could I find the default value for each tag?
- I few odd things I've noticed. Probably normal, but just in case:
* BONUS_QUANTONIUM_CRYSTAL has two entries for <TechCityTrade>:
<TechCityTrade>TECH_OPTRONICS</TechCityTrade>
<TechCityTrade>NONE</TechCityTrade>
No other bonus has two <TechCityTrade>. Is the second <TechCityTrade> really relevant?
* The following resources (not including cultures) have NONE as TechReveal:
The following (not including cultures) have NONE as TechCityTrade:
Those have NONE to both TechReveal and TechCityTrade:
* BONUS_KAVA has -30 to iMinLatitude and 10 iMaxLatitude (this is the only one with a negative value)
BONUS_GEMS has 0 to iMinLatitude and 0 iMaxLatitude
I assume all of them are located in files matching .*CIV4BonusInfo.xml :
Spoiler :
My suggestion :
- Keep the ones in Alt_Timelines inchanged
- Keep the Culture file inchanged (it contains all culture bonus except the alt_timelines ones ; I counted 215).
- Merge everything else (383 entries) into a single file. I can split them into two files (TradeGoods and Manufactured), but I'm not sure how one knows whether a resource is either. I assume the first is for resources "gathered" on the maps while the second are for those generated by buildings - but can't a resource be both? From the look of it, I'd say that TradeGoods are the one located in XML\Terrain\CIV4BonusInfos.xml and Manufactured all the ones in Modules, but I'm not really sure...
Anyway, if you want me to split this file, let me know which bonus should go where (below is the full list; the ones from XML\Terrain\CIV4BonusInfos.xml are from BONUS_BARLEY to the end). Also, I can take this opportunity to sort all entries in whatever order (for example alphabetical), tell me if you want a specific order.
Spoiler :
Quick other comments:
- Not all files have the same x-schema defined; they start with either:
<Civ4BonusInfos xmlns="x-schema:C2C_CIV4TerrainSchema.xml">
<Civ4BonusInfos xmlns="x-schema:Tools_CIV4TerrainSchema.xml">
<Civ4BonusInfos xmlns="x-schema:Hydro_CIV4TerrainSchema.xml">
etc. depending on where they are located. The content of those xml schemas do not seem identical.
Is one of them a "master file" which I can use for the merged file? Or will I have to merge them somehow?
- Some tags are nearly unused:
* <Help> is only used for BONUS_PERSONAL_REPLICATORS (it contains TXT_KEY_BONUS_PERSONAL_REPLICATORS_HELP)
* <bForceOverwrite> is only used (with value 1) for BONUS_WOOL, which has only one instance and is located in core (XML\Terrain\CIV4BonusInfos.xml)
* In the following tag that appears for nearly every bonus, the 4th subtag (<iRandApp4>0</iRandApp4>) never contains anything else than 0:
<Rands>
<iRandApp1>25</iRandApp1>
<iRandApp2>25</iRandApp2>
<iRandApp3>0</iRandApp3>
<iRandApp4>0</iRandApp4>
</Rands>
Could those tags be removed?
- More broadly, as I remember you said that reading each tag took time, would it be useful to remove all tags which contain default values? If so, where could I find the default value for each tag?
- I few odd things I've noticed. Probably normal, but just in case:
* BONUS_QUANTONIUM_CRYSTAL has two entries for <TechCityTrade>:
<TechCityTrade>TECH_OPTRONICS</TechCityTrade>
<TechCityTrade>NONE</TechCityTrade>
No other bonus has two <TechCityTrade>. Is the second <TechCityTrade> really relevant?
* The following resources (not including cultures) have NONE as TechReveal:
Spoiler :
The following (not including cultures) have NONE as TechCityTrade:
Spoiler :
Those have NONE to both TechReveal and TechCityTrade:
Spoiler :
* BONUS_KAVA has -30 to iMinLatitude and 10 iMaxLatitude (this is the only one with a negative value)
BONUS_GEMS has 0 to iMinLatitude and 0 iMaxLatitude
Dancing Hoskuld
Deity
We have 4 types of resources in C2C with the possibility of a fifth and sixth down the track. These are
- Map - occur on the map and provide the raw resources for many resource chains.
- Culture - provided by buildings and allow the whole culture development system in C2C
- Manufactured - provided by buildings mainly controls the economic development of cities with chains of buildings and resources.
- TradeGoods - special resources only for trade. They are rare in that only one nation can build them (eg Nok Sculpture) or that their requirements are difficult to complete (eg Champagne). They are not used in building/resource chains.
Map resources will have a non-zero value in iConstAppearance. Only these can appear on the map.
Culture, Manufactured and TradeGoods have zero in iConstAppearance and don't need many of the tags at all. All of these are produced by buildings.
Spoiler :
The ones that belong with optional components will need to remain with those components for quality reasons. It would cause confusion to have a bonus you can't get because you have turned off the module.
Having 4 separate files, one for each, makes it easier for the modders. I was only suggesting you do the merge in the modules area because it would be fairly simple for me to put each group into one folder for you to merge. We could then move those files into core.
having said that we probably need to move the manufactured resources out of the core file and into the manufactured file for consistency.
For modding purposes and as good practice the all files in the module folder should be prefixed by the folder name. It makes finding XML errors easier. Alberts2 is working on a replacement for the way we currently do schemas so they will be replaced eventually. Merging all the files into 4 will make this much easier in the long run.
The master file is the one in the main Assets/XML/Terrain folder.
1) Useful help would be nice but doing it is not likely to happen.
2) If there is only one WOOL then the force overwrite is not needed on it.
3) no idea.
Yes, especially all those extra tags I mention above for manufactured resources.
1) should only have one.
2) TechReveal of NONE is fine for manufactured resources because you have to build a building to get them.
3) TechCityTrade of NONE is probably not a problem for the same reason. Although it may be useful to give them one for players.
4) afaik if the iMinLatitude is zero or positive then two bands of the planet are suitable both north and south of the equator. If one is negative then it represents south of the equator. So 0 and 90 means the whole planet, 0 and 23 means the tropics and -30 to 10 means from 30 degrees south to 10 degrees north but I may be getting that mixed up with how it is done in the animal spawns.
Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
When merging the bonuses, could you sort them in a good way to be displayed in the WB? I think they show the first entry in the XML first etc. So maybe make sure all animal resources are on top, all minerals second, all farming 3rd etc... ?
And I would say, if a resources can be both: Map and manufactured, classify them as map resource. But Hydro or DH can say more about that.
And I would say, if a resources can be both: Map and manufactured, classify them as map resource. But Hydro or DH can say more about that.
Dancing Hoskuld
Deity
Resources are either map or manufactured (culture and trade goods are sub classes of manufactured) they can't be both.
The map resources are now in a logical order for WorldBuilder. That was one of the things 0100010 did. Also why I want them grouped by Map, Culture, Manufactured and Trade Goods. WorldBuilder needs to be changed so that you can't put non-map resources on the map as it can cause CTDs.
The map resources are now in a logical order for WorldBuilder. That was one of the things 0100010 did. Also why I want them grouped by Map, Culture, Manufactured and Trade Goods. WorldBuilder needs to be changed so that you can't put non-map resources on the map as it can cause CTDs.
Faustmouse
Deity
- Joined
- Jan 31, 2012
- Messages
- 3,524
could the Empire state building be changed to only give random trade goods?
Maybe this would make a good wonder as well: A wonder that only gives X random cultures!
Dancing Hoskuld
Deity
Empire State building can be changed as we like. It is only a building.
There are a couple of wonders that give Cultures and one that should. They need to be reworked.
There are a couple of wonders that give Cultures and one that should. They need to be reworked.
OK. Are "Luxury" resources (such as Feathers) considered as Manufactured or TradeGoods?
OK, I'll remove the useless tags in those.
Are there other optional components than the Alt_timelines?
I already have my "master" table and xml with all the resources into it, it's now just a matter of deciding how to reorder it, which tags to remove/correct and how to split it.
OK, I'll not look deeper into xml schema files issues then and assume that this one fits.
1) I'll keep the tag in this entry then as a reminder that the <Help> tag exists then
2) I'll remove this one
3) I'll keep this one unchanged
2 and 3) Even though it has no influence gameplay-wise, wouldn't be useful to replace the NONE with the prereq tech for the required buildings so that it's easier to see in the civopedia which tech "unlocks" the resource?
4) I'll leave the BONUS_KAVA as it is, but I find this behavior a bit surprising: this means you may spawn a resource between -30 and 0 in the southern hemisphere, but you can't spawn it between 0 and 30 (as in this case it would be interpreted as between -30 and 30)?
For BONUS_GEM it is manufactured so the tags will be removed anyway.
Yes, inside each file I can sort them in any order. I just need to know what would be the most practical; if the map resources are sorted in a certain way in WB that you'd like to mirror, where is the WB order defined?
For manufactured goods, I can think of alphabetical or by PrereqTech iGridX.
For cultures, I can think of alphabetical or by native culture (then alphabetical order).
For trade goods, I can think of alphabetical or by category (Wonder-based/luxury) then alphabetical.
Isn't salt both for example? I can't remember exactly how it works (I rarely find salt in my games...), but unless I'm mistaken there's a salt map resource that you can also manufacture in coastal cities. Also, some corporations are also manufacturing map resources.
