C2C Database Spreadsheet

Hello, folks.

As some of you may know, I'm in the process of converting all the important xml tags of the mod into a spreadsheet format.

This means every unit, every building, and more, will hopefully have all of its XML-given characteristics (cost, movement, whether or not it can see a submarine...everything ) in a single, easy-to-read file.

My hope is that by making this categorization I'm able to help the C2C project without actually having modding skills

OK, so I created this thread so I could discuss the process/ask for help/ask for ideas/get suggestions/etc. without derailing other threads.

Will be updating this as often as I can.

jm

@Thunderbrd

You weren't kidding, trying to export all the tags at once wil not work

However, I don't think it has to do with the application not being able to handle more than a certain # of tags per export. I think some of the tags are messing up the process.

For example, I went through UnitInfos to start with. I managed to export every single tag in there in one file, except the following:

bOnlyDefensive, iAirRange, iAirUnitCap, iDropRange, TerrainImpassables, TerrainPassableTechs,FeaturePassableTechs.

Every time we include any of these tags in the list of tags to export, the application says that it's "not be able to process the Civ4UnitInfos.xml file".

Don't know yet if it's the application's fault, the .xml fault or something else.

I was recently thinking about doing basically the same but with a website to display and with the information being automatically written into a (mysql) database from the XML files periodically (daily or so). But unfortunately I won't have the time for such a project but it sounds really exciting.

This sounds like a really good thing. Do you plan on doing this all manually or are you going to use tools to read the raw information from the XMLs and then add formatting, filters and other fancy Excel stuff to make the sheets usable?

Samin said:

I was recently thinking about doing basically the same but with a website to display the information and with the information being automatically written into a (mysql) database from the XML files periodically (daily or so). But unfortunately I won't have the time for such a project.

Click to expand...

Yes, one of the reasons I'm doing this is so that the actual modding folks don't need to spend their time with spreadsheets when they could be modding

Samin said:

are you going to use tools to read the raw information from the XMLs and then add formatting, filters and other fancy Excel stuff to make the sheets usable?

Click to expand...

This is exactly what I'm doing

Alright, folks, here's where I'm currently at.

https://docs.google.com/spreadsheet/ccc?key=0AhdX1nzgOxjhdFl1S2ItaV9xZE1WdGNqSjEyaXpBUkE#gid=0

This is C2C v24's CIV4UnitInfos.xml in spreadsheet form.

Note:

- This hasn't been formatted at all. Take this just as proof of concept that the xml tags can be converted into spreadsheet format fairly easily thanks to koshling's application located at http://forums.civfanatics.com/showpost.php?p=10592607&postcount=2599.

- The final version will be much more readable, with filters, etc. Also, I will be doing my best to include definitions of what each tag does, and the proper syntax to use it (lots of 1's and 0's )

- There are 8 tags missing in the spreadsheet Unit Mesh Groups I decided not do to, at least for now, but should be fairly easy if it's deemed necessary. Here are the 7 tags that I can't get koshling's app to convert to spreadsheet format:

iAirRange, iAirUnitCap, iDropRange, TerrainImpassables (complexmapping), TerrainPassableTechs (complexmapping), FeaturePassableTechs (complexmapping), and bOnlyDefensive. Whenever I include those in the list of tags to convert, the app says " CIV4UnitInfos.xml could not be processed". Will be looking more into that, and accepting any help and suggestions, of course

- I got ComplexMapping working Pretty proud of that KillOutcomes column, lol. I only need to figure out how to separate the food, hammer and coin yields (from killed animals).

- There's something weird with Apache Cavalry. In the .xml file, it looks fine, but in the spreadsheet, most data is in the wrong column.

For now, I think that'll be it, I need to get some sleep

EDIT - I'll of course be doing other .xml files. Buildings and promotions for sure, and then anything else the modders would like to see in there.

EDIT #2 - Some stuff required ComplexMappings and I only did SimpleMappings; this means that some entries with multiple values will only show 1 value...I'll fix that soon

Excellent start jmmendesp! We'll have to work on getting these files into a subfile on the svn so they can be regularly updated as well... This will help so much with planning and balancing that it should be well worth all the effort, which I know is quite a bit to handle to even get setup.

Update:

Worked some more on the UnitInfos.xml conversion. To my knowledge, now every tag that needed ComplexMapping has ComplexMapping. What this means for you folks is that stuff with different values, for example a unit that can be upgraded into two different units, will show all those values, in a separate, readable way.

I realize now that some tags are not present in every unit. This is most likely the case of tags added by mods. The problem with this is, the only way I have of knowing how many tags C2C has (in the case of UnitInfos.xml), is click on a random unit, and look through all the tags that show up.

Now I realize this is dangerous and likely to miss tags that are exclusive to some units. I only found and included the PropertyManipulators tag because I happened to click on a CityGuard This is far from optimal and must be worked around.

ApacheCavalry's conversion to spreadsheet still messing up, despite its apparently perfect .xml entry.

Still some tags that won't convert: bOnlyDefensive, iAirRange, iAirUnitCap, iDropRange, iNukeRange, TerrainImpassables, TerrainPassableTechs and FeaturePassableTechs (on top of this add the tags that I haven't found yet because they're not present in every unit.)

Thunderbrd said:

Excellent start jmmendesp! We'll have to work on getting these files into a subfile on the svn so they can be regularly updated as well... This will help so much with planning and balancing that it should be well worth all the effort, which I know is quite a bit to handle to even get setup.

Click to expand...

Turns out the exporter can handle as a big a number of tags at once as you can throw at it. Problem is, and what I assume was frustrating you, is that some tags (8 on the unitinfos.xml) just won't export for some reason, so as long as one of them is in the list, the .xml won't be processed. Anyway, I'll get them working somehow

About updating, yes, I'll try my best to create these files in a way that's easy to update and to build upon.

By the way, is 678 a reasonable number for the total amount of units showing up in UnitInfos.xml?

I realize now that some tags are not present in every unit. This is most likely the case of tags added by mods. The problem with this is, the only way I have of knowing how many tags C2C has (in the case of UnitInfos.xml), is click on a random unit, and look through all the tags that show up.

Now I realize this is dangerous and likely to miss tags that are exclusive to some units. I only found and included the PropertyManipulators tag because I happened to click on a CityGuard This is far from optimal and must be worked around.

Click to expand...

This is where getting to know the schema file would be helpful. Most tags are not required (defined as such by the schema, the file that filters the reading of the xml by the dll.) But all are listed in the schema nevertheless. The schema is a little tough to figure out how to read at first. I found most of what I know about it in the Idiots guide to the dll which I wouldn't reccommend trying to read all of but rather just how one would go about adding a tag and taking note of the portion where he's discussing the schemas.

Koshling may be able to assist you with some of the little buggy elements you've found there. I don't know if he's taken note of this thread yet. He's been busy making groundbreaking, mindblowing improvements in our coding

By the way, is 678 a reasonable number for the total amount of units showing up in UnitInfos.xml?

Click to expand...

I think we have more but I don't really know. That's probably a fairly good estimation for the CORE unitinfos file. There are still a lot of units and edits to unit infos in the Modules files.

Thunderbrd said:

This is where getting to know the schema file would be helpful. Most tags are not required (defined as such by the schema, the file that filters the reading of the xml by the dll.) But all are listed in the schema nevertheless. The schema is a little tough to figure out how to read at first. I found most of what I know about it in the Idiots guide to the dll which I wouldn't reccommend trying to read all of but rather just how one would go about adding a tag and taking note of the portion where he's discussing the schemas.

Click to expand...

Aha! Interesting. Thanks, will certainly be looking into that.

Thunderbrd said:

Koshling may be able to assist you with some of the little buggy elements you've found there. I don't know if he's taken note of this thread yet. He's been busy making groundbreaking, mindblowing improvements in our coding

Click to expand...

Yes he has

Thunderbrd said:

I think we have more but I don't really know. That's probably a fairly good estimation for the CORE unitinfos file. There are still a lot of units and edits to unit infos in the Modules files.

Click to expand...

Ah yes, for the moment I'm only converting the core file.

I have a question I think it kinda fits in here. I can't seem to find the majority of text information (TXT_KEY_*). I looked in all the *GameText.xml files in /Assets/XML/Text and /Assets/Modules/* (all sub folders) but especially the pedia entries are missing for most items. Where are those located?

Probably in the core BtS text files, or perhaps even deeper in the Civ4 text files. It will look for them in the mod first, then BtS, then the Civ4 files until it finds the correct reference.

Thunderbrd said:

This is where getting to know the schema file would be helpful. Most tags are not required (defined as such by the schema, the file that filters the reading of the xml by the dll.) But all are listed in the schema nevertheless.

Click to expand...

Alright. I've been using C2C_CIV4UnitSchema.xml to look for tags that I may have missed in the UnitInfos.xml.

Here's the thing: there are many tags that appear in the schema, but not in UnitInfos.xml

Should I:

a) Assume for the time being that those tags are in the schema because they show up in a different .xml (like PromotionInfos.xml, or something), and therefore ignore those tags while working on UnitInfos.xml until I get to converting the .xml they show up in.

b) Assume they are tags that are currently not used by UnitInfos.xml, but that may be used by it in the future, and therefore:

b, i) Put them in the spreadsheet, for the time when they start being used, and to tell everyone reading the spreadsheet that they exist.

b, ii) Not put them in the spreadsheet until they start being used.

c) Do something else?

EDIT - As an example, take the MOD Commanders tags. They are three. iCommandRange and iControlPoints show up in UnitInfos.xml, so I added those to the spreadsheet. bOnslaught, however, does not.

jmmendesp said:

Alright. I've been using C2C_CIV4UnitSchema.xml to look for tags that I may have missed in the UnitInfos.xml.

Here's the thing: there are many tags that appear in the schema, but not in UnitInfos.xml

Should I:

a) Assume for the time being that those tags are in the schema because they show up in a different .xml (like PromotionInfos.xml, or something), and therefore ignore those tags while working on UnitInfos.xml until I get to converting the .xml they show up in.

b) Assume they are tags that are currently not used by UnitInfos.xml, but that may be used by it in the future, and therefore:

b, i) Put them in the spreadsheet, for the time when they start being used, and to tell everyone reading the spreadsheet that they exist.

b, ii) Not put them in the spreadsheet until they start being used.

c) Do something else?

EDIT - As an example, take the MOD Commanders tags. They are three. iCommandRange and iControlPoints show up in UnitInfos.xml, so I added those to the spreadsheet. bOnslaught, however, does not.

Click to expand...

The unit and promotion schemas are in the same file. To tell if a tag belongs to promoions or to units (or to both) you need to look at where the element defiend is used. If it's in the PromtionInfo element definition (in the schema) then it's used by promotions, if in the untiInfo by units. bOnslaught is a promotion tag, though it doesn't appear to do anything so I guess it was a WIP by whoever wrote the GC mod. It's text says 'Continues attack while at full strength', but the code does not implement this. Initiative IV (GC promotion) has this tag (nothing else does), but as I say - it doesn't appear to be implemented, which means the pedia text for that promtion is basically lying!

Koshling said:

The unit and promotion schemas are in the same file. To tell if a tag belongs to promoions or to units (or to both) you need to look at where the element defiend is used. If it's in the PromtionInfo element definition (in the schema) then it's used by promotions, if in the untiInfo by units. bOnslaught is a promotion tag, though it doesn't appear to do anything so I guess it was a WIP by whoever wrote the GC mod. It's text says 'Continues attack while at full strength', but the code does not implement this. Initiative IV (GC promotion) has this tag (nothing else does), but as I say - it doesn't appear to be implemented, which means the pedia text for that promtion is basically lying!

Click to expand...

Thanks. This makes to gravitate towards option a).

On another topic, do you have any idea why the Civ4XMLScanner you developed will not be able to process an xml file if there are certain tags included in the scan template?

For example, it won't process UnitInfos.xml if I include the bOnlyDefensive tag in the scan template (or any of the other 6 or 7 that I've listed in previous posts in this thread).

Please note that I don't wanna take much time out of your modding endeavors!

jmmendesp said:

Thanks. This makes to gravitate towards option a).

On another topic, do you have any idea why the Civ4XMLScanner you developed will not be able to process an xml file if there are certain tags included in the scan template?

For example, it won't process UnitInfos.xml if I include the bOnlyDefensive tag in the scan template (or any of the other 6 or 7 that I've listed in previous posts in this thread).

Please note that I don't wanna take much time out of your modding endeavors!

Click to expand...

No sorry. If you post the template file that causes the failure I'll take a quick look sometime though once more urgent stuff is out of the way.

Koshling said:

No sorry. If you post the template file that causes the failure I'll take a quick look sometime though once more urgent stuff is out of the way.

Click to expand...

Sure thing!

UnitInfosTemplate is my latest working template. ExtraTags is the same, except with the non-working tags added at the end of the template. You'll find that even when you leave 1 of those extra tags in the template, Civ4UnitInfos won't be processed. A template with nothing but these tags (or even a single one of them) won't work either.

I'm using Caveman2Cosmos\Assets\XML\Units as the path to scan, by the way.

jmmendesp said:

Alright. I've been using C2C_CIV4UnitSchema.xml to look for tags that I may have missed in the UnitInfos.xml.

Here's the thing: there are many tags that appear in the schema, but not in UnitInfos.xml

Should I:

a) Assume for the time being that those tags are in the schema because they show up in a different .xml (like PromotionInfos.xml, or something), and therefore ignore those tags while working on UnitInfos.xml until I get to converting the .xml they show up in.

b) Assume they are tags that are currently not used by UnitInfos.xml, but that may be used by it in the future, and therefore:

b, i) Put them in the spreadsheet, for the time when they start being used, and to tell everyone reading the spreadsheet that they exist.

b, ii) Not put them in the spreadsheet until they start being used.

c) Do something else?

EDIT - As an example, take the MOD Commanders tags. They are three. iCommandRange and iControlPoints show up in UnitInfos.xml, so I added those to the spreadsheet. bOnslaught, however, does not.

Click to expand...

I'd like them to be brought up for sure. That's interesting what you and Koshling found out about Onslaught... hmm... Anyhow, its truly worthy of note. Finding those tags we aren't using could be quite helpful. It gives us the option of either eliminating them entirely, building it out if there's still some work to be done there, or simply starting to use it if its something we currently don't but could be. Also, keep in mind that the tag MAY be in use in the modular units still so you wouldn't want to eliminate that possibility.

I'd put them on the spreadsheet but maybe divide them from the rest of the pack with a highlight color or putting them at the end with a blank column between. I'd also put some notes on it somewhere to let us know what you've found out about it.

The unit and promotion schemas are in the same file. To tell if a tag belongs to promoions or to units (or to both) you need to look at where the element defiend is used. If it's in the PromtionInfo element definition (in the schema) then it's used by promotions, if in the untiInfo by units. bOnslaught is a promotion tag, though it doesn't appear to do anything so I guess it was a WIP by whoever wrote the GC mod. It's text says 'Continues attack while at full strength', but the code does not implement this. Initiative IV (GC promotion) has this tag (nothing else does), but as I say - it doesn't appear to be implemented, which means the pedia text for that promtion is basically lying!

Click to expand...

I could probably work this out actually... I'll have to take a look at it closely myself in the code but I've already worked out something similar in another area. There's a few places I'm wondering if they got all set up then just couldn't figure out how to execute the actual effect.

Thunderbrd said:

Finding those tags we aren't using could be quite helpful. It gives us the option of either eliminating them entirely, building it out if there's still some work to be done there, or simply starting to use it if its something we currently don't but could be. Also, keep in mind that the tag MAY be in use in the modular units still so you wouldn't want to eliminate that possibility.

Click to expand...

Yeah, I thought about these possibilities, too. Here's what I'll do. With the schema as my guide, I'm eventually gonna convert all the .xml's in each folder, with separate sections in the spreadsheet for each .xml (one for units, one for promotions, etc.). Reason being, I think only after doing all the xmls in the respective folder will I be able to find out how many tags in the schema are not being used at all.

This is assuming that for each schema, all the xmls that depend on it are located in the same folder as the schema...which I'm assuming to be true. Please tell me if it's not

It absolutely must be. A schema may only apply to the folder its in. You will note that some of the unit schemas in the modules MAY differ slightly. That's ok. But to get an xml info file to load, there must be schema to guide it IN the same folder.

Thunderbrd said:

It absolutely must be. A schema may only apply to the folder its in. You will note that some of the unit schemas in the modules MAY differ slightly. That's ok. But to get an xml info file to load, there must be schema to guide it IN the same folder.

Click to expand...

Good news for me Thanks.