Tool for XML Validation

Thunderbrd · Oct 14, 2014

You do understand that Testing is not a phase of loading. If something comes up missing it errors. Simple.

Mind you, I think the first suggestion sounds possible to implement...

But I'll wait for Alberts to say more.

Dancing Hoskuld · Oct 14, 2014

By testing I meant validation, sorry for confusing you.

When I was writing computer systems, way back in the stone age

, you would read in the data validating that it was of the correct record format with the correct data types in the correct columns. After that you would start validating the values in the columns were consistent and valid. Then you would check between records, especially with data in structures similar to the class - infos structure in Civ. In that particular set of systems getting an infos without a class was a big error but having a class without any infos just meant that someone was late and need a prod (with a cattle prod sometimes

).

alberts8 · Oct 15, 2014

Dancing Hoskuld said:
Who mentioned stuff that was off because of modules? I'll try to explain what I am saying in a different way as it is possible to distinguish between spelling errors and WoC turned off stuff.

WoC will turn off the Infos if the dependency says so. All you need is remember which infos have been turned off and the class they belong to. When checking the Class for the existence of Infos if they have no Infos element but have at least one turned off element then they are not a spelling mistake candidate and should not be reported. Otherwise they should. It is a very straight forward test no magic required.

Code:

While doing WoC processing of Infos XML element if not removed by dependencies do as usual else [B]write Class and Infos to new file[/B] do rest as usual. ... When Checking for errors in Class XML If Class has no Infos [B] check to see if class is in the new file if not error[/B]

Why should it matter if something was turned of because of missing dependencies?

You don't know the reason why a dependency type is missing, it could be intentional or a human error or whatever.
One example for such an error would be the recent merging of hydro's stuff and resulting missing promotions. No say you had a dependency to one of those missing promotions and that dependency fails now, how would you notice that? If you just ignore those errors you never know about it and think everything is working.

Dancing Hoskuld · Oct 15, 2014

It stops an error being reported which is not an error because you asked it to be turned off. this will be especially annoying for non modders who want to turn a module off. To them it ill appear there are errors and the only way to get rid of the error is to have the module on. Which means the module is not optional, so why try making is so to start with

Thunderbrd · Oct 15, 2014

I think if we miss error reports on dependency checks alone that won't be too risky in terms of error reporting... particularly if then that info is logged. Shouldn't expose us to all that much risk if that's the ONLY reason that it wouldn't display an error.

Would take a little setup but it doesn't sound like an overwhelming effort (not that I know how and where to do it of course.)

alberts8 · Oct 15, 2014

Dancing Hoskuld said:
It stops an error being reported which is not an error because you asked it to be turned off. this will be especially annoying for non modders who want to turn a module off. To them it ill appear there are errors and the only way to get rid of the error is to have the module on. Which means the module is not optional, so why try making is so to start with

Again what if you didn't ask it to be turned off but it is turned of by accident like in the example in my last post. Turning of some modules wasn't even possible before i changed anything.

Thunderbrd said:
I think if we miss error reports on dependency checks alone that won't be too risky in terms of error reporting... particularly if then that info is logged. Shouldn't expose us to all that much risk if that's the ONLY reason that it wouldn't display an error.

Would take a little setup but it doesn't sound like an overwhelming effort (not that I know how and where to do it of course.)

Failing dependency checks don't create error messages but they are logged.

Thunderbrd · Oct 15, 2014

alberts2 said:
Again what if you didn't ask it to be turned off but it is turned of by accident like in the example in my last post. Turning of some modules wasn't even possible before i changed anything.

Failing dependency checks don't create error messages but they are logged.

Sure but so are all other otherwise error messages unless you're running the debug dll right? So shouldn't ALL OTHER messages still show the error the way it would if you were running the debug dll?

alberts8 · Oct 16, 2014

Thunderbrd said:
Sure but so are all other otherwise error messages unless you're running the debug dll right? So shouldn't ALL OTHER messages still show the error the way it would if you were running the debug dll?

Failing dependency checks don't give error messages not even in the debug dll. It always was this way.

Missing UnitArtStyles always give error messages. I did this to avoid CTD's and some say it also breaks WoC.

Sometimes other missing stuff gives error messages in the release dll and it always was this way.

The rest only in the debug dll that is what i wanted to change and what we are talking about.

alberts8 · Oct 16, 2014

Dancing Hoskuld said:
By testing I meant validation, sorry for confusing you.

When I was writing computer systems, way back in the stone age , you would read in the data validating that it was of the correct record format with the correct data types in the correct columns. After that you would start validating the values in the columns were consistent and valid. Then you would check between records, especially with data in structures similar to the class - infos structure in Civ. In that particular set of systems getting an infos without a class was a big error but having a class without any infos just meant that someone was late and need a prod (with a cattle prod sometimes).

Our data is stored in xml files so i wanted to readd Xml Schema Validation using a XSD Schema. But this has nothing to do with the actual data stored in those files.

XSD (XML Schema Definition), a Recommendation of the World Wide Web Consortium (W3C), specifies how to formally describe the elements in an Extensible Markup Language (XML) document. It can be used by programmers to verify each piece of item content in a document. They can check it adheres to the description of the element it is placed in.[1]

Like all XML schema languages, XSD can be used to express a set of rules to which an XML document must conform in order to be considered 'valid' according to that schema. However, unlike most other schema languages, XSD was also designed with the intent that determination of a document's validity would produce a collection of information adhering to specific data types. Such a post-validation infoset can be useful in the development of XML document processing software. However, the schema language's dependence on specific data types has provoked criticism.

Source

I was told this can't be part of the mod because it gives anoying errors in case of XML errors :confused:

. I don't understand this because even plain Civ4 has it using those *Schema.xml files.

Thunderbrd · Oct 16, 2014

I've read everything posted on the forums here and I don't recall anyone ever saying that we cannot or even should not utilize validation mechanisms on our xml.

On the contrary, we SHOULD!!!

As it has been distilled down in conversation here, apparently DH's only concern is making sure that he can correctly and properly utilize WoC modularization and dependencies despite normal validation tools in use. Some of this requires that special provisions be made to make this possible so that some structures can be arranged in a manner that enables this modularization.

Nobody's asking for there to be no validation, just for some conditions to be enabled to bypass the normal validation.

alberts8 · Oct 16, 2014

It was said that if i wanted to do it it has to be seperate from the mod because

It gives anoying errors in case of wrong xml.
I wanted to use the same xsd file for all our xml files instead of seperate files in each folder like it was in Civ4. Because it is much easier to update and we don't need hundeds of files if one does the job. But that is not an option because you have to set the correct relative path to the xsd file in each xml file. Apparently this makes copying files around the living hell.
And so on.............. until i finally gave it up.

At the moment we only have it for the GameText and a few others.

Thunderbrd · Oct 16, 2014

because you have to set the correct relative path to the xsd

What do you mean by relative path?

Errors when the xml is wrong is a good thing... the problem was (so far as I could tell) the errors were violating the above noted requests to allow for proper modularization. Is it not possible to have our cake and eat it too here?

alberts8 · Oct 17, 2014

Thunderbrd said:
What do you mean by relative path?

Errors when the xml is wrong is a good thing... the problem was (so far as I could tell) the errors were violating the above noted requests to allow for proper modularization. Is it not possible to have our cake and eat it too here?

Each xml file must have the xsi:schemaLocation with the correct URI, in our case the relative path to the XSD Schema file.

Example:

Code:

<Civ4GameText xmlns="http://www.firaxis.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [B]xsi:schemaLocation="http://www.firaxis.com ../Schema/Civ4Gametext.xsd">[/B]
	<TEXT>
		<Tag>TXT_KEY_BONUS_WAX</Tag>
		<English>Wax</English>
		<French>Wax</French>
		<German>Wax</German>
		<Italian>Wax</Italian>
		<Spanish>Wax</Spanish>
	</TEXT>
............

About relative paths.

Paths

The path to a specified file consists of one or more components, separated by a special character (a backslash), with each component usually being a directory name or file name, but with some notable exceptions discussed below. It is often critical to the system's interpretation of a path what the beginning, or prefix, of the path looks like. This prefix determines the namespace the path is using, and additionally what special characters are used in which position within the path, including the last character.

If a component of a path is a file name, it must be the last component.

Each component of a path will also be constrained by the maximum length specified for a particular file system. In general, these rules fall into two categories: short and long. Note that directory names are stored by the file system as a special type of file, but naming rules for files also apply to directory names. To summarize, a path is simply the string representation of the hierarchy between all of the directories that exist for a particular file or directory name.

Fully Qualified vs. Relative Paths

For Windows API functions that manipulate files, file names can often be relative to the current directory, while some APIs require a fully qualified path. A file name is relative to the current directory if it does not begin with one of the following:

A UNC name of any format, which always start with two backslash characters ("\\"). For more information, see the next section.

A disk designator with a backslash, for example "C:\" or "d:\".

A single backslash, for example, "\directory" or "\file.txt". This is also referred to as an absolute path.

If a file name begins with only a disk designator but not the backslash after the colon, it is interpreted as a relative path to the current directory on the drive with the specified letter. Note that the current directory may or may not be the root directory depending on what it was set to during the most recent "change directory" operation on that disk. Examples of this format are as follows:

"C:tmp.txt" refers to a file named "tmp.txt" in the current directory on drive C.

"C:tempdir\tmp.txt" refers to a file in a subdirectory to the current directory on drive C.

A path is also said to be relative if it contains "double-dots"; that is, two periods together in one component of the path. This special specifier is used to denote the directory above the current directory, otherwise known as the "parent directory". Examples of this format are as follows:

"..\tmp.txt" specifies a file named tmp.txt located in the parent of the current directory.

"..\..\tmp.txt" specifies a file that is two directories above the current directory.

"..\tempdir\tmp.txt" specifies a file named tmp.txt located in a directory named

tempdir that is a peer directory to the current directory.

Relative paths can combine both example types, for example "C:..\tmp.txt". This is useful because, although the system keeps track of the current drive along with the current directory of that drive, it also keeps track of the current directories in each of the different drive letters (if your system has more than one), regardless of which drive designator is set as the current drive.

Source

I had planned to provide a better version of the tool i wrote to set the correct schemaLocation in all files. To make it easier to import stuff from other mods it could also reformat those files to the new format and so on.

Schema Validation was considered to be a bad thing like all the other error checking i put in. The C2C release dll already had no Schema Validation before i switched the Xml-Parser so i guess it also wasn't wanted there before i joined.

As i understood it the dll should magicly handle all kinds of errors and never ever put out error messages or CTD or whatever. If it fails those darn dll programmers are to blame.

Thunderbrd · Oct 17, 2014

It's not really a relative location at all. It's a fixed location. It doesn't change if you move an xml file from one folder to another... where the schema is is where it stays. Therefore moving an xml file will NOT mean you must point to a new location for the schema. That sounds more convenient than the current method where the new folder may have a new schema name (and will need to at least HAVE a schema!)

When you joined we were left in a state between validations. I'm not sure what AIAndy was thinking other than trying to streamline loading but n47 had just changed a lot of things there and vanished mid-project. So it was NOT setup the way it really should've been even then.

What was expressed before was not that the errors were bad in and of themselves but that they were coming up under the wrong situations - where they should've been ignored due to modularizing/dependency. Personally I still can't tell if it was a situation where the error should or shouldn't have fired.

I believe it was coming up because an object removed by a dependency was called for elsewhere and there was no 'list of objects removed by dependencies' for the error checker to check against before making sure it should be throwing up the error. THAT was the complaint. Somehow the OLD validation mechanism used to allow what he was doing... perhaps simply by being substandard.

At least, I think that was the perception of what the problem was. I think the difference is that the missing object was missing due to the removal of a module and had nothing to do with dependency. In that case, no error popup structure can account for that right? Because the files in modules that are turned off are completely invisible to the dll... no loading process checks them at all and they are thus make it impossible to track what's missing due to current MLF settings. But if a dependency checks for an object and the object doesn't exist then we are supposed to be able to assume that it's due to MLF settings and that should therefore not error out.

DH has long been a rare fan of WoC modularization and dependencies and plans nearly all he does around it. So when it stops working as he expects it to he's gonna get a little testy. Don't over-extend those messages of frustration to mean ALL validation should not be taking place. He was also one of the voices complaining when we lost validation on the main dll in the first place. (Yes... before your arrival.)

alberts8 · Oct 17, 2014

Thunderbrd said:
It's not really a relative location at all. It's a fixed location. It doesn't change if you move an xml file from one folder to another... where the schema is is where it stays. Therefore moving an xml file will NOT mean you must point to a new location for the schema. That sounds more convenient than the current method where the new folder may have a new schema name (and will need to at least HAVE a schema!)

It is relative and it could change if you copy files arround.

If you have a file in lets say "Caveman2Cosmos\Assets\Modules\DancingHoskuld\Subdue_Animals\z_experimental\Arcticfox" it is.

Code:

../../../../../Xml/Schema/Civ4Gametext.xsd

If you Copy that file to "Caveman2Cosmos\Assets\Modules\DancingHoskuld\Subdue_Animals" it is.

Code:

../../../Xml/Schema/Civ4Gametext.xsd

That is why i wanted to create a tool to fix them all at once.

Thunderbrd said:
When you joined we were left in a state between validations. I'm not sure what AIAndy was thinking other than trying to streamline loading but n47 had just changed a lot of things there and vanished mid-project. So it was NOT setup the way it really should've been even then.

What was expressed before was not that the errors were bad in and of themselves but that they were coming up under the wrong situations - where they should've been ignored due to modularizing/dependency. Personally I still can't tell if it was a situation where the error should or shouldn't have fired.

I believe it was coming up because an object removed by a dependency was called for elsewhere and there was no 'list of objects removed by dependencies' for the error checker to check against before making sure it should be throwing up the error. THAT was the complaint. Somehow the OLD validation mechanism used to allow what he was doing... perhaps simply by being substandard.

At least, I think that was the perception of what the problem was. I think the difference is that the missing object was missing due to the removal of a module and had nothing to do with dependency. In that case, no error popup structure can account for that right? Because the files in modules that are turned off are completely invisible to the dll... no loading process checks them at all and they are thus make it impossible to track what's missing due to current MLF settings. But if a dependency checks for an object and the object doesn't exist then we are supposed to be able to assume that it's due to MLF settings and that should therefore not error out.

DH has long been a rare fan of WoC modularization and dependencies and plans nearly all he does around it. So when it stops working as he expects it to he's gonna get a little testy. Don't over-extend those messages of frustration to mean ALL validation should not be taking place. He was also one of the voices complaining when we lost validation on the main dll in the first place. (Yes... before your arrival.)

But as i said before i did not create any of those error checks they existed already in the original WoC mod. There are a few reasons for the AndDependencyTypes tag and i don't understand why it is used almost nowhere. Using it could solve a few problems DH has and maybe we could improve this tag a bit because it has limitations. Maybe it needs a bit more xml if you use this tag but that is how WoC was built.

Here is a example for a missing AndDependencyTypes tag, this xml was created to extend BUILDING_MICROLITH_WORKSHOP:

Code:

		<BuildingInfo>
			<BuildingClass>BUILDINGCLASS_MICROLITH_WORKSHOP</BuildingClass>
			<Type>BUILDING_MICROLITH_WORKSHOP</Type>
			<Bonus>BONUS_STONE</Bonus>
			<BuildingClassNeededs>
				<BuildingClassNeeded>
					<BuildingClassType>BUILDINGCLASS_STORAGE_PIT</BuildingClassType>
					<bNeededInCity>1</bNeededInCity>
				</BuildingClassNeeded>
			</BuildingClassNeededs>
		</BuildingInfo>

But if that building would not exist because a module is turned of or whatever, this xml adds a basicly emty version of BUILDING_MICROLITH_WORKSHOP. So BUILDING_MICROLITH_WORKSHOP has to be a AndDependencyType here.

But if this xml is loaded before the actual BUILDING_MICROLITH_WORKSHOP the AndDependencyTypes tag fails. This issue exists since WoC was created and it should have been fixed a long time ago insted of not using this tag at all.

This is a much bigger issue if you want to turn modules off because it creates lots of problems!!

Dancing Hoskuld · Oct 17, 2014

alberts2 said:
... i don't understand why it is used almost nowhere. Using it could solve a few problems DH has and maybe we could improve this tag a bit because it has limitations. Maybe it needs a bit more xml if you use this tag but that is how WoC was built.

I am the only one who uses them because I was the only one who was attempting making optional mods. Everyone else was making non optional "core" changes so what they were referring to in the XML has to exist or it is an error.

In a full WoC implementation everything would be in its own module folder each XML file would contain a full set of dependencies and every folder would contain a copy of the needed schema files. This means that there would be over 3000 building folders in the Modules folder alone. (It is very difficult to keep the schema files up to date in such a situation which is why I applaud the schema work being done.)

In less full implementations of a WoC based mod you put as much for an option can in one folder. Then you have to do fewer dependencies since the majority are implicit. The problem comes when many modders are working on stuff and assume that what they are doing is in the "core". You don't find out this until you change the options by turning off modules and get XML errors. Getting "false positive" errors makes the debugging even harder.

alberts2 said:

Here is a example for a missing AndDependencyTypes tag, this xml was created to extend BUILDING_MICROLITH_WORKSHOP:

Code:

		<BuildingInfo>
			<BuildingClass>BUILDINGCLASS_MICROLITH_WORKSHOP</BuildingClass>
			<Type>BUILDING_MICROLITH_WORKSHOP</Type>
			<Bonus>BONUS_STONE</Bonus>
			<BuildingClassNeededs>
				<BuildingClassNeeded>
					<BuildingClassType>BUILDINGCLASS_STORAGE_PIT</BuildingClassType>
					<bNeededInCity>1</bNeededInCity>
				</BuildingClassNeeded>
			</BuildingClassNeededs>
		</BuildingInfo>

To be fully correct there should be AndDependencyTypes for all four entities; BUILDINGCLASS_MICROLITH_WORKSHOP, BUILDING_MICROLITH_WORKSHOP, BONUS_STONE and BUILDINGCLASS_STORAGE_PIT.

Edit Of corse you would not put the AndDependencies in until after you had done the first set of testing otherwise you wont get the proper error messages to get the entity working as if everything was included.

Thunderbrd · Oct 17, 2014

It is relative and it could change if you copy files arround.

If you have a file in lets say "Caveman2Cosmos\Assets\Modules\DancingHoskuld\Subdu e_Animals\z_experimental\Arcticfox" it is.
Code:
../../../../../Xml/Schema/Civ4Gametext.xsd
If you Copy that file to "Caveman2Cosmos\Assets\Modules\DancingHoskuld\Subdu e_Animals" it is.
Code:
../../../Xml/Schema/Civ4Gametext.xsd
That is why i wanted to create a tool to fix them all at once.

Ah... very DOS - ish. Ok - that doesn't sound too bad. Certainly no living hell (imo).

But as i said before i did not create any of those error checks they existed already in the original WoC mod.

And that's why all along I've been trying to tell you not to take it personal. Obviously there has been some misunderstandings here as to what can and cannot be done successfully. And again, some of the things that could be done might've only been possible because something was broken and thus allowed it when it shouldn't have.

I think you're right though that dependencies were needing to have been applied in ways they weren't in the original situation that brought it all up in the first place.

alberts8 · Oct 17, 2014

Dancing Hoskuld said:
To be fully correct there should be AndDependencyTypes for all four entities; BUILDINGCLASS_MICROLITH_WORKSHOP, BUILDING_MICROLITH_WORKSHOP, BONUS_STONE and BUILDINGCLASS_STORAGE_PIT.

Yes that would be the complete way to do this, the most important one in that case is BUILDING_MICROLITH_WORKSHOP. The BuildingClass tag in that example isn't needed so one could remove it and end up with three AndDependencyTypes.

Dancing Hoskuld · Nov 12, 2014

Except sometimes the Class is needed. The way to do flavor units/buildings is to have the default unit/building and all the flavor units/buildings have the same class. You can see an existing use of this with the female missionaries which have the same class as their male counterparts but different infos elements. In that case it is just about the graphics but in others everything is different.

alberts8 · Nov 17, 2014

Dancing Hoskuld said:
Except sometimes the Class is needed. The way to do flavor units/buildings is to have the default unit/building and all the flavor units/buildings have the same class. You can see an existing use of this with the female missionaries which have the same class as their male counterparts but different infos elements. In that case it is just about the graphics but in others everything is different.

That is a totally different case.

The example from above

Code:

		<BuildingInfo>
			<BuildingClass>BUILDINGCLASS_MICROLITH_WORKSHOP</BuildingClass>
			<Type>BUILDING_MICROLITH_WORKSHOP</Type>
			<Bonus>BONUS_STONE</Bonus>
			<BuildingClassNeededs>
				<BuildingClassNeeded>
					<BuildingClassType>BUILDINGCLASS_STORAGE_PIT</BuildingClassType>
					<bNeededInCity>1</bNeededInCity>
				</BuildingClassNeeded>
			</BuildingClassNeededs>
		</BuildingInfo>

This changes a already existing building and the BuildingClass is not needed here because it is the same as it is in the initial BUILDING_MICROLITH_WORKSHOP.

Tool for XML Validation

C2C War Dog

Deity

Emperor

Deity

C2C War Dog

Emperor

C2C War Dog

Emperor

Emperor

C2C War Dog

Emperor

C2C War Dog

Emperor

C2C War Dog

Emperor

Deity

C2C War Dog

Emperor

Deity

Emperor

Similar threads