XML Inter-dependencies

Belizan · May 22, 2006

I've been looking at writing an XML edittor which would test certain interdependencies between the various XML files which do not seem to get tested during mod loading, but can cause some serious problems. I know of a few myself, and I figure within the community there have to be a great many modders more familiar with the foibles of XML then I. To that end, I had hoped to try to get some sort of catalogue of all the known XML gotchas people have found over the last.. nine months? Or however long we've been at this.

I know there are a lot of tags to link up, potentially... Domains, UNITCOMBAT types, AITypes, etc., etc. What I'm most interested in here are things which the loader itself doesn't catch (and which may lead to CTDs).

So by way of example, the ones I know of...

<Button> elements need to refer to extant .dds files (or an index into the button array)
ArtDefineTags need to exist in the appropriate file (e.g. in CIV4ArtDefines_Unit.xml for Units)
Only one BUILDING per BUILDINGCLASS
Sound tags need to exist in the appropriate file

<nif>, <ShadowNif>, <shadernif> and <kfm> files must exist if specified.
<filename> files in Audiodefines need to exist (at least for referenced defines)

Anyone ever run into bugs caused by mismatched or missing <FootstepSounds>?

Anyway, I'm sure there are quite a few more gotchas. I'd really appreciate it if we could gather our collective knowledge of these issues here. (Even if I never finish the edittor, it would still be an invaluable reference for modders trying to find those esoteric XML bugs).

Belizan · May 22, 2006

Hrmm.. ok. Maybe no one else has run into any troubles? Or maybe it's just not a desireable tool. *shrugs*

The Great Apple · May 22, 2006

Missed this thread completely first time around.

I think alot of these can be fixed by poking around with the SDK and adding error handling. For example, I think SimCutie's fix here addresses alot of the art crashes you can get.

This is the only crash I ever came up against...

Belizan · May 22, 2006

Maybe, but I still think a general Civ-aware validating XML editting tool would be so amazingly useful. But, I guess I am alone in that :/. Ah well.

The Great Apple · May 22, 2006

Belizan said:
Maybe, but I still think a general Civ-aware validating XML editting tool would be so amazingly useful. But, I guess I am alone in that :/. Ah well.

Actually, I kinda skimmed over the big bit of text at the top. I think it would be useful to have something you could click on and it would tell you what was working and what wasn't. I'm not entirely sure, however, that it would be worth the hassle as the game picks up most of them.

It would be good for the showstoppers, but surely making them not stop the show would be event better, right?

Belizan · May 22, 2006

The Great Apple said:
Actually, I kinda skimmed over the big bit of text at the top. I think it would be useful to have something you could click on and it would tell you what was working and what wasn't. I'm not entirely sure, however, that it would be worth the hassle as the game picks up most of them.

It would be good for the showstoppers, but surely making them not stop the show would be event better, right?

Yes and no, preventing them from "stopping the show" doesn't particularly help with finding, identifying and removing them 8). On the moon mod we've been wacked by some really bizarro XML related bugs, that have come close to killing the project. With some of these things I worry that the issue is that even if we short up the leaks we can find, we won't have actually created an environment in which a modder can find, identify and remove these bugs. Instead, we'll have a petry dish of ever mutating strange and unreliable behaviour that doesn't crash 8).

Shqype · May 22, 2006

I know I've come into a few things which have caused problems ... some severe, others not. The problem is, I haven't written them down and usually forget what they were

I do know of something which is an annoyance, although the game continues after the error message.

If you have incorrect tags you will get an "error" message while loading the game that is essentially blank. Let's say instead of having:

Code:

<Tag>NONE</Tag>

you have:

Code:

<Tag/>

It will cause this popup "error" upon loading the XML when starting the game. It will say that file that this occurred in but the "error" message will otherwise be blank. If you press okay the game still loads without a problem, but it's just a small nuisance to be aware of.

Agent 009 · May 24, 2006

One of the things I tend to do when editing XML is make a change and then test it. Of course if the change requires multiple file edits then I change all of them first.

I do this simply to avoid dumb little typo errors that can occur from time to time. Once, one of those times was leaving out a < somewhere which cause a whole bunch of errors in many different XML files. :wallbash:

As aggravating as that is, it is something that would very likely be a massive undertaking that would require you to have the program relate to the Schema files as well so that all of the XML files could be test loaded to see if any errors come up.

I have altova here and I never use it cause I can't get it to recognize the schema. It keeps giving me an error saying it isn't a valid schema file...

So, I just use notepad instead, and do my own troubleshooting.

Belizan · May 24, 2006

The sorts of XML problems I'm worried about only crop up after 100s of turns of play. Such a level of incremental play testing is simply not feasible

.

Agent 009 · May 25, 2006

Belizan said:
The sorts of XML problems I'm worried about only crop up after 100s of turns of play. Such a level of incremental play testing is simply not feasible .

Hmm.... I never had that kind of problem. :rolleyes:

Shqype · May 25, 2006

Belizan said:
The sorts of XML problems I'm worried about only crop up after 100s of turns of play. Such a level of incremental play testing is simply not feasible .

I didn't know that XML can produce such errors ... unless, of course, there is a problem with a tag for a building/unit that gets created in the mid - late game, but that's something else.

Kael · May 25, 2006

I just wipe the xml.log, load the mod, then go through the xml.log file. Usually anything like you are mentioning shows up in there. I have even seen SDK typos listed in there (C++ was checking to see if a player had a particular civic selected and I had mistyped the trait name). The really nice thing is that it parses all the code, you don't have to actually run the python function or build the xml object to see the error listed.

Its awesome that Firaxis logs all of these errors though I suspect a lot of modmakers don't realize they are there.

Padmewan · May 25, 2006

Shqype said:
If you have incorrect tags you will get an "error" message while loading the game that is essentially blank.

Man, I wish you'd shared this earlier! This was such an inexplicable error that I never knew how to find and kill it.

Padmewan · May 25, 2006

Anyway, as to the point of this thread, I'd inquired before about even just having an XML validator. I mean, the schema is there.

Shqype · May 26, 2006

Kael said:
I just wipe the xml.log, load the mod, then go through the xml.log file. Usually anything like you are mentioning shows up in there. I have even seen SDK typos listed in there (C++ was checking to see if a player had a particular civic selected and I had mistyped the trait name). The really nice thing is that it parses all the code, you don't have to actually run the python function or build the xml object to see the error listed.

Its awesome that Firaxis logs all of these errors though I suspect a lot of modmakers don't realize they are there.

I haven't found anything useful in the log files for this case (or maybe I just don't know how to interpret it). I recieved some XML files for the Star Wars Mod and have been putting it together, but I get a CTD every time. The mod loads fine with no errors but as soon as a city is built (if not by me, I skip to the next turn and it gets built by the AI) there is an automatic CTD. I have no idea what this could be except for faulty city graphics, but I didn't touch those; I only renamed the eras.

Any ideas?

Belizan · May 26, 2006

We've had two bouts of XML-related ~100 turn delayed reaction CTD bugs in the Moon Mod. I'm sure they have to do with art related issues for later era buttons, models or terrain, given our mod (and the vagaries of when such get introduced). The first one Matthewv tracked down. The second we never found, and had to scrap that version and revert. :/.

I haven't ever looked through that xml.log file, to be honest, Kael. I'll have to check that out. That's a good tip, not sure why I never thought to look there :/. I assume that the XML monkeys on our team did, however, and they didn't find our bugs either...

As for a validating XML edittor, that's easily done. I had thought to go a step past that (as only about 6 of the XML formats have schemas to my recollection). It seems though that only Padmewan and myself would be particularly interested in such a tool. Maybe I just don't have a good enough XML edittor. Notepad++ or SciTE (same thing, really) are not bad but do no validation of the sorts I'm interested in. I like DiffDog for XML diffing, but my eval key expired, and they want $170 for a full license, which is more money then I'm willing to put out for a work-like hobby 8). XMLSpy didn't impress me, and in the end I let my key time out without ever really using it.

Agent 009 · May 28, 2006

Originally Posted by Shqype
If you have incorrect tags you will get an "error" message while loading the game that is essentially blank.

I've had that one when I made Throughout the Ages.

Basically, what happend was, when loading the XML, I'd get an error related to the techinfos file which said:

Tag: ` in info class is incorrect. The current file is CIV4TechInfos.xml.

Searching the file for an apostrophe or a space or even both yielded no results whatsoever. I was completely dumbfounded. I was so aggravated by this error that I even went through the entire file and still couldn't find the problem.

It then suddenly occured to me... What if it wasn't an apostrophe? What if it was an ASCII value instead. So I looked for the ASCII codes and found one that looked like an apostrophe.

Look at this difference.

Standard apostrophe: '
ASCII value 96: `

Both totally different. :eek:

I was so angry at my PC because it was Notepad/Microsoft that put two of these apostrophes in my techinfos file. It certainly wasn't me, nor was it anyone else in my household as I am the only one that uses this PC.

At this point I am certain it wasn't someone else cause the problem occured again in another file. This time the file was open and while I was editing it, I noticed suddenly that there was this apostrophe in a similar place as the last two. I hadn't even got up from the PC so there was no way it could have been my brother just joking around or something.

I believe now the problem was indeed notepad. So my advice to all modders who use notepad just as I do, if you get a similar error, think ASCII.

Kael · May 28, 2006

Shqype said:
I haven't found anything useful in the log files for this case (or maybe I just don't know how to interpret it). I recieved some XML files for the Star Wars Mod and have been putting it together, but I get a CTD every time. The mod loads fine with no errors but as soon as a city is built (if not by me, I skip to the next turn and it gets built by the AI) there is an automatic CTD. I have no idea what this could be except for faulty city graphics, but I didn't touch those; I only renamed the eras.

Any ideas?

Yeah, art errors won't be caught in the log files. SimCutie posted an SDK fix that should keep some art errors from CTDing the game (there is an error in the code). (http://forums.civfanatics.com/showthread.php?t=169012)

For myself I noticed just tons of little typo kind of things in the XML log that I didn't even realize were problems until I looked there.

Shqype · May 28, 2006

Kael said:
Yeah, art errors won't be caught in the log files. SimCutie posted an SDK fix that should keep some art errors from CTDing the game (there is an error in the code). (http://forums.civfanatics.com/showthread.php?t=169012)

For myself I noticed just tons of little typo kind of things in the XML log that I didn't even realize were problems until I looked there.

Thanks Kael. I made the change and compiled a new DLL and replaced it with the vanilla one. Unfortunately I still have the same problem. The game loads fine, the game begins fine, but then I move on to settle my first city. The popup appears for the city name and at the top of the screen all the researchable technology icons appear, but when I OK the name of the city the game automatically CTDs without any error messages at all, not even Windows' "Civilization4.exe must be closed" error. I really don't know why this is happening, and it's really frustrating!

Shqype · May 28, 2006

I just checked the logs and I found some interesting things in there. The xml.log gave me a whole lot of "[8615.625] info type NONE not found, Current XML file is: Units/CIV4UnitInfos.xml" kind of errors, and similar ones for buildings. Do you know what causes these?

The resmgr.log gave me this:

RESMGR: Texture Art/Interface/Buttons/Units/ANAKIN.dds failed to load
RESMGR: Texture Art/Interface/Buttons/Units/DEATH_RAY.dds failed to load
RESMGR: Texture PlayerColor01.tga failed to load
RESMGR: Texture PlayerColor02.tga failed to load

I don't have the graphics for the first two, so that makes sense, but where are the last two?

And, the PythonDbg.log ends with me founding the city:

Spoiler :

Do you have any other thoughts as to what might be causing the CTD whenever I build a city (and OK its name)?

XML Inter-dependencies

Emperor

Emperor

Big Cheese

Emperor

Big Cheese

Emperor

Shqyptar

Civ 4 Addict

Emperor

Civ 4 Addict

Shqyptar

Deity

King

King

Shqyptar

Emperor

Civ 4 Addict

Deity

Shqyptar

Shqyptar

Similar threads