Modular Promotion Issues

[xienwolf]

I've been working on Modular Mods for a while now, and I recently moved into modding the SDK. While I was there I ran into a nifty pattern which I recognized from the fields that fail to load in Modular XML mods.

It is explained in more depth in the spoiler, but the general gist of it is that the XML is loaded up into the DLL file and utilized from there. Once the XML has laoded upon starting the game, it is never actually touched again.

This loading is done in phases (reasons detailed more in Spoiler), and it is in moving from the first phase into the later ones that Modular Loading fails. The manifestation of this failure is that certain fields will appear on the main mod components instead of appearing on your new modular components (things like Upgrading Improvements, and Promotion Pre-Requisites).

Spoiler Detailed Account :

It is part of the SDK that is responsible for loading the XML. Since the XML is loaded in a certain order, and some parts make references to pieces which are not yet loaded, those parts cannot be loaded yet, and are saved for the second pass through the XML (pass 2).

An example of this would be if a Promotion had a Pre-Req Promotion which came after it in the alphabet (since they are mostly listed in the XML alphabetically). When the game tries to load that pre-req, it would want to convert the string PROMOTION_BLAHBLAH into a number, based on the enumeration of PROMOTION_BLAHBLAH. But since that promotion is not yet loaded, it would give you an error.

Thus the Pre-reqs must wait until after each promotion was loaded (without the Pre-req information) before going back through and attaching that extra information to each field.

Anything which is loaded in the first pass works for Modules properly. But everything loaded in the second pass fails because it then tries to assign it by the Enumeration rather than by the Type name, and the enumeration starts over again at 0 when it loads the module in passes 2 & 3.

So I guess what I am mainly asking is: Does anybody know enough about the DLL to modify the linking of Passes 2 & 3 properly to Pass 1 with Modules involved? Or at least enough to provide a hint as to how I could start working on a way to fix it.

 

[davidlallen]

Similar problems here and here.

This is weird! In my case changing whitespace (deleting a comment) changes which prereq's are messed up. With the file I linked, some prereqs are moved two later in the list (eg, Erratic Driving 1 should allow erratic driving 2 but actually allows hardpoint, which is two after erratic driving 2) and some are moved one later in the list (less fuel 1 should allow less fuel 2, but actually allows less fuel 3). When I delete a comment, then Erratic Driving 1 allows Erratic Driving 3, which is still wrong but one later in the list instead of two; and the less fuel promotions work correctly.

If it is possible to workaround this problem by putting the promotions in a different order, what might be the right solution? Putting my new promo's into the list in alphabetical order? I had them segregated at the end of the file.

I am sure there are big mods which have added lots of promotions, yet do not seem to have run into this problem. I will ask over there too, eg FFH2.

 

[xienwolf]

Well, I think that in your case, with the comments, it is an issue of the XML loading expecting 1 item per line. Thus if you move your comments to be on the same line as the fields, you will be covered.

EXAMPLE:

            <Type>PROMOTION_COMBAT1</Type>  <!--  Increased the Bonus to 20% instead of 10% -->

The above should work. The below would cause minor issues.

            <Type>PROMOTION_COMBAT1</Type>
<!--  Increased the Bonus to 20% instead of 10% -->

What I am talking about in the first post only pertains to Modules, and far more than just promotions. But Promotions is the one which people will run into the issue most often (second being Improvements IMO).

Not sure how the things were messing up with the second post you linked Wierdness.

 

[davidlallen]

I have never come across the requirement that a comment cannot go on a line by itself. Is that true for xml in general, for all civ files, or just for the promotion file?

I have deleted the comment. Also I fixed one problem where a promo required itself (hardpoint). In my example file there was an apparently unrelated problem that I had duplicated one of the promos (sentry). After these corrections, my prereqs are now showing correctly. But, just removing the comment was not enough, so there is some other problem.

Regarding the order dependent loading, I guess you are saying that if a promo requires some other promo which is later in the file, then this type of prereq ordering problem could occur. I looked further into the standard BTS promo file. It is almost always true that if promo A requires promo B, then A comes after B in the file. But I found one counterexample: March requires either Medic 1 or Combat 3, and it occurs before Medic 1 in the file.

So whatever is the underlying problem you are tracking down, it may be interesting to see why it does not affect March in the standard BTS.

 

[woc]

http://www.worldofcivilization.net

btw, we can solve your comment skipping error(which indeed exists on several spots in the SDK code) generally, don't put comments INSIDE a type definition to avoid this, generally, never add comments in the xml files themself..they are BS and not needed, anyone can use winmerge if he feels the need to know what's different on a file, thats more professional then commenting a file like crazy

2ndly, yes we totally control and have almost completely rewritten the loading mechanism for BtS..modular loading is completely different done in WoC..

 

[davidlallen]

Did you just say, comments are unprofessional and people should use diff to find what is changed ??

As a professional programmer for 20 years, I disagree. Comments are critical to understand *why* something was changed, which diff alone cannot tell you.

I would like to understand more about the restrictions the civ xml parser places upon comments in the files. I was not expecting to be bitten by this, and I refuse to stop putting comments into my code, so does anybody have specifics on the problem?

 

[NikNaks]

The XML parser just chokes when it gets to comments. There's no real way of knowing when or if it will break a file.

 

[davidlallen]

Is comment handling in the civ xml parser really that broken ??

All of the standard BTS files have multiple comments at the start of the file. I always keep a changelog here, so I hope that is ok.

I have been putting comments in the middle of my files; this is the only time I know it has caused problems. xienwolf suggests avoiding comments on lines by themselves, which seems like a workaround but should not be fatal.

However, one thing I often do is use a comment to temporarily get rid of a block of xml. For example, I want to remove a unit, but before deleting it I want to make sure I understand all the other places the unit is referenced. (For example, I would not have thought that the leaderhead file would make references to watermills!) Once I have fixed, or at least understand, all the references then I will actually delete it.

Can commenting out xml blocks introduce some other sort of problem? It seems that would be a real drag to proper development.

 

[xienwolf]

The reason that March doesn't run into the issue I discussed in post 1 is that it isn't being done in a Module. And the second pass isn't something used only for cases where the promotion referenced comes later, but for all references of promotions (just in case it happens to come later).

The comment issue makes a bit of sense as being inside of an Element causing the problem. The way that the SDK loads the XML is line-by-line. It doesn't know what to expect next (even though the Schema could easily tell it that), instead of just reads what is on one line, loads it as instructed, and then moves to the next line and attempts to load that. Thus I think that placing a Comment on a line all by itself causes the game to decide that it cannot load what it is looking at, and it moves on to the next field (thus a comment will cause all fields following the comment to be loaded 1 entry off the mark).

This could be a text-only issue though, as when I had run into the problem personally all that was shifted was the promotion pre-reqs and next-levels, not any of the raw numbers for the INT or Boolean Fields (though I didn't exactly check to see if they were all 0 after the comment).


I'll check out WoC and see how you guys have re-written Modular Loading. Hopefully your uncompiled DLL is easily available?

EDIT: To address DavidAllen's questions posted while I was typing:

As long as your comments are all in-between </UnitInfo> from one entry, and <UnitInfo> from the next, you shouldn't have issues. The only problems you'll encouter I believe are when you make a comment that has a line to itself in between <UnitInfo> & </UnitInfo> of a single entry.

And that means that commenting out an entire entry would be safe, as it would then be securely in-between the entry before and after it.

EDIT2: WoC: Have you guys noticed/Solved the issue with Vista Users not being able to use Modular Loading? Many people have run into instant CtD if they have modular loading enabled, but when they disable it they notice that SOME of the modular items load into the game (but not all, and thus rather buggy)

 

[davidlallen]

@xienwolf: Thanks for the help, sorry for partly hijacking your thread to discuss comments. FWIW, in my original file, the offending comment is between a </PromotionInfo> and the next <PromotionInfo>. Maybe I am confused about the difference between an element and an entry. I will just avoid comments on lines by themselves except at the start of the file.

 

[NikNaks]

I'll check out WoC and see how you guys have re-written Modular Loading. Hopefully your uncompiled DLL is easily available?

Yes it is. On our SVN.

EDIT2: WoC: Have you guys noticed/Solved the issue with Vista Users not being able to use Modular Loading? Many people have run into instant CtD if they have modular loading enabled, but when they disable it they notice that SOME of the modular items load into the game (but not all, and thus rather buggy)

It runs perfectly well on Vista. It even works on Linux.

 

[woc]

Did you just say, comments are unprofessional and people should use diff to find what is changed ??

obviously either you or me have a problem with the english language, but no, i didn't say that, i said:

don't put comments INSIDE a type definition to avoid this, generally, never add comments in the xml files themself

but maybe my english is too bad and I don't see that in english these 2 statements are 1 and the same, in my language they are not the same, sorry for the misunderstanding.

As a professional programmer for 20 years, I disagree. Comments are critical to understand *why* something was changed, which diff alone cannot tell you.

Then you haven't worked much with XML and the XML schema validation offered by MS in their msxml3.dll used by firaxis. The point is, every single tag inside a XML object must be defined in the schema, so if you wanna comment on a tag, you usually don't do that in the tag, or above the tag in the XML files(you'd had then to comment dozens of files if you were going to make dozens of files using the same schema definitions) instead, you comment the schema file(that's how all good modders here at CFC work, theLopez, Dale, and bunch of other guys) But feel free to disagree...

I would like to understand more about the restrictions the civ xml parser places upon comments in the files. I was not expecting to be bitten by this, and I refuse to stop putting comments into my code, so does anybody have specifics on the problem?

Not meant to byte anyone, and putting comments I'm not referring to code, but to the XML as explained above, if you understand how XML works, you can easily see you add a comment to a schema file(which can be used 100times for 100files) instead of putting the comment on a tag into 100 files..well, you could, but it's not advisable


Let me explain it more simple, there's a code that loops through the object classes in the sdk, each class has it's own xml(well most do as of BtS) in warlords, many were in a single file..anyway, each XML then is looped for the types(infoclasses, well, not types, but it uses the types as main tag for each infoclass), this loop uses a "search next sibbling and skip comments" method,
then inside each class/infoclass(SDK), type/infoclass(xml) so class(sdk)=type/infoclass(xml) the code runs through the single tags for that type, most of them search for a tag untill they find it(fastest is of course to have the same XML schema definition as the SDK walks through the tags, else it searches a bit longer hehehe, anyway, most have that, some don't, they don't "search untill found and skip comments"...you can consider that firaxian bugs, but can easily be avoided if you comment your own made tags in the schema file instead of commenting the xml file.

damn, im bad at explaining, hehe..anyway, don't pick on words, i think it should be understandable how I explained, at least the rough how it works..

comments on top of the file, as well on the bottom don't concern the SDK completely, because the SDK doesn't even see that, it jumps directly to the class structure in the XML, hence skipping anything above that 100%,

not meant to bite you! but we, and i guess most modders here, found it more professional to put a comment in the schema file about a tag, and not commenting the same tag over and over again in dozens of xml files..sorry if that bites you, im not a vampire, so it's not meant like that

And surely i didn't say:"writing comments is not professional" ...i refered to the comments INSIDE a
<...classinfo>
<type>BLALBLALBAL</type>
<tags></tags>
.............
<tagsXXX></tagsXXX>
</...classinfo>
but feel free to call me an ashole for giving you just the clue not to put them there..just wanted to be helpfull, sorry for feeling biten

 

[woc]

As long as your comments are all in-between </UnitInfo> from one entry, and <UnitInfo> from the next, you shouldn't have issues. The only problems you'll encouter I believe are when you make a comment that has a line to itself in between <UnitInfo> & </UnitInfo> of a single entry.

And that means that commenting out an entire entry would be safe, as it would then be securely in-between the entry before and after it.

You explained it correct! hope the guy being biten understands with your explanation that I didn't meant to bite him, but just trying to be helpfull...as foreigner, don't pick me on my words...

EDIT2: WoC: Have you guys noticed/Solved the issue with Vista Users not being able to use Modular Loading? Many people have run into instant CtD if they have modular loading enabled, but when they disable it they notice that SOME of the modular items load into the game (but not all, and thus rather buggy)

Both issues have been adressed yes
first is an error with msxml3.dll crashing(just not always telling you why it crashes), i debugged it along time ago and you should know that an updated(not yet by MS released msxml3.dll from Microsoft) is available in the prerelease of the service packs for both windows XP and windows vista which should include it(if not, you can grab it from the woc website)

and secondly, can't write any info about the firaxis patch because of the NDA, but you can be optimistic! that's all i can say


the other issue from you, that not all is loaded, is why we started WoC,
WoC was started by Impaler[WrG]+ViSa Modpack Team..Since Impaler went to Firaxis to work, he can't continue his work, but therefore many others have joined WoC by now..feel free to join as well and work together with us(or by using our stuff)
also, WoC doesn't do Python mods...because we experienced that Python slows down the game to much, everything we do is done SDK(except User Interface like popups, buttons, etc) but all next turn stuff, calculations, etc is done in the SDK

 

[woc]

@xienwolf: Thanks for the help, sorry for partly hijacking your thread to discuss comments. FWIW, in my original file, the offending comment is between a </PromotionInfo> and the next <PromotionInfo>. Maybe I am confused about the difference between an element and an entry. I will just avoid comments on lines by themselves except at the start of the file.

actually, that shouldn't crash..at least never crashed on me..but then again, can't remember the list time i commented an XML instead of the corresponding schema file

 

[davidlallen]

@woc: I understand the difference between putting one comment in a schema vs dozens of comments for each instance of the tag. If that is what you meant, great. I was responding to:

generally, never add comments in the xml files themself..they are BS and not needed, anyone can use winmerge if he feels the need to know what's different on a file, thats more professional then commenting a file like crazy

My original problem was that I added a few PromotionInfo to the standard CIV4PromotionInfos.xml, along with one comment line to indicate where I had made a change. Then removing the comment changed the behavior of the file.

 

[woc]

@woc: I understand the difference between putting one comment in a schema vs dozens of comments for each instance of the tag. If that is what you meant, great. I was responding to:



My original problem was that I added a few PromotionInfo to the standard CIV4PromotionInfos.xml, along with one comment line to indicate where I had made a change. Then removing the comment changed the behavior of the file.

ah i c, now i got it no, actually that shouldn't crash...at least never happened to me..put the file online, i start a game with it and debug it, see what happens

EDIT, if you don't wanna upload your file, i can only give you some hints where to debug

the loop, checks for a next non comment sibbling, and uses this method:

bool CvXMLLoadUtility::SkipToNextVal()
{
	// we will loop through and skip over any comment nodes
	while (gDLL->getXMLIFace()->IsLastLocatedNodeCommentNode(m_pFXml))
	{
		// if we cannot set the current xml node to it's next sibling then we will break out of the for loop
		// otherwise we will continue looping
		if (!gDLL->getXMLIFace()->NextSibling(m_pFXml))
		{
			return false;	// couldn't find any non-comment nodes
		}
	}
	return true;
}

thats in te cvxmlloadutility.cpp.. put a break on it wherever you assume it doesn't skip the comment properly,

this is called from the
cvxmlloadutilityset.cpp
in the CvXMLLoadUtility::SetGlobalClassInfo method
can easily be found, loop looks like:

// if we successfully locate the tag name in the xml file
	if (gDLL->getXMLIFace()->LocateNode(m_pFXml, szTagName))
	{
		// loop through each tag
		do
		{
			SkipToNextVal();	// skip to the next non-comment node

won't put the whole code online, but this should be enough to point you where your code hgangs on your file

 

[davidlallen]

@ woc: Thanks for the SDK reference, but I haven't gotten into SDK stuff yet. Please see my original post here for what I was trying to solve, and the example file. By removing the comment, and fixing one other reference, I was able to solve my problem.

 

[Impaler[WrG]]

In normal BTS Modular loading the multiple 'passes' are only per file. So the game can make sense of one xml file full of dependent references at a time. Promotions, Technologies, Buildings and a few other data types are the only ones affected by this limitation because they have pre-requisites of the same data type. If a chain of dependent items A -> B -> C are all on the same xml file the multiple read passes correctly index and assign everything but if their on different files theirs the potential for them to load in an unpredictable sequence (often alphabetic) resulting in dangling references.

I recommend either that one bundle together the moduals into "lines" of promotions ex Heal I, Heal II, Heal III if your staying with default BTS or to use the WoC Dll.

 

[xienwolf]

It is sounding like this WoC is a good thing to join in on I'll check out what exactly Jabber is sometime tomorrow and see if I can find you guys to discuss things you have done, and things I intend to do I'd absolutely adore to help create a single DLL which covers every mod (well, at least all the ones which I enjoy ).

 

[Tarquelne]

Whoops, wrong thread.