My To Do List

[alberts8]

I do it like the others so here is my To Do List.

For now i work on the loading Process(Xml, Cache...) and try to help with the bughunting.

Working on right now

• Changing the XmlParser used by the dll to Xerces-C. It is faster but not as fast as RapidXml and instead more reliable Xml Parser.
  • Merging n47's work from a month ago into the current svn as a starting point. Done!
  • Fix the most errors to get the game running again. Done!
  • Optimize it and get it fully working.Done!
  • Better detection of missplaced Xml Elements in the loading code to avoid not working things and crashes.
  • Optimize the Xml loading Code in the Info Classes to reduce the size of the Code.Done for now!
  • Document the changes for other Dll Modders.
  • It's a big change for a new guy so i post more about it later.

Possible Changes and Proposals

• If possible and approved remove emty or default Tags from the XmlFiles.
  Pros
  • Reduces the size of the files and makes them easier to edit.
  • It's faster to load them into the game.
• If possible with the new XmlParser and needed get rid of all the Schema files in the Modules and use the same Schema for all files with the same InfoType. Schemas could be in the Xml directory.

 

[Thunderbrd]

You've got the merge done successfully already? NICE job! You're proving to be an incredible resource for the team Alberts! Thank you!

 

[Faustmouse]

Just curious: Is the WB issue fixed?

Edit:
I just saw you fixed a potential overflow with culture. I had an issue with overflowing science some time ago. I still have a savegame but it might be outdated, if you need one. Could you have a look at this, too?

 

[alberts8]

You've got the merge done successfully already? NICE job! You're proving to be an incredible resource for the team Alberts! Thank you!

The merging was the easy part the real work just started.

Just curious: Is the WB issue fixed?

Edit:
I just saw you fixed a potential overflow with culture. I had an issue with overflowing science some time ago. I still have a savegame but it might be outdated, if you need one. Could you have a look at this, too?

The WB was fixed a few days ago.

There are alot more possible overflows i just fixed that one while looking for something else. There are lines like that look at the 100 *. So if the intmax is 2147483647 everything over 21474836 is to much. Then you instead divide the ...Times100 by 100 you can go up to 2147483647. I don't know why it is done like this but i think a int division is a bit slower.

if (pCity->getCultureTimes100(getID()) >=[COLOR="Red"] 100 *[/COLOR] GC.getGameINLINE().getCultureThreshold((CultureLevelTypes)(GC.getNumCultureLevelInfos() - 1)))

if (pCity->getCultureTimes100(getID()) [COLOR="Red"]/ 100[/COLOR] >= GC.getGameINLINE().getCultureThreshold((CultureLevelTypes)(GC.getNumCultureLevelInfos() - 1)))

 

[Faustmouse]

Nice to hear the WB is fixed now

As for research, I am not sure if the overflow happened because of global overflow or it was just a city. The game was an OCC and I wanted a slave based economy so I set capture chance to 90% with Banditry and removed the negatives from slaves. Right before the TH era, the overflow occurs.

Long story short: Since the Technology costs in the galactic era explode, I think we need to find a way to not only stop the overflow but set a higher value to be the maximum in the long run. I'm afraid this is true for other commerces, yields etc as well

 

[Thunderbrd]

The merging was the easy part the real work just started.


The WB was fixed a few days ago.

There are alot more possible overflows i just fixed that one while looking for something else. There are lines like that look at the 100 *. So if the intmax is 2147483647 everything over 21474836 is to much. Then you instead divide the ...Times100 by 100 you can go up to 2147483647. I don't know why it is done like this but i think a int division is a bit slower.

if (pCity->getCultureTimes100(getID()) >=[COLOR="Red"] 100 *[/COLOR] GC.getGameINLINE().getCultureThreshold((CultureLevelTypes)(GC.getNumCultureLevelInfos() - 1)))

if (pCity->getCultureTimes100(getID()) [COLOR="Red"]/ 100[/COLOR] >= GC.getGameINLINE().getCultureThreshold((CultureLevelTypes)(GC.getNumCultureLevelInfos() - 1)))

True but aren't you then negating the whole point of the decimalization in the first place by rounding it out every time the value is sent through such a process?

 

[platyping]

I asked this before and I ask again.
What is the point of using the time100 value in this situation?
You waste time to do an extra calculation.
You risk going over int limit.
Just use the normal culture value here...

 

[alberts8]

Nice to hear the WB is fixed now

As for research, I am not sure if the overflow happened because of global overflow or it was just a city. The game was an OCC and I wanted a slave based economy so I set capture chance to 90% with Banditry and removed the negatives from slaves. Right before the TH era, the overflow occurs.

Long story short: Since the Technology costs in the galactic era explode, I think we need to find a way to not only stop the overflow but set a higher value to be the maximum in the long run. I'm afraid this is true for other commerces, yields etc as well

Yes this is something that has to be done at some point.

True but aren't you then negating the whole point of the decimalization in the first place by rounding it out every time the value is sent through such a process?

Yes you are right that is that is the answer for "I don't know why it is done like this". For now i commented all CultureLevel's with a iThreshold over 21474836 because they make no sense. Even that is to much because there are funktions in the game like this one.

int CvCity::countTotalCultureTimes100() const
{
	int iTotalCulture;
	int iI;

	iTotalCulture = 0;

	for (iI = 0; iI < MAX_PLAYERS; iI++)
	{
		if (GET_PLAYER((PlayerTypes)iI).isAlive())
		{
			iTotalCulture += getCultureTimes100((PlayerTypes)iI);
		}
	}

	return iTotalCulture;
}

This means that at game level the maxCultureTimes100 is 2147483647. To solve this we could if possible change the Type of all the different Times100. Also those Levels don't go below zero so a normal int is not necessary we could use a unsigned int this would double the maximum.

 

[Thunderbrd]

I don't think it's a bad idea to incorporate a little of both Platyping's statements and yours. We'd have to consider the role of the calculation a bit more carefully. A grand total like that one... who cares about a decimal right? But at the same time we should be giving it the MAXIMUM possible berth so an unsigned long long might be in order as it was for the leaderhead level processing.

But if we're talking about a small calculation within a city then the decimal may mean something so we should take steps to preserve the decimalization there. And if we're talking any greater a scale, even final city totals, rounding becomes very acceptable, though perhaps a second number to track remainders like we would do with division problems when first learning division would be cool to enable a fractional compiling system that gives us a full two more digits in our equations..

 

[Faustmouse]

The Decimal is only important in the very early state of a new city. When you start a game you usually don't get more then 0.8 per turn. If you round down this then your city won't grow at all.

But once you hit like 100 per turn, who cares about the decimal? Is it possible / does it make sense to change the equation midgame? You could say: before Modern Era use the Decimal Equation, after that use the other one.

 

[Azurian]

Can we make the limit 42 million? I can work with that.. So it becomes /50

 

[platyping]

It is still /100.

What he meant is that currently these values are stored as integers, which goes from -2.1b to 2.1b.
But who cares about the negative side?

So using unsigned integer, it becomes 0 to 4.2b.
So Culture value is still Time100 value /100, not /50

 

[alberts8]

Little update about my current project.

I'am a bit stuck in the moment because then i compare the three Cache's created by the current svn (Both Parsers MSXML or RapidXml) and my modified one using the Xerces parser there are differences between all three.
Some of them come from different Whitespace or Comment handling inside of Xml Element's.
Or for example the the Msxml cannot read one of the LeaderHeadInfos completely.
Some might be caused by me, n47 or during merging her changes and others come from small errors in the current svn.
For some differences it is hard to tell what causes them and if they even make any difference in the game so this is very time consuming.
But after all the new parser works great so far and the Debug Dll runs better without Msxml i had no more crashes.

 

[Thunderbrd]

Great news from your update I believe. Is there any way we can make whitespace not matter in the least bit in the xml? I really shouldn't even though I'd almost prefer it if we were enforced into being a bit more organized. Comment line handling has always been annoying so I'm hoping we CAN have comment lines that don't cause problems after all's said and done. Very helpful to use them at times.

 

[strategyonly]

Little update about my current project.

I'am a bit stuck in the moment because then i compare the three Cache's created by the current svn (Both Parsers MSXML or RapidXml) and my modified one using the Xerces parser there are differences between all three.
Some of them come from different Whitespace or Comment handling inside of Xml Element's.
Or for example the the Msxml cannot read one of the LeaderHeadInfos completely.
Some might be caused by me, n47 or during merging her changes and others come from small errors in the current svn.
For some differences it is hard to tell what causes them and if they even make any difference in the game so this is very time consuming.
But after all the new parser works great so far and the Debug Dll runs better without Msxml i had no more crashes.

I believe there is ALOT of whitespace in the TEXT areas of most modular stuff, someone changed something about 4 months ago, and it left all kinds of spacing and ending of sentences only with 5 or 6 words then going to the next line for some odd reason.

 

[alberts8]

If nobody has anything against it i'am going to commit the first step of the parser change sometime this weekend. The other steps follow in the days after.

• The first step includes the new parser and some other small changes i made in the dll.
  • We have a additional dll. This dll must be somewhere the Bts.exe can find it. There are some options how to do this.
    • Copy the dll to the Bts directory.
    • Delayed loading of the dll would be the easy way but we can't do this with xerces so this is not an option.
    • Put the dll in the assets dir and have a small exe or bat to add the assets directory to the PATH Environment Variable and then start the game.
    • Or include the xerces code in the dll the way it is done with Boost.Thread but i have to try if is this is possible.
  • This changes alot in the classes CvInfos, CvUnitInfos, CvBuildingInfos and other's with xml reading code.
  • I have to change some xml files but i post more about this later.
  • The dll will include xsd schema validation but it is not active because it takes time to create the new xsd schema's and i want to make them as good as possible.
  • I can also reformat all the xml files and convert them to Utf8 if you want me to.
• The second step is to fix errors that come from step one.
  
• The third step is to complete schema validation.
  
• The forth step is more error fixing and code cleanup.

 

[Azurian]

[/LIST]

[*]Put the dll in the assets dir and have a small exe or bat to add the assets directory to the PATH Environment Variable and then start the game.

[*]I can also reformat all the xml files and convert them to Utf8 if you want me to.

I would recommend the first option. I would make it bat.

How long would it take to reformat all files to Utf8? You can ask Strategyonly to do a FREEZE which means no new code can be added while you do a reformat.

 

[strategyonly]

If nobody has anything against it i'am going to commit the first step of the parser change sometime this weekend. The other steps follow in the days after.

Spoiler :

• The first step includes the new parser and some other small changes i made in the dll.
  • We have a additional dll. This dll must be somewhere the Bts.exe can find it. There are some options how to do this.
    • Copy the dll to the Bts directory.
    • Delayed loading of the dll would be the easy way but we can't do this with xerces so this is not an option.
    • Put the dll in the assets dir and have a small exe or bat to add the assets directory to the PATH Environment Variable and then start the game.
    • Or include the xerces code in the dll the way it is done with Boost.Thread but i have to try if is this is possible.
  • This changes alot in the classes CvInfos, CvUnitInfos, CvBuildingInfos and other's with xml reading code.
  • I have to change some xml files but i post more about this later.
  • The dll will include xsd schema validation but it is not active because it takes time to create the new xsd schema's and i want to make them as good as possible.
  • I can also reformat all the xml files and convert them to Utf8 if you want me to.

• The second step is to fix errors that come from step one.
  
• The third step is to complete schema validation.
  
• The forth step is more error fixing and code cleanup.

One thing WE must never do is change ANY of the original files at all, infact NEVER. But i am sure you akready know that.

So we have to make sure what we change is ONLY in C2C modmod changes.

 

[Dancing Hoskuld]

A BAT may be useful anyway because we may eventually want a set up style program with all the C2C options explained to be run first.

 

[alberts8]

A BAT may be useful anyway because we may eventually want a set up style program with all the C2C options explained to be run first.

I could also make a small programm for that but this takes time. The texts for it can come from xml or another text file.

One thing WE must never do is change ANY of the original files at all, infact NEVER. But i am sure you akready know that.

So we have to make sure what we change is ONLY in C2C modmod changes.

Yes i know.

I would recommend the first option. I would make it bat.

How long would it take to reformat all files to Utf8? You can ask Strategyonly to do a FREEZE which means no new code can be added while you do a reformat.

I can reformat(I mean the Xml Structure) all of them in 2 Seconds and to converting them to Utf8 doesen't take longer but there migth be invalid characters and those must be fixed by hand.