C2C - mass XML changes parser

[ls612]

The XML is not read by the engine at all, only by the DLL. The engine reads it only as entity properties from APIs exposed by the DLL. As ls612 says, we could entirely replace XML storage of assets with SQL at runtime. However, we'd still need export and import to/from SQL to some serialized format for installation and upgrade, so practically we'd probably be exporting to/from some flat format (and XML is as good as any) which would relegate the SQL to a runtime store of the asset definition. IMO it's too much effort to justify just for that. If it also stored game state (so units etc.) as databases, and allowed runtime access to the gamestate via SQL (so it doesn't have to all be memory resident) then it might be justified (and I think that's what drove it for Civ V). However, if game states became database instances we'd then have compatibility issues since we'd have to be able to load them across schema changes, which means we'd AGAIN need to serialize to/from some other format (like current save compatibility format), which would be a bit of a nightmare (Civ V doesn't allow games to be saved in one schema and loaded in another I presume?? [i.e. - doesn't have the equivalent of our compatibility save capability])

The way civ 5 works is that on first run it reads the XML files from the core directory (ie, vanilla) and does two things. The first thing it does is convert all XML entries into an equivalent SQL statement (so a new entry in the Units XML would be converted into INSERT INTO Units ...) and then it creates a database file on disk (essentially acting as a cache for future loads). It then caches the database into memory (essentially what we do with XML now, using infos classes). So in Civ 5 adding a 'new tag' is actually adding and populating a new column to an SQL table and then adding supporting code for that in the DLL.

In civ 5 then mods modify the vanilla database, either through raw SQL statements like my example above or through XML files which mimic SQL statements (probably this was added so civ 4 modders could get into it more easily). That wouldn't be necessary for C2C as we just junk the base files entirely and create our own data cache on load from scratch. The aforementioned functionality to use SQL statements to modify 'tables' of data (and a table would be in C2C's case a first level XML tag, so UnitInfos, UnitClassInfos, etc) is I think a more elegant solution to what Nimek is trying to do with his parser, and uses easier syntax than XSLT, which would also work and do basically the same thing. It might also be able to reuse some of AIAndy's caching code, although I'm not sure about it.

But I can say that from working on CiV that full SQL support is the best thing (aside from the underlying engine) that Civ 5 has for modding that civ 4 doesn't. Seriously, it is like night and day for asset creation.

 

[n47]

@ls612, how does it deal with nested tags? Like our UnitAIs? The special UnitAIs table + joining? Editing this looks like a little nightmare without queries. Especially, for our <TrainCondition>, which can have whatever depth we like. Of course I may be wrong, but isn't SQL so nice for you, because of a flat structure, that we lack?

 

[ls612]

@ls612, how does it deal with nested tags? Like our UnitAIs? The special UnitAIs table + joining? Editing this looks like a little nightmare without queries. Especially, for our <TrainCondition>, which can have whatever depth we like. Of course I may be wrong, but isn't SQL so nice for you, because of a flat structure, that we lack?

Each nested table gets it's own table, like so.

	<Unit_Flavors>
		<Row>
			<UnitType>UNIT_SLAVE_MELEE</UnitType>
			<FlavorType>FLAVOR_OFFENSE</FlavorType>
			<Flavor>3</Flavor>
		</Row>
		<Row>
			<UnitType>UNIT_SLAVE_MELEE</UnitType>
			<FlavorType>FLAVOR_DEFENSE</FlavorType>
			<Flavor>1</Flavor>
		</Row>
		<Row>
			<UnitType>UNIT_SLAVE_ARCHER</UnitType>
			<FlavorType>FLAVOR_DEFENSE</FlavorType>
			<Flavor>4</Flavor>
		</Row>
		<Row>
			<UnitType>UNIT_SLAVE_WORKER</UnitType>
			<FlavorType>FLAVOR_TILE_IMPROVEMENT</FlavorType>
			<Flavor>20</Flavor>
		</Row>
	</Unit_Flavors>

This is XML from my mod, you can clearly see the mapping onto SQL syntax in the tag names but that naming convention isn't strictly necessary.

Meaning that you have the Units table, the Units_DomainCombatModifiers table, a Unit_Flavors table and so forth, each the equivalent of a top level XML tag in civ 4. I'm guessing that with some creative use of joining that civ 4 XML could be translated to a similar underlying db format automatically by the DLL at load.

 

[n47]

I'm guessing that with some creative use of joining that civ 4 XML could be translated to a similar underlying db format automatically by the DLL at load.

No. <TrainCondition> can not. Or at least, the translation won't be trivial.

For example

                  <UnitInfo>
                        <Type>UNIT_SEAPLANE</Type>
                        <TrainCondition>
                                <Or>
				<And>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_A</ID>
					</Has>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_B</ID>
					</Has>
				</And>
				<And>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_C</ID>
					</Has>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_D</ID>
					</Has>
				</And>
                                </Or>
			</TrainCondition>
                        ...
                  </UnitInfo>

would translate to

                  <UnitInfos>
                        <row>
                                   <Type>UNIT_SEAPLANE</Type>
                                   <TrainCondition>123</TrainCondition>
                                   ...
                        </row>
                  </UnitInfo>

                  <TrainConditions>
                        <row>
                                   <id>123</id>
                                   <condition_type>Or</condition_type>
                                   <operand1>124</operand1>
                                   <operand2>125</operand2>
                        <row/>
                        <row>
                                   <id>124</id>
                                   <condition_type>And</condition_type>
                                   <operand1>126</operand1>
                                   <operand2>127</operand2>
                        <row/>
                        <row>
                                   <id>125</id>
                                   <condition_type>And</condition_type>
                                   <operand1>128</operand1>
                                   <operand2>129</operand2>
                        <row/>
                        <row>
                                   <id>126</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>321</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>127</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>322</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>128</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>323</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>129</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>323</operand1>
                                   <operand2/>
                        <row/>
                  </TrainConditions>

                  <Hases>
                        <row>
                                   <id>321</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_A</_ID>
                        </row>
                        <row>
                                   <id>322</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_B</_ID>
                        </row>
                        <row>
                                   <id>323</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_C</_ID>
                        </row>
                        <row>
                                   <id>324</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_D</_ID>
                        </row>
		</Hases>

I can imagine this ecstatic mass of modders after we would implement it.

I've thought, the case you described would be simpler, but still need to use ids like here. But yes, using types as keys rescue this case. Here however you can't use anything nice as a key in this case.

Welcome to the world of recursive structures.

 

[Koshling]

No. <TrainCondition> can not. Or at least, the translation won't be trivial.

For example

                  <UnitInfo>
                        <Type>UNIT_SEAPLANE</Type>
                        <TrainCondition>
                                <Or>
				<And>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_A</ID>
					</Has>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_B</ID>
					</Has>
				</And>
				<And>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_C</ID>
					</Has>
					<Has>
						<GOMType>GOM_BONUS</GOMType>
						<ID>BONUS_D</ID>
					</Has>
				</And>
                                </Or>
			</TrainCondition>
                        ...
                  </UnitInfo>

would translate to

                  <UnitInfos>
                        <row>
                                   <Type>UNIT_SEAPLANE</Type>
                                   <TrainCondition>123</TrainCondition>
                                   ...
                        </row>
                  </UnitInfo>

                  <TrainConditions>
                        <row>
                                   <id>123</id>
                                   <condition_type>Or</condition_type>
                                   <operand1>124</operand1>
                                   <operand2>125</operand2>
                        <row/>
                        <row>
                                   <id>124</id>
                                   <condition_type>And</condition_type>
                                   <operand1>126</operand1>
                                   <operand2>127</operand2>
                        <row/>
                        <row>
                                   <id>125</id>
                                   <condition_type>And</condition_type>
                                   <operand1>128</operand1>
                                   <operand2>129</operand2>
                        <row/>
                        <row>
                                   <id>126</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>321</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>127</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>322</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>128</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>323</operand1>
                                   <operand2/>
                        <row/>
                        <row>
                                   <id>129</id>
                                   <condition_type>Has</condition_type>
                                   <operand1>323</operand1>
                                   <operand2/>
                        <row/>
                  </TrainConditions>

                  <Hases>
                        <row>
                                   <id>321</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_A</_ID>
                        </row>
                        <row>
                                   <id>322</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_B</_ID>
                        </row>
                        <row>
                                   <id>323</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_C</_ID>
                        </row>
                        <row>
                                   <id>324</id>
                                   <GOMType>GOM_BONUS</GOM_BONUS>
                                   <_ID>BONUS_D</_ID>
                        </row>
		</Hases>

I can imagine this ecstatic mass of modders after we would implement it.

I've thought, the case you described would be simpler, but still need to use ids like here. But yes, using types as keys rescue this case. Here however you can't use anything nice as a key in this case.

Welcome to the world of recursive structures.

You can better handle that sort of thing by building dynamic functional expressions that act as constraints into the engine, so that instead of specifying the condition in sub tables you declare constraint functions, providing a small built in set of composable functions in the database implementation (which would include logical conjunctions etc.). There was some discussion of this in the AXXXE blog and thread. Of course, it's not SQL anymore, it's a database structure that is no longer strictly relational, so you lose the benefits of SQL as a manipulation and query language (or at least you'd have to implement it in a somewhat different environment, which may or may not be readily possible)

 

[n47]

You can better handle that sort of thing by building dynamic functional expressions that act as constraints into the engine, so that instead of specifying the condition in sub tables you declare constraint functions, providing a small built in set of composable functions in the database implementation (which would include logical conjunctions etc.). There was some discussion of this in the AXXXE blog and thread. Of course, it's not SQL anymore, it's a database structure that is no longer strictly relational, so you lose the benefits of SQL as a manipulation and query language (or at least you'd have to implement it in a somewhat different environment, which may or may not be readily possible)

Can you provide a link? But are you aware sqlite do not have user-definable functions?

And yes, I can think of some workarounds, but I feel like they would be a greater mess, then this.

 

[Nimek]

Can someone fix non utf-8 char in CIV4UnitInfos.xml line 161306 or close to that

<UniqueName>Sandra Day O'Connor</UniqueName> sign ' is not coded in proper utf-8 so my parser give errors

New improvements

• ability to search via simple nested tags like subcombattypes
• option to choose file from server
• speed ups improvement in building search fields (stored in session now after creation)

 

[Koshling]

Can you provide a link? But are you aware sqlite do not have user-definable functions?

And yes, I can think of some workarounds, but I feel like they would be a greater mess, then this.

See the 'Core Data Model' section of the blog: http://axxxxe.blogspot.com/. There is also a fair bit of discussion of this on the AXXXE thread on these boards, but it's scattered so I can't give you a specific link.

 

[Nimek]

http://hiarte.hostings.pl/civ4/

New changes

• selected displayed columns are stored in seesipn now so system remember what columns you want to display.
• add checkbox of search select list creation (on left at header name)
• some speedups

Now most searches takes aprox 0.2 sec (in over 700 records file). I not count search box creations time because after created once they are store in session.

I need tester who want to look over its xml file i do some inline editing if needed.
@Hydro maybe you?

 

[n47]

See the 'Core Data Model' section of the blog: http://axxxxe.blogspot.com/. There is also a fair bit of discussion of this on the AXXXE thread on these boards, but it's scattered so I can't give you a specific link.

I probably don't get the picture, as I have a feeling those functions are only a layer and you still need to store the conditions in tables. Just have some comfortable functions to manipulate them. If I am wrong, can you bring some small example?

 

[Koshling]

I probably don't get the picture, as I have a feeling those functions are only a layer and you still need to store the conditions in tables. Just have some comfortable functions to manipulate them. If I am wrong, can you bring some small example?

No, you're not wrong, but you'd manipulate via the layer above, so we'd have some sort of more specific or extended query language that dynamically took account of such constraints/dynamically calculated values is all

 

[Nimek]

New updates

• added table footer with sum of all displayed values

This sum will be usefull for balance reasons I think.

Did you know that all crime related buildings will cause 910 revolution instability per turn.

 

[Nimek]

New updates

• added ability to read/edit TrainCondition Tags
• added ability to read/edit Outcomes tag (including Yelds)
• added abililty to edit inline empty values on non nested tags
• added checkboxes column on left with ability to select deselect all and textfield in non nested tags headers as preparations for mass changes (they not doing anything at now)
• added slide up/down div to show Displayed Columns Configuration
• changed displayed columns labels to alphabetical order in columns

With all this changes this tool is quite powerfull and handy to read xml files and do some small inline tweaks.

 

[Thunderbrd]

You're kickin' ass here Nimek! Keep it up! I'll try to have some units combat classes fully defined for you by the end of the weekend (with any hope all of them but unlikely.)

 

[Nimek]

Another updates

• KICKING ASS SPEED UPS - two times faster on my local machine and even 10X faster on hosted site (I bet because today weekend is started).
• add function to add masss changes with respect currently displayed order (for future use).
• add function that recognizes in "inteligent" way what type of field xml tag is (string, numeric, unique, nested) for future use.

Now search takes even 0.02 sec. There exists some places to make it even faster but for now it is ok. So next speed ups after mass chnages support.

 

[n47]

@Nimek, I beg you for mercy. If you can't stop doing something which is totally out of the web scope with tools signed for the web use, and can't use tools which were designed for this, then at least have some sense of reason about what you are doing with what you have. There is no difference from the user point of view, will search take 0.2 or 0.02 sec. If you would like to perform more then one search per one user operation, then you could care about it. But there is no need for something like this here.

@Koshling, can you seriously consider switching to SQL or some other data storage approach and set some final decision? There can be loss of effort of some modders, if this will delay. I would say, there can be some benefit of resigning from XML, IMO. For example, SQlite may be used as a storage approach for unused data, which can be load on request. This I can't see with XML and parsing it per each request. -- Probably data loaded form XML are small enough to not care about holding all of them in the memory. But that is why I asked about some consideration here. To check is there really nothing serious we can get.

 

[Nimek]

@n47
I know that maybe now this is no big difference (from user perspective). But note that i tested it on 700 records unit file. But if there will be 7000 than it is a difference.

I also love to write effective code I inherited it from Koshling.

And you are very motivating to work

You said: php is slow to read XML. - look how fast it is now
You said: does your XML parser will be able to read TrainCondition and other heavy nested tags - now it is
You said: does it can to add +2 or +20% to each selected item - work in progress - it will

I love you girl your talking is much more motivating than money

 

[Koshling]

@Nimek, I beg you for mercy. If you can't stop doing something which is totally out of the web scope with tools signed for the web use, and can't use tools which were designed for this, then at least have some sense of reason about what you are doing with what you have. There is no difference from the user point of view, will search take 0.2 or 0.02 sec. If you would like to perform more then one search per one user operation, then you could care about it. But there is no need for something like this here.

@Koshling, can you seriously consider switching to SQL or some other data storage approach and set some final decision? There can be loss of effort of some modders, if this will delay. I would say, there can be some benefit of resigning from XML, IMO. For example, SQlite may be used as a storage approach for unused data, which can be load on request. This I can't see with XML and parsing it per each request. -- Probably data loaded form XML are small enough to not care about holding all of them in the memory. But that is why I asked about some consideration here. To check is there really nothing serious we can get.

I have no current intention to change c2c away from XML (persisted) to in-memory (runtime) as it has always been.

If we ever moved away from civ to a totally new base like the AXXXE project, then SQL would be a very strong contender. While we're on the civ base however, I don't think such a change is likely to be worth the effort and disruption.

 

[Nimek]

New update

• added function that checks if xml tag had all unique values (for future use).
• search box generation depends on realType tag so it will generate textfields for tags that stores unique or almost unique values and for numeric type tags.
• support for numeric operators for numeric fileds

You will notice that after creation searchbox for columns sometimes you will get textfield and sometimes select depending on field real type.
With operators you can also search ex units > x by typing ">x"

 

[Thunderbrd]

On my CC planning document, I have input all CCs for the A-Bomb unit. You wanted a test case so there you have it.

Still working overtime on trying to get the rest of the document fleshed out. May not seem like much has been done but trust and believe its been a LOT! More to come still. I'll rearrange the order of the unitcombat categories later and let you know once its ready for the full input project.