C2C Spreadsheeting Tool

Thunderbrd · Oct 16, 2011

A little while back, Koshling built us an amazing tool that assists us in automatically generating a spreadsheet from xml file data. I'm STILL (due to my own delays and distractions) trying to get this thing to work quite right for myself and as I had some questions for him on the tool, he asked me to start a thread for it since others have had some questions on the matter as well.

This tool is GREAT! And believe you me, as soon as I can get it faithfully up and running and have developed a complete understanding of how to make it work, you'll be soon seeing a database for our mod that includes buildings, technologies, units, etc with all corresponding linking information, a modder's bible that we should be able to fairly easily keep up to date, use for pulling data queries, pull together game balance metrics, and utilize for ease in modding, as well as showing us what tools already exist in the xml that we may be overlooking or under-utilizing.

So this is an introduction to what we're talking about.

Now, onto some questions...

Thunderbrd · Oct 16, 2011

For those moments when we have a simple embedded list such as:

Code:

			<TechTypes>
				<PrereqTech>TECH_SHELTER_BUILDING</PrereqTech>
				<PrereqTech>TECH_BARTER</PrereqTech>
			</TechTypes>

we use:

Code:

	<ComplexMapping sourceTag="TechTypes" outColumn="PrereqTechs">
		<dataElement sourceTag="PrereqTech" />
	</ComplexMapping>

in the template to pull the list into one field in the spreadsheed labeled PrereqTechs.

However, what are we supposed to do when we have a list of embedded fields within embedded fields as in the following?

Code:

			<ReligionChanges>
				<ReligionChange>
					<ReligionType>RELIGION_ISLAM</ReligionType>
					<iReligionChange>3</iReligionChange>
				</ReligionChange>
				<ReligionChange>
					<ReligionType>RELIGION_CHRISTIAN</ReligionType>
					<iReligionChange>3</iReligionChange>
				</ReligionChange>
			</ReligionChanges>

Dancing Hoskuld · Oct 16, 2011

So will you be adding some documentation on how to use it to the Modder's Documentation thread?

Thunderbrd · Oct 16, 2011

Once I have it sorted out, yes... for both the tool and the access db as well (though it'll require having access I think...)

Koshling · Oct 17, 2011

Thunderbrd said:
For those moments when we have a simple embedded list such as:

Code:

<TechTypes> <PrereqTech>TECH_SHELTER_BUILDING</PrereqTech> <PrereqTech>TECH_BARTER</PrereqTech> </TechTypes>

we use:

Code:

<ComplexMapping sourceTag="TechTypes" outColumn="PrereqTechs"> <dataElement sourceTag="PrereqTech" /> </ComplexMapping>

in the template to pull the list into one field in the spreadsheed labeled PrereqTechs.

However, what are we supposed to do when we have a list of embedded fields within embedded fields as in the following?

Code:

<ReligionChanges> <ReligionChange> <ReligionType>RELIGION_ISLAM</ReligionType> <iReligionChange>3</iReligionChange> </ReligionChange> <ReligionChange> <ReligionType>RELIGION_CHRISTIAN</ReligionType> <iReligionChange>3</iReligionChange> </ReligionChange> </ReligionChanges>

That's what the '<ComplexMapping>' tag does (pulls data out of nested structures). If you run with this template:

Code:

<ScanTemplate sourceTag="BuildingInfos/BuildingInfo">
	<SimpleMapping sourceTag="BuildingClass" outColumn="Class" />
	<SimpleMapping sourceTag="Type" outColumn="Type" />
	<ComplexMapping sourceTag="ReligionChanges" outColumn="ReligionChanges">
		<dataElement sourceTag="ReligionType" />
		<dataElement sourceTag="iReligionChange" />
	</ComplexMapping>
</ScanTemplate>

you'll get columns for the building's class, type and religion changes (whatever they are - I'm not sure exactly what that tag does!). The 'ReligionChanges' column has to hold data-pairs from the two contributing fields for each instance (so per relgion) specified, so it formats as (hope you can read BNF!):

Code:

'(' <datatuple> [,<datatuple>]* ')'
where
datatuple: <fieldVal> [<fieldVal>]*
fieldVal: '[' <fieldName> '=' <fieldValue> ']'

The result of running that template on the assets gives output (columns don't line up very well here - looks better in spreadsheet) like this (subsection of the actual output):

Code:

BUILDINGCLASS_ANDEAN_CATHEDRAL	BUILDING_ANDEAN_CATHEDRAL	([ReligionType=RELIGION_ANDEAN] [iReligionChange=3])
BUILDINGCLASS_ANDEAN_MONASTERY	BUILDING_ANDEAN_MONASTERY	([ReligionType=RELIGION_ANDEAN] [iReligionChange=1])
BUILDINGCLASS_ANDEAN_SHRINE	BUILDING_ANDEAN_SHRINE	([ReligionType=RELIGION_ANDEAN] [iReligionChange=1])
BUILDINGCLASS_ANDEAN_TEMPLE	BUILDING_ANDEAN_TEMPLE	([ReligionType=RELIGION_ANDEAN] [iReligionChange=2])
BUILDINGCLASS_CHOQUEQUIRAO	BUILDING_CHOQUEQUIRAO	([ReligionType=RELIGION_ANDEAN] [iReligionChange=2])
BUILDINGCLASS_DORADO	BUILDING_DORADO	([ReligionType=RELIGION_ANDEAN] [iReligionChange=2])
BUILDINGCLASS_MACHU_PICCHU	BUILDING_MACHU_PICCHU	([ReligionType=RELIGION_ANDEAN] [iReligionChange=2])

If you're not dealign with nested tags you can stick to <SimpleMapping> - the <ComplexMapping> should only be required for nested data where the output column name derives from higher level tag than the actual data instances.

Thunderbrd · Oct 17, 2011

Well, I can't read that language but I get your point. Thanks for the answer! I was just worried that ignoring the <ReligionChange> within <ReligionChanges> might lead to some less than desirable effects. But from what I'm seeing, it seems I just need to let it know what our field is going to be labeled on the basis of, then, if we're using complex mapping, we can assume any of the fields within will be arranged appropriately by just listing off the end result readouts we're looking for.

Thunderbrd · Oct 18, 2011

Ok, seems everything is working and nicely audited to the newest building schemas (assuming Hydro's are the most updated...).

We discussed this before, but the answer was long ago so I'm sorry but I have to ask again: Why would the following files be coming up as unscannable and how do I address the problem?

\CIV4BuildingInfos.xml
\Craft_CIV4BuildingInfos.xml
\Electricity_CIV4BuildingInfos.xml
\Entertainment_CIV4BuildingInfos.xml
\Health_CIV4BuildingInfos.xml
\Housing_CIV4BuildingInfos.xml
\Hunting_CIV4BuildingInfos.xml
\Louvre_CIV4BuildingInfos.xml
\Mesopotamian_CIV4BuildingInfos.xml
\Palace_CIV4BuildingInfos.xml
\Pets_CIV4BuildingInfos.xml
\Science_CIV4BuildingInfos.xml
\Sports_CIV4BuildingInfos.xml
\Transportation_CIV4BuildingInfos.xml
\Tweaks_CIV4BuildingInfos.xml
\UB_CIV4BuildingInfos.xml
\Wonders_CIV4BuildingInfos.xml

I've isolated all the building files into one side file and scanned just that so these are the building info files that don't work while most of them actually do scan just fine.

For your reference, this is my scan template:

Spoiler :

Code:

<ScanTemplate sourceTag="BuildingInfos/BuildingInfo">
	<SimpleMapping sourceTag="BuildingClass" outColumn="Class" />
	<SimpleMapping sourceTag="Type" outColumn="Type" />
	<SimpleMapping sourceTag="SpecialBuildingType" outColumn="SpecialBuildingType" />
	<SimpleMapping sourceTag="Description" outColumn="Description" />
	<SimpleMapping sourceTag="Civilopedia" outColumn="Civilopedia" />
	<SimpleMapping sourceTag="Help" outColumn="Help" />
	<SimpleMapping sourceTag="Strategy" outColumn="Strategy" />
	<SimpleMapping sourceTag="Advisor" outColumn="Advisor" />
	<SimpleMapping sourceTag="ArtDefineTag" outColumn="ArtDefineTag" />
	<SimpleMapping sourceTag="MovieDefineTag" outColumn="MovieDefineTag" />
	<SimpleMapping sourceTag="HolyCity" outColumn="HolyCity" />
	<SimpleMapping sourceTag="ReligionType" outColumn="ReligionType" />
	<SimpleMapping sourceTag="StateReligion" outColumn="StateReligion" />
	<SimpleMapping sourceTag="bStateReligion" outColumn="bStateReligion" />
	<SimpleMapping sourceTag="PrereqReligion" outColumn="PrereqReligion" />
	<SimpleMapping sourceTag="PrereqCorporation" outColumn="PrereqCorporation" />
	<SimpleMapping sourceTag="FoundsCorporation" outColumn="FoundsCorporation" />
	<SimpleMapping sourceTag="GlobalReligionCommerce" outColumn="GlobalReligionCommerce" />
	<SimpleMapping sourceTag="GlobalCorporationCommerce" outColumn="GlobalCorporationCommerce" />
	<SimpleMapping sourceTag="VictoryPrereq" outColumn="VictoryPrereq" />
	<SimpleMapping sourceTag="FreeStartEra" outColumn="FreeStartEra" />
	<SimpleMapping sourceTag="MaxStartEra" outColumn="MaxStartEra" />
	<SimpleMapping sourceTag="ObsoleteTech" outColumn="ObsoleteTech" />
	<SimpleMapping sourceTag="PrereqTech" outColumn="PrimaryTechPrereq" />
	<ComplexMapping sourceTag="TechTypes" outColumn="PrereqTechs">
		<dataElement sourceTag="PrereqTech" />
	</ComplexMapping>
	<SimpleMapping sourceTag="Bonus" outColumn="BonusResource" />
	<ComplexMapping sourceTag="PrereqBonuses" outColumn="PrereqBonuses">
		<dataElement sourceTag="Bonus" />
	</ComplexMapping>
	<ComplexMapping sourceTag="ProductionTraits" outColumn="ProductionTraits">
		<dataElement sourceTag="ProductionTraitType" />
		<dataElement sourceTag="iProductionTrait" />
	</ComplexMapping>
	<ComplexMapping sourceTag="HappinessTraits" outColumn="HappinessTraits">
		<dataElement sourceTag="HappinessTraitType" />
		<dataElement sourceTag="iHappinessTrait" />
	</ComplexMapping>
	<SimpleMapping sourceTag="NoBonus" outColumn="NoBonus" />
	<SimpleMapping sourceTag="PowerBonus" outColumn="PowerBonus" />	
	<SimpleMapping sourceTag="FreeBonus" outColumn="FreeBonus" />
	<SimpleMapping sourceTag="iNumFreeBonuses" outColumn="iNumFreeBonuses" />
	<SimpleMapping sourceTag="FreeBuilding" outColumn="FreeBuilding" />
	<SimpleMapping sourceTag="FreeAreaBuilding" outColumn="FreeAreaBuilding" />
	<SimpleMapping sourceTag="FreeTradeRegionBuilding" outColumn="FreeTradeRegionBuilding" />
	<SimpleMapping sourceTag="bTeamShare" outColumn="bTeamShare" />
	<SimpleMapping sourceTag="bWater" outColumn="bWater" />
	<SimpleMapping sourceTag="bRiver" outColumn="bRiver" />
	<SimpleMapping sourceTag="bFreshWater" outColumn="bFreshWater" />
	<SimpleMapping sourceTag="bPower" outColumn="bPower" />
	<SimpleMapping sourceTag="bDirtyPower" outColumn="bDirtyPower" />
	<SimpleMapping sourceTag="bAreaCleanPower" outColumn="bAreaCleanPower" />
	<SimpleMapping sourceTag="DiploVoteType" outColumn="DiploVoteType" />
	<SimpleMapping sourceTag="bForceTeamVoteEligible" outColumn="bForceTeamVoteEligible" />
	<SimpleMapping sourceTag="bCapital" outColumn="bCapital" />
	<SimpleMapping sourceTag="bGovernmentCenter" outColumn="bGovernmentCenter" />
	<SimpleMapping sourceTag="bGoldenAge" outColumn="bGoldenAge" />
	<SimpleMapping sourceTag="bAllowsNukes" outColumn="bAllowsNukes" />
	<SimpleMapping sourceTag="bMapCentering" outColumn="bMapCentering" />
	<SimpleMapping sourceTag="bNoUnhappiness" outColumn="bNoUnhappiness" />
	<SimpleMapping sourceTag="bNoUnhealthyPopulation" outColumn="bNoUnhealthyPopulation" />
	<SimpleMapping sourceTag="bBuildingOnlyHealthy" outColumn="bBuildingOnlyHealthy" />
	<SimpleMapping sourceTag="bNeverCapture" outColumn="bNeverCapture" />
	<SimpleMapping sourceTag="bNukeImmune" outColumn="bNukeImmune" />
	<SimpleMapping sourceTag="bPrereqReligion" outColumn="bPrereqReligion" />
	<SimpleMapping sourceTag="bCenterInCity" outColumn="bCenterInCity" />
	<SimpleMapping sourceTag="iAIWeight" outColumn="iAIWeight" />
	<SimpleMapping sourceTag="iCost" outColumn="iCost" />
	<SimpleMapping sourceTag="iHurryCostModifier" outColumn="iHurryCostModifier" />
	<SimpleMapping sourceTag="iAdvancedStartCost" outColumn="iAdvancedStartCost" />
	<SimpleMapping sourceTag="iAdvancedStartCostIncrease" outColumn="iAdvancedStartCostIncrease" />
	<SimpleMapping sourceTag="iMinAreaSize" outColumn="iMinAreaSize" />
	<SimpleMapping sourceTag="iConquestProb" outColumn="iConquestProb" />
	<SimpleMapping sourceTag="iCitiesPrereq" outColumn="iCitiesPrereq" />
	<SimpleMapping sourceTag="iTeamsPrereq" outColumn="iTeamsPrereq" />
	<SimpleMapping sourceTag="iLevelPrereq" outColumn="iLevelPrereq" />
	<SimpleMapping sourceTag="iMinLatitude" outColumn="iMinLatitude" />
	<SimpleMapping sourceTag="iMaxLatitude" outColumn="iMaxLatitude" />
	<SimpleMapping sourceTag="iGreatPeopleRateModifier" outColumn="iGreatPeopleRateModifier" />
	<SimpleMapping sourceTag="iGreatGeneralRateModifier" outColumn="iGreatGeneralRateModifier" />
	<SimpleMapping sourceTag="iDomesticGreatGeneralRateModifier" outColumn="iDomesticGreatGeneralRateModifier" />
	<SimpleMapping sourceTag="iGlobalGreatPeopleRateModifier" outColumn="iGlobalGreatPeopleRateModifier" />
	<SimpleMapping sourceTag="iAnarchyModifier" outColumn="iAnarchyModifier" />
	<SimpleMapping sourceTag="iGoldenAgeModifier" outColumn="iGoldenAgeModifier" />
	<SimpleMapping sourceTag="iGlobalHurryModifier" outColumn="iGlobalHurryModifier" />
	<SimpleMapping sourceTag="iExperience" outColumn="Experience" />
	<SimpleMapping sourceTag="iGlobalExperience" outColumn="iGlobalExperience" />
	<SimpleMapping sourceTag="iFoodKept" outColumn="iFoodKept" />
	<SimpleMapping sourceTag="iAirlift" outColumn="iAirlift" />
	<SimpleMapping sourceTag="iAirModifier" outColumn="iAirModifier" />
	<SimpleMapping sourceTag="iAirUnitCapacity" outColumn="iAirUnitCapacity" />
	<SimpleMapping sourceTag="INukeModifier" outColumn="iNukeModifier" />
	<SimpleMapping sourceTag="iNukeExplosionRand" outColumn="iNukeExplosionRand" />
	<SimpleMapping sourceTag="iFreeSpecialist" outColumn="FreeSpecialist" />
	<SimpleMapping sourceTag="iAreaFreeSpecialist" outColumn="iAreaFreeSpecialist" />
	<SimpleMapping sourceTag="iGlobalFreeSpecialist" outColumn="iGlobalFreeSpecialist" />
	<SimpleMapping sourceTag="iMaintenanceModifier" outColumn="iMaintenanceModifier" />
	<SimpleMapping sourceTag="iGlobalMaintenanceModifier" outColumn="iGlobalMaintenanceModifier" />
	<SimpleMapping sourceTag="iAreaMaintenanceModifier" outColumn="iAreaMaintenanceModifier" />
	<SimpleMapping sourceTag="iOtherAreaMaintenanceModifier" outColumn="iOtherAreaMaintenanceModifier" />
	<SimpleMapping sourceTag="iDistanceMaintenanceModifier" outColumn="iDistanceMaintenanceModifier" />
	<SimpleMapping sourceTag="iNumCitiesMaintenanceModifier" outColumn="iNumCitiesMaintenanceModifier" />
	<SimpleMapping sourceTag="iCoastalDistanceMaintenanceModifier" outColumn="iCoastalDistanceMaintenanceModifier" />
	<SimpleMapping sourceTag="iConnectedCityMaintenanceModifier" outColumn="iConnectedCityMaintenanceModifier" />
	<SimpleMapping sourceTag="iWarWearinessModifier" outColumn="iWarWearinessModifier" />
	<SimpleMapping sourceTag="iGlobalWarWearinessModifier" outColumn="iGlobalWarWearinessModifier" />
	<SimpleMapping sourceTag="iEnemyWarWearinessModifier" outColumn="iEnemyWarWearinessModifier" />
	<SimpleMapping sourceTag="iHealRateChange" outColumn="iHealRateChange" />
	<SimpleMapping sourceTag="iHealth" outColumn="iHealth" />
	<SimpleMapping sourceTag="iAreaHealth" outColumn="iAreaHealth" />
	<SimpleMapping sourceTag="iGlobalHealth" outColumn="iGlobalHealth" />
	<SimpleMapping sourceTag="iHappiness" outColumn="iHappiness" />
	<SimpleMapping sourceTag="iAreaHappiness" outColumn="iAreaHappiness" />
	<SimpleMapping sourceTag="iGlobalHappiness" outColumn="iGlobalHappiness" />
	<SimpleMapping sourceTag="iStateReligionHappiness" outColumn="iStateReligionHappiness" />
	<SimpleMapping sourceTag="iWorkerSpeedModifier" outColumn="iWorkerSpeedModifier" />
	<SimpleMapping sourceTag="iMilitaryProductionModifier" outColumn="iMilitaryProductionModifier" />
	<SimpleMapping sourceTag="iSpaceProductionModifier" outColumn="iSpaceProductionModifier" />
	<SimpleMapping sourceTag="iGlobalSpaceProductionModifier" outColumn="iGlobalSpaceProductionModifier" />
	<SimpleMapping sourceTag="iTradeRoutes" outColumn="iTradeRoutes" />
	<SimpleMapping sourceTag="iCoastalTradeRoutes" outColumn="iCoastalTradeRoutes" />
	<SimpleMapping sourceTag="iGlobalTradeRoutes" outColumn="iGlobalTradeRoutes" />
	<SimpleMapping sourceTag="iTradeRouteModifier" outColumn="iTradeRouteModifier" />
	<SimpleMapping sourceTag="iForeignTradeRouteModifier" outColumn="iForeignTradeRouteModifier" />
	<SimpleMapping sourceTag="iGlobalPopulationChange" outColumn="iGlobalPopulationChange" />
	<SimpleMapping sourceTag="iFreeTechs" outColumn="iFreeTechs" />
	<SimpleMapping sourceTag="iDefense" outColumn="iDefense" />
	<SimpleMapping sourceTag="iBombardDefense" outColumn="iBombardDefense" />
	<SimpleMapping sourceTag="iAllCityDefense" outColumn="iAllCityDefense" />
	<SimpleMapping sourceTag="iEspionageDefense" outColumn="iEspionageDefense" />
	<SimpleMapping sourceTag="iUnitUpgradePriceModifier" outColumn="iUnitUpgradePriceModifier" />
	<SimpleMapping sourceTag="iRevIdxLocal" outColumn="iRevIdxLocal" />
	<SimpleMapping sourceTag="iRevIdxNational" outColumn="iRevIdxNational" />
	<SimpleMapping sourceTag="iRevIdxDistanceModifier" outColumn="IRevIdxDistanceModifier" />
	<SimpleMapping sourceTag="iAsset" outColumn="iAsset" />
	<SimpleMapping sourceTag="iPower" outColumn="iPower" />
	<SimpleMapping sourceTag="iPopulationgrowthratepercentage" outColumn="iPopulationgrowthratepercentage" />
	<SimpleMapping sourceTag="iGlobalPopulationgrowthratepercentage" outColumn="iGlobalPopulationgrowthratepercentage" />
	<SimpleMapping sourceTag="iRevolutionIndexModifier" outColumn="iRevolutionIndexModifier" />
	<SimpleMapping sourceTag="iWorldTradeRoutes" outColumn="iWorldTradeRoutes" />
	<SimpleMapping sourceTag="FreePromotion_2" outColumn="FreePromotion_2" />
	<SimpleMapping sourceTag="FreePromotion_3" outColumn="FreePromotion_3" />
	<SimpleMapping sourceTag="PrereqGameOption" outColumn="PrereqGameOption" />
	<SimpleMapping sourceTag="NotGameOption" outColumn="NotGameOption" />
	<SimpleMapping sourceTag="bBuildOnlyOnPeaks" outColumn="bBuildOnlyOnPeaks" />
	<SimpleMapping sourceTag="bApplyFreePromotionOnMove" outColumn="bApplyFreePromotionOnMove" />
	<SimpleMapping sourceTag="bForceNoPrereqScaling" outColumn="bForceNoPrereqScaling" />
	<SimpleMapping sourceTag="bProvidesFreshWater" outColumn="bProvidesFreshWater" />
	<SimpleMapping sourceTag="bForceAllTradeRoutes" outColumn="bForceAllTradeRoutes" />
	<SimpleMapping sourceTag="bPrereqWar" outColumn="bPrereqWar" />
	<SimpleMapping sourceTag="bRequiresActiveCivics" outColumn="bRequiresActiveCivics" />
	<SimpleMapping sourceTag="bPrereqPower" outColumn="bPrereqPower" />
	<SimpleMapping sourceTag="iDarkAgePointsObsolete" outColumn="iDarkAgePointsObsolete" />
	<SimpleMapping sourceTag="iDarkAgePointsWeight" outColumn="iDarkAgePointsWeight" />
	<SimpleMapping sourceTag="iLineOfSight" outColumn="iLineOfSight" />
	<SimpleMapping sourceTag="ProductionContinueBuildingClass" outColumn="ProductionContinueBuildingClass" />
	<SimpleMapping sourceTag="PrereqCultureLevel" outColumn="PrereqCultureLevel" />
	<SimpleMapping sourceTag="iPrereqPopulation" outColumn="iPrereqPopulation" />
	<SimpleMapping sourceTag="iWorkableRadius" outColumn="iWorkableRadius" />
	<SimpleMapping sourceTag="bZoneOfControl" outColumn="bZoneOfControl" />
	<SimpleMapping sourceTag="bProtectedCulture" outColumn="bProtectedCulture" />
	<SimpleMapping sourceTag="iAdjacentDamagePercent" outColumn="iAdjacentDamagePercent" />
	<SimpleMapping sourceTag="iInvasionChance" outColumn="iInvasionChance" />
	<SimpleMapping sourceTag="iNoEntryDefenseLevel" outColumn="iNoEntryDefenseLevel" />
	<SimpleMapping sourceTag="iNumUnitFullHeal" outColumn="iNumUnitFullHeal" />
	<SimpleMapping sourceTag="iOccupationTimeModifier" outColumn="iOccupationTimeModifier" />
	<SimpleMapping sourceTag="ExtendsBuildingClass" outColumn="ExtendsBuildingClass" />
	<SimpleMapping sourceTag="iNumPopulationEmployed" outColumn="iNumPopulationEmployed" />
	<SimpleMapping sourceTag="iHealthPercentPerPopulation" outColumn="iHealthPercentPerPopulation" />
	<SimpleMapping sourceTag="iHappinessPercentPerPopulation" outColumn="iHappinessPercentPerPopulation" />
	<SimpleMapping sourceTag="fVisibilityPriority" outColumn="fVisibilityPriority" />
	<SimpleMapping sourceTag="FreePromotion" outColumn="FreePromotion" />
	<SimpleMapping sourceTag="CivicOption" outColumn="CivicOption" />
	<SimpleMapping sourceTag="GreatPeopleUnitClass" outColumn="GreatPeopleUnitClass" />
	<SimpleMapping sourceTag="iGreatPeopleRateChange" outColumn="iGreatPeopleRateChange" />
	<SimpleMapping sourceTag="iHurryAngerModifier" outColumn="iHurryAngerModifier" />
	<SimpleMapping sourceTag="bBorderObstacle" outColumn="bBorderObstacle" />
	<ComplexMapping sourceTag="SeaPlotYieldChanges" outColumns="Sea Plot Yield 1(food),Sea Plot Yield 2(production),Sea Plot Yield 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="RiverPlotYieldChanges" outColumns="River Plot Yield 1(food),River Plot Yield 2(production),River Plot Yield 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="GlobalSeaPlotYieldChanges" outColumns="Global Sea Plot Yield 1(food),Global Sea Plot Yield 2(production),Global Sea Plot Yield 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="YieldChanges" outColumns="Yield Change 1(food),Yield Change 2(production),Yield Change 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="YieldModifiers" outColumns="Yield Modifier 1(food),Yield Modifier 2(production),Yield Modifier 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="PowerYieldModifiers" outColumns="Power Yield Modifier 1(food),Power Yield Modifier 2(production),Power Yield Modifier 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="AreaYieldModifiers" outColumns="AreaYield Modifier 1(food),Area Yield Modifier 2(production),Area Yield Modifier 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="GlobalYieldModifiers" outColumns="Global Yield Modifier 1(food),Global Yield Modifier 2(production),Global Yield Modifier 3(gold)">
		<dataElement sourceTag="iYield" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceChanges" outColumns="Commerce(Gold),Commerce(Research),Commerce(Culture),Commerce(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="ObsoleteSafeCommerceChanges" outColumns="Obsolete Safe Commerce(Gold),Obsolete Safe Commerce(Research),Obsolete Safe Commerce(Culture),Obsolete Commerce(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceChangeDoubleTimes" outColumns="Commerce Change Double Times(Gold),Commerce Change Double Times(Research),Commerce Change Double Times(Culture),Commerce Change Double Times(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceModifiers" outColumns="Commerce Modifiers(Gold),Commerce Modifiers(Research),Commerce Modifiers(Culture),Commerce Modifiers(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="GlobalCommerceModifiers" outColumns="Global Commerce Modifiers(Gold),Global Commerce Modifiers(Research),Global Commerce Modifiers(Culture),Global Commerce Modifiers(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="SpecialistExtraCommerces" outColumns="Specialist Extra Commerce(Gold),Specialist Extra Commerce(Research),Specialist Extra Commerce(Culture),Specialist Extra Commerce(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="StateReligionCommerces" outColumns="State Religion Commerce(Gold),State Religion Commerce(Research),State Religion Commerce(Culture),State Religion Commerce(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceHappinesses" outColumns="Commerce Happinesses(Gold),Commerce Happinesses(Research),Commerce Happinesses(Culture),Commerce Happinesses(Espionage)">
		<dataElement sourceTag="iCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="ReligionChanges" outColumn="ReligionChanges">
		<dataElement sourceTag="ReligionType" />
		<dataElement sourceTag="iReligionChange" />
	</ComplexMapping>
	<ComplexMapping sourceTag="SpecialistCount" outColumn="SpecialistCount">
		<dataElement sourceTag="SpecialistType" />
		<dataElement sourceTag="iSpecialistCount" />
	</ComplexMapping>
	<ComplexMapping sourceTag="FreeSpecialistCount" outColumn="FreeSpecialistCount">
		<dataElement sourceTag="SpecialistType" />
		<dataElement sourceTag="iFreeSpecialistCount" />
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceFlexibles" outColumns="Commerce Flexible(Gold),Commerce Flexible(Research),Commerce Flexible(Culture),Commerce Flexible(Espionage)">
		<dataElement sourceTag="bCommerce" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="CommerceChangeOriginalOwners" outColumns="Commerce Change Original Owners(Gold),Commerce Change Original Owners(Research),Commerce Change Original Owners(Culture),Commerce Change Original Owners(Espionage)">
		<dataElement sourceTag="bCommerceChangeOriginalOwner" default="0"/>
	</ComplexMapping>
	<SimpleMapping sourceTag="ConstructSound" outColumn="ConstructSound" />
	<ComplexMapping sourceTag="BonusHealthChanges" outColumn="BonusHealthChanges">
		<dataElement sourceTag="BonusType" />
		<dataElement sourceTag="iHealthChange" />
	</ComplexMapping>
	<ComplexMapping sourceTag="BonusHappinessChanges" outColumn="BonusHappinessChanges">
		<dataElement sourceTag="BonusType" />
		<dataElement sourceTag="iHappinessChange" />
	</ComplexMapping>
	<ComplexMapping sourceTag="BonusProductionModifiers" outColumn="BonusProductionModifiers">
		<dataElement sourceTag="BonusType" />
		<dataElement sourceTag="iProductionModifier" />
	</ComplexMapping>
	<ComplexMapping sourceTag="UnitCombatFreeExperiences" outColumn="UnitCombatFreeExperiences">
		<dataElement sourceTag="UnitCombatType" />
		<dataElement sourceTag="iExperience" />
	</ComplexMapping>
	<ComplexMapping sourceTag="DomainFreeExperiences" outColumn="DomainFreeExperiences">
		<dataElement sourceTag="DomainType" />
		<dataElement sourceTag="iExperience" />
	</ComplexMapping>
	<ComplexMapping sourceTag="DomainProductionModifiers" outColumn="DomainProductionModifiers">
		<dataElement sourceTag="DomainType" />
		<dataElement sourceTag="iProductionModifier" />
	</ComplexMapping>
	<ComplexMapping sourceTag="BuildingHappinessChanges" outColumn="BuildingHappinessChanges">
		<dataElement sourceTag="BuildingType" />
		<dataElement sourceTag="iHappinessChange" />
	</ComplexMapping>
	<ComplexMapping sourceTag="PrereqBuildingClasses" outColumn="PrereqBuildingClasses">
		<dataElement sourceTag="BuildingClassType" />
		<dataElement sourceTag="iNumBuildingNeeded" />
	</ComplexMapping>
	<ComplexMapping sourceTag="BuildingClassNeededs" outColumn="BuildingClassNeededs">
		<dataElement sourceTag="BuildingClassType" />
		<dataElement sourceTag="bNeededInCity" />
	</ComplexMapping>
	<ComplexMapping sourceTag="SpecialistYieldChanges" outColumn="Specialist Yield Modifier 1(food),Specialist Yield Modifier 2(production),Specialist Yield Modifier 3(gold)">
		<dataElement sourceTag="SpecialistType" default="0" />
		<dataElement sourceTag="YieldChanges" default="0" />
	</ComplexMapping>
	<ComplexMapping sourceTag="BonusYieldModifiers" outColumn="Bonus Yield Modifiers 1(food),Bonus Yield Modifiers 2(production),Bonus Yield Modifiers 3(gold)">
		<dataElement sourceTag="BonusType" default="0"/>
		<dataElement sourceTag="YieldModifiers" default="0"/>
	</ComplexMapping>
	<ComplexMapping sourceTag="ImprovementFreeSpecialists" outColumn="ImprovementFreeSpecialists">
		<dataElement sourceTag="ImprovementType" />
		<dataElement sourceTag="iFreeSpecialistCount" />
	</ComplexMapping>
	<ComplexMapping sourceTag="Flavors" outColumn="Flavors">
		<dataElement sourceTag="FlavorType" />
		<dataElement sourceTag="iFlavor" />
	</ComplexMapping>
	<SimpleMapping sourceTag="HotKey" outColumn="HotKey" />
	<SimpleMapping sourceTag="bAltDown" outColumn="bAltDown" />
	<SimpleMapping sourceTag="bShiftDown" outColumn="bShiftDown" />
	<SimpleMapping sourceTag="bCtrlDown" outColumn="bCtrlDown" />
	<SimpleMapping sourceTag="iHotKeyPriority" outColumn="iHotKeyPriority" />
	<SimpleMapping sourceTag="iOrderPriority" outColumn="iOrderPriority" />
	<SimpleMapping sourceTag="bDCMNukesOkay" outColumn="bDCMNukesOkay" />
	<SimpleMapping sourceTag="iDCMAirbombMission" outColumn="iDCMAirbombMission" />
	<SimpleMapping sourceTag="GenericBuildingInfo" outColumn="GenericBuildingInfo" />
</ScanTemplate>

Dancing Hoskuld · Oct 19, 2011

I suspect the first line of the XML myself. Fhe only difference I can see in the Mesopotamian one is in the first line.

Code:

<?xml version="1.0" encoding="UTF-8"?>

Thunderbrd · Oct 19, 2011

Well... I remember there being xml errors in most of the files that I couldn't scan correctly before but I can't quite recall how to find them. That may well be much like what I was finding before...

Koshling · Oct 19, 2011

Dancing Hoskuld said:
I suspect the first line of the XML myself. Fhe only difference I can see in the Mesopotamian one is in the first line.

Code:

<?xml version="1.0" encoding="UTF-8"?>

I've looked at a couple of the failing files, and yes this is the cause. The files that are failing all contain characters that are not valid in UTF-8 encoding (e.g. - the accented 'e' in the French 'Louvre' in the descriptive text for the Louvre's build ing def). The encoding needs to be changed to utf-16 I think (that worked for me when I did it in an XML editor, but I have not yet checked it doesn't cause Civ any issues, though it shouldn't)

Dancing Hoskuld · Oct 19, 2011

Koshling said:
I've looked at a couple of the failing files, and yes this is the cause. The files that are failing all contain characters that are not valid in UTF-8 encoding (e.g. - the accented 'e' in the French 'Louvre' in the descriptive text for the Louvre's build ing def). The encoding needs to be changed to utf-16 I think (that worked for me when I did it in an XML editor, but I have not yet checked it doesn't cause Civ any issues, though it shouldn't)

I think it is a hold over from when some of the files had game text in them rather than links to game text. They, the encoding part of the comment, can probably be gotten rid of

Thunderbrd · Oct 19, 2011

hmm... not sure I have a program (xml editor) that can do that for those files... any suggestions? (notepad++ only seems to cover utf-8 and just changing the first line doesn't help)

Koshling · Oct 20, 2011

Thunderbrd said:
hmm... not sure I have a program (xml editor) that can do that for those files... any suggestions? (notepad++ only seems to cover utf-8 and just changing the first line doesn't help)

As DH said - the CORRECT way to fix this is to remove the text that needs accented characters etc. from these files and put them in th text definitions instead - all text really SHOULD be in the text defintions anyway for (in principal) localizability. Doing that requires nothing more than Wordpad.

Thunderbrd · Oct 20, 2011

Ok, fair enough. Is there a quick and easy way to scan the document for invalid characters then? How will I know what is and is not acceptable?

ugh... I feel like I'm really slow to pick this up.

Koshling · Oct 20, 2011

Thunderbrd said:
Ok, fair enough. Is there a quick and easy way to scan the document for invalid characters then? How will I know what is and is not acceptable?

ugh... I feel like I'm really slow to pick this up.

Well - the main def files just shouldn't include ANY free form text relally (even though without special characetrs it's harmless). Just move them all to text defintions as they should be. For now just do the ones that cause your scans problems, but long term it's a good tidy-up to have completed uniformly.

Thunderbrd · Oct 20, 2011

well... ok, so I took from that that the offending characters are defined as 'anything you wouldn't just normally be able to type without special methods' and that we only have to worry about the files that won't scan right.

Are you saying that its going to be necessary to look at each character in each document that doesn't scan, scouring with the eyes for an out of place offender? whew... wouldn't there be an automated means in a notepad program somewhere to find these buggers?

Dancing Hoskuld · Oct 20, 2011

Thunderbrd said:
well... ok, so I took from that that the offending characters are defined as 'anything you wouldn't just normally be able to type without special methods' and that we only have to worry about the files that won't scan right.

Are you saying that its going to be necessary to look at each character in each document that doesn't scan, scouring with the eyes for an out of place offender? whew... wouldn't there be an automated means in a notepad program somewhere to find these buggers?

Usually free text occurs in the tags Description, Civilopedia or Strategy. Those are the ones that need looking at. Tedious I know. I just search on <Description> and look at the three lines.

Thunderbrd · Oct 21, 2011

Well... ok. Are we sure taking that step will be enough? I suppose it would get us closer to the cure on each of these files. And I'm also perhaps most overwhelmed by the fact that we have the core building file coming up in this. But that seems simple enough, provided we're not being thrown off by characters that we can't see...

Actually, I have an idea... hmm...

Thunderbrd · Oct 21, 2011

I did some things with the font display to make such characters stand out. There are still a number of files that cannot be scanned despite being able to verify that there are no offending characters outside the usual range.

One example of such a file is \Mesopotamian_CIV4BuildingInfos.xml

This one is short enough (not like the core building infos file) that one can quickly and easily verify that such characters are NOT the problem here. What IS the problem is still a bit of a mystery and may answer to why other, longer files that don't seem to have out of place characters are still giving me trouble when scanning (like the core building infos file).

This is not to say that eliminating such offending characters didn't provide a fix... on some files, the Louvre for example, it was the solution (and the first line, <?xml version="1.0" encoding="UTF-8"?>, was as displayed and caused no further problems once the out of bounds character was resolved.

So yeah, that helps but apparently its not the whole answer as to why some of them aren't scanning properly. Any other theories I can check?

Thunderbrd · Oct 21, 2011

Sorry for so many posts in a row but all are distinct updates here...

I believe I've isolated the problem. By isolating one scanned field at a time, I've found that I can debug this rather easily. It shows me what field is causing a problem somewhere in the files that refuse to scan. So far, in EVERY case, this means that there were duplications of the same field within one entry.

Example:

Code:

			<TechTypes>
			<PrereqTech>TECH_SHELTER_BUILDING</PrereqTech>
			</TechTypes>
                        <TechTypes/>

Easy mistake to make when modding and since the files seem to load just fine when such errors are in place, they are all the more easily overlooked and forgotten. So I'll be editing a lot of xml building files for these kinds of issues and should have this wrapped up into a spreadsheet by the end of the night

Victory!!! :woohoo:

C2C Spreadsheeting Tool

C2C War Dog

C2C War Dog

Deity

C2C War Dog

Vorlon

C2C War Dog

C2C War Dog

Deity

C2C War Dog

Vorlon

Deity

C2C War Dog

Vorlon

C2C War Dog

Vorlon

C2C War Dog

Deity

C2C War Dog

C2C War Dog

C2C War Dog

Similar threads