civchecker - avoid crashes in your mod

[God-Emperor]

It looks like Civcheck doesn't like comments in the XML. They really seem to mess it up.

I think the problem is that a comment tag has no closing tag and is not set up like a tag that doesn't need one (i.e. it is just "<!--X -->", not "<!-- X --/>", and there is no corresponding "</!-- X -->") so it somehow thinks it is one level deeper than it is (with the comment as the parent tag) and stops looking for the right tags for definitions.

Example: In Final Frontier Plus many of the buildings are numbered in the BuildingInfo like so:

		<BuildingInfo><!-- 64 -->
			<BuildingClass>BUILDINGCLASS_THE_ARSENAL</BuildingClass>
			<Type>BUILDING_THE_ARSENAL</Type>

This makes it fail to load the BUILDING_THE_ARSENAL definition (which it should because it is in a Type tag inside a BuildingInfo, as defined in the .config file). It complains that this is an unused tag in this file at this location. Comments apparently count as a new level of child for the current parent, and the parent of all that follows it thereby making it think the Type tag is a child of the "!-- 64 --" tag instead of the BuildingInfo tag and since the .config file has no add_define of a child tag for "!-- 64 --", it doesn't get defined.

That's my guess, anyway.

 

[davidlallen]

Thanks! There are some bugs in the xml parser part. I have a new version which I have been using locally. I will try to update the documentation and post it shortly. I believe it should fix this problem.

 

[God-Emperor]

Argh. Bad example. Apparently also an incorrect guess as to the actual problem.

Looking at the BUILDING_THE_ARSENAL stuff again it seems I had moved from the "UndefinedSymbol" section to the "Definition" section - so it did pick up that definition. That's what I get for turning on the extra verbosity things.

The problem I'm seeing may be only happening for definitions in the TEXT tag. I see a bunch that are reported in the UndefinedSymbol section that do exist (and appear just fine in the game). I haven't checked them all, but the ones I have looked at appear to all be defined with comments at the end of the Tag line, along the lines of:

	<TEXT>
		<Tag>TXT_KEY_BUILDING_SURVIVAL_SHRINE_PEDIA</Tag><!-- Untranslated -->
		etc.

which results in CivCheck reporting this:

UndefinedSymbol: TXT_KEY_BUILDING_SURVIVAL_SHRINE_PEDIA
   Used: FFP buildings/civ4buildinginfos.xml at line 6703

I won't make a guess as to why a comment after the tag is causing a problem, but it looks like all of the ones that have them are reported as UndefinedSymbol when they are used elsewhere. The text keys in the same file that do not have comments after the </Tag> do show in the Definition list, not as UndefinedSymbol.

Except some that don't appear anywhere. It is looking like technologies/CIV4TechInfos.xml might not be checked for missing tags since although it is listed in the FileRead stuff, the text keys defined for use in that file that have comments after the Tag tag are not reported in either the UndefinedSymbol or Definition sections in the output (or the UnusedSymbol section, for that matter). A little more checking shows that it looks like none of the text keys used by that file appear in anywhere in the CivChecker output, whether the ones I added or the originals from the Final Frontier mod's text related files. Strange. How is a text key type symbol not in any of the output choices: UndefinedSymbol, Definition, or UnusedSymbol?

 

[davidlallen]

Congratulations, you have gotten further into the guts of civcheck than any other poster in this thread. There is no special treatment for any xml file, so there is no obvious reason why symbols in the techinfos file should be treated differently. There is a fourth category besides undefined, definition or unused; many symbols are filtered out to reduce noise. For example, please note the following part of the civcheck.config file:

# You may want to find undefined text strings; then comment this out
# add_prefix TXT_KEY
# But there are many vanilla undefined text strings, so don't report these
add_prefix TXT_KEY_ACTION TXT_KEY_CALENDAR TXT_KEY_EMPHASIZE TXT_KEY_ERA
add_prefix TXT_KEY_FORCE_CONTROL TXT_KEY_GAMESPEED TXT_KEY_MISSION
add_prefix TXT_KEY_SPECIALIST TXT_KEY_TUTORIAL TXT_KEY_UPKEEP TXT_KEY_WORLD

Also, civchecker only counts a string as a symbol if it is all upper case or numbers or underscores. I have never seen a civ xml file with mixed case symbols; this also helps to reduce some noise. Do either of these help explain the behavior you see?

 

[deanej]

The comments thing is a great observation. I've looked at the two files that give errors all the time even when there are no problems - Civ4ThroneRoomStyleInfos and Civ4SpaceShipInfos. Both have comments between many entries.

 

[God-Emperor]

Congratulations, you have gotten further into the guts of civcheck than any other poster in this thread. There is no special treatment for any xml file, so there is no obvious reason why symbols in the techinfos file should be treated differently. There is a fourth category besides undefined, definition or unused; many symbols are filtered out to reduce noise. For example, please note the following part of the civcheck.config file:

# You may want to find undefined text strings; then comment this out
# add_prefix TXT_KEY
# But there are many vanilla undefined text strings, so don't report these
add_prefix TXT_KEY_ACTION TXT_KEY_CALENDAR TXT_KEY_EMPHASIZE TXT_KEY_ERA
add_prefix TXT_KEY_FORCE_CONTROL TXT_KEY_GAMESPEED TXT_KEY_MISSION
add_prefix TXT_KEY_SPECIALIST TXT_KEY_TUTORIAL TXT_KEY_UPKEEP TXT_KEY_WORLD

Also, civchecker only counts a string as a symbol if it is all upper case or numbers or underscores. I have never seen a civ xml file with mixed case symbols; this also helps to reduce some noise. Do either of these help explain the behavior you see?

Not really. One that doesn't appear anywhere in the output file is TXT_KEY_TECH_MISC_0, which is used in the Description tag of a TechInfo. It does have a comment after the Tag key in its TEXT entry. Because of this, I would expect it to show up in an UndefinedSymbol (like the others with comments) pointing to the line in CIV4TechInfos.xml where it is used. But there is no such entry. However there is also no entry in full list labeled with Definition. The TXT_KEY_TECH prefix doesn't match anything in the list of things to ignore (the more general TXT_KEY line was commented out, which would be why I see the ones with comments in them in the undefined list). Since it isn't defined (or it would appear in the Definition section), its use in CIV4TechInfos.xml ought to leave the entry in UndefinedSymbol if the file is being checked, therefore the file is not actually being checked - unless something weird is happening, which is always possible.

 

[davidlallen]

I have changed the code actually a long time ago without releasing it, so I forget when I fixed this problem. If you change the "0" to "ZERO" or remove it temporarily, then do you get the expected behavior? At one point, it only accepted a symbol which was all upper case or underscores, no numbers.

 

[Bahmo]

What does "uncomment" mean?

 

[davidlallen]

In most programming languages there is a string which indicates that the rest of the line is a comment. For example, in C++ the string "//" means the rest of the line is a comment; in tcl, python and other languages the string is "#". Very often programmers may want to give an example of some syntax, while keeping it invisible to the actual program. So they put the example after a comment character. To "uncomment" a line means to remove the comment character at the beginning of the line, so it becomes visible to the program.

 

[smeagolheart]

re: uncomment

An example is if you change this:

# You may want to find undefined text strings; then comment this out
# add_prefix TXT_KEY

to this:

You may want to find undefined text strings; then comment this out
add_prefix TXT_KEY

The game will actually try to execute the words (code) instead of just skipping over the lines.

 

[Bahmo]

Either I'm doing something terribly wrong with Civchecker, or it isn't good enough to find what's wrong with my mod. Here's the configuration, in my attempt to check my "World 2010" mod:

Spoiler :

# The installation path is probably right. Change it if you installed
# the game somewhere else.
set install_dir "c:/program files/firaxis games/sid meier's civilization 4"

# If you have files in your local CustomAssets you want checked,
# uncomment the next line and make sure the path is correct. For
# non-English language installations, you will need to change "my documents"
# to the right name also.
# add_directory custom "c:/documents and settings/jena/my documents/my games/beyond the sword/customassets"

# If you want to run this on a mod, uncomment the next line and make
# sure the path points to your mod.
add_directory charlmagne "$install_dir/beyond the sword/mods/World 2010"

# If you don't have BTS or warlords, comment out the corresponding lines
add_directory bts "$install_dir/beyond the sword/assets"
add_directory warlords "$install_dir/warlords/assets"
add_directory vanilla "$install_dir/assets"

# Uncomment these lines to enable some more verbose reports
report_filenames
report_definitions
report_unused

# You may want to find undefined text strings; then comment this out
add_prefix TXT_KEY

### From here down, you should only change if you know what you are doing.

# Values inside these keys will be checked for art file existence
add_art_key Button KFM NIF SHADERNIF

# General list of symbol prefixes to ignore. You may prefer to locate
# the corresponding type definition and use add_define instead.
add_prefix ADVISOR ANIMATIONPATH ANIMCAT AQUEDUCT ARTSTYLE ASSS
add_prefix ATTACHABLE AUTOMATE AXIS BONUSCLASS CAMERA CITY
add_prefix CITYBILLBOARD CITYSIZE CITYTAB COLOR COMMAND CONCEPT
add_prefix CONTACT CONTROL CULTURELEVEL CURSOR DELTA DF DIPLOMACYPOWER
add_prefix EFFECT ENDSTEP FIREWORKS FLAVOR FOOTSTEP FORCECONTROL GAME
add_prefix GAMESPEED GLOBE GRAPHICOPTION INTERFACE INTERFACEMODE KB
add_prefix LIGHTING LOOPSTEP LSYSTEM MAIN_MENU MAINMENU MENU PLAYEROPTION
add_prefix PLOT RIVER SEALEVEL SYSTEM TILE_ART TUTORIAL TXT_MAIN_MENU
add_prefix UNIT_HEALTH UPKEEP WALL WIDGET WORLDBUILDER WORLDSIZE YIELD

# If an all-cap string happens to appear in a language tag, don't take it
add_language English French German Italian Spanish

# Some exact strings are undefined, which we don't want
add_string NONE UNIT GENERAL SIEGE WORKERSWORKING STEAM UNIT_SCALE
add_string LIGHT_TYPE_SUN ICBM IDS ONU SDI SOUNDSCAPE TEMP MD
add_string FOOTPRINT FOG_TYPE_NONE LANDSCAPE_DEFAULT LOOPINGEFFECT_SMOKE
add_string DISSOLVE FADEIN POP

# These are the containers and keys that perform a definition
add_define AttitudeInfo Type
add_define BonusArtInfo Type
add_define BonusInfo Type
add_define BuildInfo Type
add_define BuildingArtInfo Type
add_define BuildingClassInfo Type
add_define BuildingInfo Type
add_define CalendarInfo Type
add_define CivicInfo Type
add_define CivicOptionInfo Type
add_define CivilizationArtInfo Type
add_define CivilizationInfo Type
add_define ClimateInfo Type
add_define CommerceInfo Type
add_define CorporationInfo Type
add_define DenialInfo Type
add_define DiplomacyInfo Type
add_define DomainInfo Type
add_define EffectTypes EffectType
add_define EmphasizeInfo Type
add_define EntityEventInfo Type
add_define EraInfo Type
add_define EspionageMissionInfo Type
add_define EventInfo Type
add_define EventTriggerInfo Type
add_define Fader Name
add_define FeatureArtInfo Type
add_define FeatureInfo Type
add_define FormationInfo Type
add_define GameOptionInfo Type
add_define GoodyInfo Type
add_define HandicapInfo Type
add_define HurryInfo Type
add_define ImprovementArtInfo Type
add_define ImprovementInfo Type
add_define InterfaceArtInfo Type
add_define InvisibleInfo Type
add_define LeaderheadArtInfo Type
add_define LeaderHeadInfo Type
add_define LoadTypes LoadType
add_define MemoryInfo Type
add_define MiscArtInfo Type
add_define MissionInfo Type
add_define MonthInfo Type
add_define MovieArtInfo Type
add_define MPOptionInfo Type
add_define PathEntry Operator
add_define PlayerColorInfo Type
add_define PositionTypes PositionType
add_define ProcessInfo Type
add_define ProjectInfo Type
add_define PromotionInfo Type
add_define ReligionInfo Type
add_define RouteInfo Type
add_define SeasonInfo Type
add_define SlideShowInfos TransitionType
add_define SoundData SoundID
add_define SpaceShipInfo InfoType
add_define SpecialBuildingInfo Type
add_define SpecialistInfo Type
add_define SpecialUnitInfo Type
add_define TechInfo Type
add_define TerrainArtInfo Type
add_define TerrainInfo Type
add_define TEXT Tag
add_define TraitInfo Type
add_define TurnTimerInfo Type
add_define UnitAIInfo Type
add_define UnitArtInfo Type
add_define UnitArtStyleTypeInfo Type
add_define UnitClassInfo Type
add_define UnitCombatInfo Type
add_define UnitFormation FormationType
add_define UnitInfo Type
add_define VictoryInfo Type
add_define VoteInfo Type
add_define VoteSourceInfo Type

Anything wrong there?

 

[davidlallen]

There is no way to look at the setup file, by itself, and tell whether it is correct. What messages are you getting? What type of problem are you looking for?

 

[phungus420]

Could you release the version you have that checks the CityLSystem and PlotLSystem files? I downloaded the version in the OP link, and it apears to be the same that was originally created, and I don't see anything in the logs that looks like it checked the LSystem files. These are the main files I need checked, since every crash that has occurred in LoR since it's release was due to Ethnic City Styles, and I'd like to update this component with Avain's optimized version of this mod component, but I'm almost certain there are issues in it with misnamed and/or missing files that will cause difficult to track down art crashes (a missing art file in the LSystem causes a crash, it does not just show up pink).

 

[phungus420]

Hello?

 

[deanej]

Can this be made to work with FFH? I managed to take care of the undefined symbols by editing the config file, but there is a lot of missing art that gets detected even though everything works fine in game.

 

[davidlallen]

Have you unpak-ed the art files and listed the unpak directory in the search path? I do not think there should be anything special about the FFH art.

 

[deanej]

I looks like I did at least. I have it unpacked and pointed civchecker at the directory. Is there a way to find out if something went wrong in the unpacking? It loads art from that location.

 

[davidlallen]

Sorry, I did not quite understand. When you say "it" loads art from this location, do you mean the Civ game or civchecker? That is, suppose the unpak directory contains art for units A and B. If A is listed in a civchecker "missing art" message but B is not listed, this means civchecker is able to process this directory since it found B.

Another way to tell whether the unpak directory is processed by civchecker is to run civchecker twice, once with the unpak directory specified, and once without. If the size of the output file does not change, then probably it is not processing the directory.

I should add an error message for the case when a directory name is spelled wrong, instead of silently assuming this directory is not needed. But, I have not worked on this code in the past year or so.

 

[deanej]

Civchecker says it loads 9047 files from that directory, and with it included finds 22 missing art files (2921 missing without it included).

 

[davidlallen]

Are you positive that these 22 units display correctly in game? Is it possible that there is some third directory, containing this art, which is read by the game but not read by civchecker?

If there is something unique about these 22 units, that may provide some clue.