[SDKMOD] Modular XML Loading 2.0

Impaler[WrG]

Civ4:Col UI programmer
Joined
Dec 5, 2005
Messages
1,750
Location
Vallejo, California
MXMLL 3.0 by Impaler[WrG] & Chinese American

DOWNLOAD
GET MODULAR XML LOADING 3.0 DLL, Source Code, Templates and Instructions HERE

INTRODUCTION
Modular XML Loading (MXMLL) is a mod to the Civ4 Warlords DLL which allows the game to load and read multiple XML files of many types. It uses a standardized naming and organizing convention which has the promise of delivering full drag and drop modularity to the games content (almost). Properly configured modules will be able to alter the games default data or add new content such as a Tech, Wonder, building, Unit, Leader or whole Civs with such elements nested inside it. The end of tedious cutting and pasting to assemble Mod content will herald a new age in Civ4 Moding.

NEW IN THIS VERSION
3.0 is a deep re-write of MXMLL 2.0 itself an update and extension of modular loading 1.2 by Chinese American which can be found at http://forums.civfanatics.com/showthread.php?t=184907. Despite this all modules made under 2.0 remain compatible with this 3.0 version which removes the restriction in 2.0 which prevented a base element from referencing a module element. In addition moders can safely reference any XML element on any supported file or within a file rather then just those that were earlier in the loading order. Lastly the ability to read and write modular data to the games cache allows for faster loading both when using modules or not. Future version will seek to address the remaining shortcomings particularly in the area of Audio and Diplomacy text.

HOW IT WORKS
The mod operates entirely during the xml loading portion of the game. First the games "Base" content is added to a list of targeted files. Base contents are the games original XML files as they would otherwise be loaded meaning Custom Assets at the highest precedence, then the Mod if any and lastly Assets from the Civ4 directory. Next regular expression searching is done of the whole XML folder Mod folder and custom assets are both found and adding to the file list. All Tag names on these files are then hashed for future reference and the data indexes assigned. Then a second round of loading occurs and the remaining data is loaded into the predetermined indexes (as opposed to post-determined as was originally done). Modules load after Base content and can step-on aka replace Base data that has a matching Type Tag name, modules will also step on each other with the last one to load being the one seen in game. Module loading order is not affected by Custom Assets vs Mod, either could step on the other so simply avoid ever having two conflicting modules.

WHAT DOSN'T WORK
Unfortunately do the the nature of the LSystem.xml files for Buildings and Improvements its not possible to modularize these files because they don't pass through the SDK. Sound and Audio data has not been addressed yet either and several file types are linked with Enums in the SDK thus any extensions of these files would have no effect without a corresponding modification in the SDK. All other data both XML and art can be made into a module but without LPlot data the building or improvement will be invisible in the game, though it will have every other game play affect and display in the Pedia. A copy of a modified LSystem or instructions on how to modify it should be included in a Building or Improvement module and some notes made on the readme.txt. Similarly custom Diplomacy Text common in new Civs isn't supported due the the complex nature of this file. Both of these files will require traditional cut and paste.

SUPPORTED FILE TYPES
The following types of files are supported, in all instances the game will search and find files matching xml/*_TYPE.xml ware TYPE is one of the following strings.

CIV4ArtDefines_Unit
CIV4ArtDefines_Building
CIV4ArtDefines_Improvement
CIV4ArtDefines_Civilization
CIV4ArtDefines_Leaderhead
CIV4ArtDefines_Bonus

CIV4GameSpeedInfo
CIV4TurnTimerInfo
CIV4WorldInfo
CIV4ClimateInfo
CIV4SeaLevelInfo
CIV4AdvisorInfos
CIV4TerrainInfos
CIV4EraInfos
CIV4UnitClassInfos
CIV4SpecialistInfos
CIV4TechInfos
CIV4FeatureInfos
CIV4TraitInfos
CIV4GoodyInfo
CIV4HandicapInfo
CIV4CivicOptionInfos
CIV4UpkeepInfo
CIV4HurryInfo
CIV4SpecialBuildingInfos
CIV4ReligionInfo
CIV4CultureLevelInfo
CIV4BonusClassInfos
CIV4VictoryInfo
CIV4BonusInfos
CIV4BuildingClassInfos
CIV4BuildingInfos
CIV4SpecialUnitInfos
CIV4ImprovementInfos
CIV4CivicInfos
CIV4LeaderHeadInfos
CIV4ColorVals
CIV4PlayerColorInfos
CIV4EffectInfos
CIV4BuildInfos
CIV4UnitInfos
CIV4CivilizationInfos

CIV4GameText
GlobalDefines

NAMING RULES
The file naming system that must be used to get a modular xml file to load is strict but simple and universal. Because * is a wild card you can place a descriptive name at the front such as Ninja_CIV4UnitInfos.xml also their can be any number of intervening directories under the xml directory so data can be nested to your hearts content. This is in contrast to the games default loading which was and still is very rigid in its structure. Failure to have a file match the regular expression is the most likely reason for content to silently (no load warning) not show up in the game.

SCHEMAS
The main drawback of this directory flexibility is that every XML file still needs to be validated before loading and the games validation code is off limits to me. It demands the Schema be in the same directory as the file its validating, if not the loading fails and a warning is thrown which you can OK past to continue loading but their will then likely be subsequent "ITEM NOT FOUND" warnings as the other files associated with the module are left with dangling references. Their have been odd cross validation errors I cant explain arising from modules using schemas of the same name, thus I recommend that all module schemas be renamed with a simple "module name_" at the front to match the module the module their in. Remember update the header of the files that use that Schema. This is especially true for Modules that are going to use a new XML tag that that interacts with a modified DLL, the local Schema will allow your module to load under a unmoded DLL and will isolate the modified Schema from invalidating other XML files.

ORGANIZATION
Though not required I would strongly recommend content creators and moders use my organization proposals to keep everyone on the same page and make modules easy to use. Within the xml their will be two folders for each of the major data types, one for Modifications and one for new material called Custom. Things like a Civ which have dependent sub elements should have their sub elements nested within their packages so the single package contains everything necessary to add or remove the Civ from the game. In general content should be in the smallest pieces able to standalone such as a single Unit, Tech or Building. Only if multiple elements are inter-dependent should they be packaged together.

TEMPLATES
To help make the creation of modules easier I have included as set of templates for several of the major data types in the core download. These folders contain an empty copy of each fill that needs to be included to get a new element of that type into the game. A modified element only requires a copy of the modified file so for example to change the Pedia of a Unit and Its art Def you would need only a Text and ArtDef file. A typical template looks like this.

Code:
<?xml version="1.0"?>
<!-- edited with XMLSPY v2004 rel. 2 U ([url]http://www.xmlspy.com[/url]) by Soren Johnson (Firaxis Games) -->
<!-- Sid Meier's Civilization 4 -->
<!-- Copyright Firaxis Games 2005 -->
<!-- -->
<!-- Tech Infos -->
<Civ4TechInfos xmlns="x-schema:CIV4TechnologiesSchema.xml">
	<TechInfos>
		
	</TechInfos>
</Civ4TechInfos>

EXAMPLES
As a separate download (at the CCCP Module Library Source Forge repository at http://sourceforge.net/projects/civ4ccp/) you can find a growing collection of modules of all types. These modules are free to download, use, modify and distribute. Moders are invited to submit new modules at the Home thread (see top) or by E-mail to me at impalerwrg@yahoo.com

MAKING MODULES
Because all previous mods have been in the "Monolithic" format, aka one file of each type, most new Modules will be made by extraction of existing content out of these mods or efforts to break them down completely into Modules. Another popular method will be from scratch construction especially with new Art assets that have just been released. Once you've chosen the type of module your making and have the contents at hand begin by select the appropriate Template folder, make a copy and rename it, use a good descriptive name that is unlikely to be used twice. Dump all the art files directly into the folder .nif's .dds's .kfm's .etc .etc. Then if your doing an extraction simply copy and paste the appropriate elements on to the template xml's. Rename each file by adding the Module name before the '_' also add it for the schemas. The ArtDef Info will need to be edited, change what ever file patway was being used before to "xml/TYPEOFMODULE/MODULENAME/FILENAME". TypeOfModule would be things like "Custom units" or "Modified Buildings", you might need a different organization thats more appropriate for your purposes but try to keep it organized. Some file types have Button pathways on them directly (Units) be sure to edit these as well. Pink/Magenta buttons result if the pathway is wrong, wrong .nif and .kfm files will appear blank or generate a python exception if Debugging is on. If your creating from scratch copy and paste from an original object that is a close match for what your making and edit it into what you want. Lastly discard any files you you didn't use, for example if making a Unique Unit you don't need to edit the UnitClass.xml so just discard it. Once your ready load up and look in the Civilopedia or start a game (don't try to run a save made without the module it will be incomputable). You may have to go through a few cycles of tweaking Art scales fixing broken pathways, balancing stats etc etc. When building up a large multi layered Module like a whole Civ its recommended that you start at the top level the Civ get it working and then add Leaders, Units and Buildings as sub-folders, this helps organize them and makes it easier to add and remove them later, be sure to take the parent directory into account when doing the art pathways. If any loading errors occur track them down by activating logging and examine the xml.log generated at load time, it will list each file loaded and what elements were found on them, errors are usually the result of copy paste errors or failure failure to properly reference the Schema. Finally fill out the readme so others who use your module can credit you and contact you, include any special configuration options or dependencies that will be needed.

CACHE LOADING
The 3.0 release includes the ability to load from the Cache like a normal unmodified DLL (this ability had been sacrificed in the 2.0 version). When cache loading is active the regular expression searching will be skipped for each file that can be loaded in this way (the 13 largest files comprising the bulk of the content), other file types will still search and load modules. When the game detects changes in Custom Assets it will conduct the full search and write the contents of all modules it finds into the Cache files, these will load on subsequent games at near the speed of the unmodified DLL's cache loading. It appears from testing that the game can in some case be tricked into falsely trying to cache load when Modules are added or removed so its recommended that you explicitly DE-activate cache loading after adding or removing modules to be on the safe side. If you get an error during Cache loading just reload and it should be fine.

SDK MERGING
This mod is highly mergable with other SDK mods. Not only dose it operate at a narrow time and place unlikely to interfere with any other code the files themselves are easy to combine because of the four files modified, three (CvXMLLoadUtility.h , CvXMLLoadUtility.cpp CvXMLLoadUtilitySet.cpp) are almost never touched in other mods and the third (CvInfos.cpp) has only a few modest additions in the XML readings. You are free to merge this code with your own so long as I am credited and the end user is given links to the SF Module library. Of particular interest is the Civilizeditor by Lopez which has great synergy with with mod.


CHANGELOG
Spoiler :


1.0 - By Chinese American, Sep 4th, 2006
Reads partial xml files for *_CIV4ArtDefines_Unit.xml, *_CIV4UnitClassInfos.xml, *_CIV4UnitInfos.xml.

1.1 - By Chinese American, Sep, 2006
Will replace old types with new ones.
Reads partial xml files for *_CIV4ArtDefines_Building.xml, *_CIV4BuildingClassInfos.xml, *_CIV4BuildingInfos.xml.

1.2 - By Chinese American, Sep 9th, 2006
Workaround for bug where buildings get obscene happiness from new buildings.
Added reading techs, improvements, builds, traits, leaderheads, civilizations.
Reads partial xml files for *_Arts.xml, *_Classes.xml, and *_Infos.xml in xml\units\ and xml\buildings\.
Reads partial xml files for *_UnitArts.xml, *_UnitClasses.xml, *_UnitInfos.xml, and
*_Units.xml in xml\units\.
Reads partial xml files for *_BuildingArts.xml, *_BuildingClasses.xml, *_BuildingInfos.xml, and *_Buildings.xml in xml\buildings\.
Reads partial xml files for *_CIV4TechInfos.xml, *_TechInfos.xml, *_Techs.xml, and *_Infos.xml in xml\technologies.
Reads partial xm files for *_CIV4ImprovementInfos.xml, *_ImprovementInfos.xml and *_Improvements.xml in xml\terrain, and xml\art\*_CIV4ArtDefines_Improvement.xml and xml\terrain\*_ImprovementArts xml.
Reads partial xml files for *_CIV4BuildInfos.xml, *_BuildInfos.xml and *_Builds.xml in xml\units.
Reads partial xml files for *_CIV4CivilizationInfos.xml, *_CivilizationInfos.xml and *_Civilizations.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Civilization.xml and xml\civilizations\*_CivilizationArts.xml.
Reads partial xml files for *_CIV4LeaderheadInfos.xml, *_LeaderheadInfos.xml and *_Leaderheads.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Leaderhead.xml and xml\civilizations\*_LeaderheadArts.xml.

2.0 Initial Re-Release by Impaler[WrG] on Dec 20th, 2006
standardizes all file expressions to "xml/*_File.xml"
Adds support for following files
CIV4GameSpeedInfo, CIV4TurnTimerInfo, CIV4WorldInfo, CIV4ClimateInfo, CIV4SeaLevelInfo, CIV4AdvisorInfos, CIV4TerrainInfos, CIV4EraInfos, CIV4UnitClassInfos, CIV4SpecialistInfos, CIV4FeatureInfos, CIV4GoodyInfo, CIV4HandicapInfo, CIV4CivicOptionInfos, CIV4UpkeepInfo, CIV4HurryInfo, CIV4SpecialBuildingInfos, CIV4ReligionInfo, CIV4CultureLevelInfo, CIV4BonusClassInfos, CIV4VictoryInfo, CIV4BonusInfos, CIV4SpecialUnitInfos, CIV4CivicInfos, CIV4ColorVals, CIV4PlayerColorInfos, CIV4EffectInfos, CIV4GameText, GlobalDefines

3.0 Update by Impaler[WrG] Feb 1st, 2007
deep re-write adds caching of module data, universal cross file referencing, fixed incorrect error messages durring loading, many earlier functions by CA/Impaler[WrG] and Firaxis obsoleted and removed.
 
Modual Library v1.0

DOWNLOAD HERE >>> http://forums.civfanatics.com/uploads/73222/Modual_Library_v1.0.zip

Impalers Modified University - a pin point modification to the University

Sphinx and LeonardosWorkshop - two new Wonders previously seen in Kets composite mod

Medic - a new standalone unit based on resent artwork by asioasioasio

Aborigines - a whole civ from Civ Gold containing 2 leaders a Unique Unit and Unique Buildings

Impalers Terrain Mod - some slight modifications to desert, jungles (less terrible) and floodplains (nerfed but are now in more terrain)

To use place the modular loading DLL in Custom Assets and place a mod folder with its parent folder (Custom Units, Custom Wonders ect) in xml. Placing either inside of a mod folder and running the mod should work as well.
 
reserved for me
 
Beautiful work, I will definitly want to get this incorporated into FfH at some point.
 
Very cool Impaler... I will definately have to include this in the Civilizeditor and throw in an option to allow people to create modlets :D
 
Ok great. Time to try this one since I was having issues with chineese americans :) thanks for pointing it out :)... and of course thanks for all the hard work.

Update: I have now fully integrated this into my mod. So far it's working great! The unit I tested it with loads perfectly, and I can use that as a template for future units. The potential for this is incredible. Again Great work and Thank You!
 
My future plans for this mod are...

Squash any bugs that are found,

Expand to more file types, Formations for one will be included in the next update,

Continue to enlarge the Modual Library, I've been modularizing several of asiox3's Wonders for addition, I'll also see if I can get some Religion moduals made after Christmas. Civ Gold Civilizations files have already been preped by Perplexus and others so these can be processed quickly once he E-mails them to me.

Longer term plans involve a deeper re-organization of file loading which may be able to solve what I call the downhill reference problem. All files are loaded in a very specific order because each file can only reference data on files loaded before it. Techs come early and are followed by buildings and units bring up the rear with lots of other files sprinkled in their too. This puts a lot of fundamental limitations on what kind of mods can be made and how their XML files must be organized. My plan is to map all info tags from all files before proceeding with the remainining data loading, this will allow any file to reference any other file without restriction.

Theirs also potential for a feature that can be of assistence in Lopez's Civilizeditor, a Modual taging system. Basicaly every data opject would be taged with the string of the file pathway from which it originated from. The editor could extract this string to know from what modual or moduals any piece of game data was loaded from and then inteligently manipulate or display this data to the editor user.

Lastly a boolean switch to disable the loading of Base content for thouse total convertion mods out their, or some other method of using a modual to selectivly knock out an element.
 
Impaler[WrG];4910848 said:
My future plans for this mod are...

Squash any bugs that are found,

Expand to more file types, Formations for one will be included in the next update,

Continue to enlarge the Modual Library, I've been modularizing several of asiox3's Wonders for addition, I'll also see if I can get some Religion moduals made after Christmas. Civ Gold Civilizations files have already been preped by Perplexus and others so these can be processed quickly once he E-mails them to me.

Longer term plans involve a deeper re-organization of file loading which may be able to solve what I call the downhill reference problem. All files are loaded in a very specific order because each file can only reference data on files loaded before it. Techs come early and are followed by buildings and units bring up the rear with lots of other files sprinkled in their too. This puts a lot of fundamental limitations on what kind of mods can be made and how their XML files must be organized. My plan is to map all info tags from all files before proceeding with the remainining data loading, this will allow any file to reference any other file without restriction.

Theirs also potential for a feature that can be of assistence in Lopez's Civilizeditor, a Modual taging system. Basicaly every data opject would be taged with the string of the file pathway from which it originated from. The editor could extract this string to know from what modual or moduals any piece of game data was loaded from and then inteligently manipulate or display this data to the editor user.

Lastly a boolean switch to disable the loading of Base content for thouse total convertion mods out their, or some other method of using a modual to selectivly knock out an element.

Might I suggest a "base" mod thats simply civ4/civ4 with warlords completly modularized? This will make it easier for people starting new mods to pick up the orginization, as well as make change to stock code.

On another note i put a big push for this on my mods thread - Hopefully other modders will see it and check it out. I'll report any bugs I might find :)

Oh yeah... when you upgrade could you do your best to ensure that people using the older versions wont have to re-do all their files :)
 
Full back compatability will be maintained I assure you, the core reg-expression code is stable and wont be changed, I'm just manipulationg the structures that it is nested inside so theirs no chance of breaking compatability. Also its possible to modify an existing game element by creating a modual with the same name (XML name that is). The new data will over-write the base data, thus its not nessary to modularize the games base data, also that would be a LOT of work. Example are good to but I will be using all new content for them to give the mod more desirability. The larger the library the greater labor savings for the mod maker, leading to more incorporation in mod DLL's, leading to more content distribution by modual..etc etc...upward network effect spiraling :goodjob:
 
Sounds good :)

So you know I modularized both the Palace and the Barbarian Civs with no issue (i'm going to be making major changes to both).

I did however find one small (and likely unfixable) bug. When an error message occurs due to the game not being able to find a particular piece of information (in the example where I found it I removed part of the Obolisk but missed one of the declarations) it will report the file to be the last read building file (in my case the palace) instead of the location where the missing data is. Just a heads up :)
 
Found another bug for ya...

If you have a Modular Technology, techs that aren't modular cannot be Dependant on it (either OR Dependant or AND Dependant --same issue occurs with both) -- just spent an hour tracking this one down due to misrepresented error messages lol :)

Not a huge deal for me since i'm modularizing my whole mod....but could cause issues if someone wanted to throw a tech in without modularizing any existing techs....
 
Sorry I should have mentioned that limitation, its one of the things the Flat loading I'm working on now would adress. Any other similar dependency should also be affected like Promotions and Improvments. I think I have a work around though, on the TechInfos.xml of the new tech your creating add a second TechInfo element AFTER the tech your adding. The second Info should be a copy of the existing tech you want to make dependent on your new tech with the new dependency added. This should solve the problem as the new tech will process first being at the top of the document and the replacement data will then over-write the original data and the reference should connect. Because Base data is loaded first a reference their will not find anything in a modual but a modual can replace the base data at a later point.

Thanks for the bug reports, I'll try to adress that bad bug reporting string in the next patch so people dont get sent on wild goose chases. I'd also like to develop a more detailed tutorial based on your feedback, their are certainly many details I have omited in hast. Also as you grow the number of moduals your using keep track of your loading times, I'm interested in seeing how much it is slowed under large numbers of moduals, if it becomes an issue Lopez's Civilizeditor will bail me out as it alredy has the ability to merge moduals together.
 
Impaler[WrG];4911789 said:
Sorry I should have mentioned that limitation, its one of the things the Flat loading I'm working on now would adress. Any other similar dependency should also be affected like Promotions and Improvments. I think I have a work around though, on the TechInfos.xml of the new tech your creating add a second TechInfo element AFTER the tech your adding. The second Info should be a copy of the existing tech you want to make dependent on your new tech with the new dependency added. This should solve the problem as the new tech will process first being at the top of the document and the replacement data will then over-write the original data and the reference should connect. Because Base data is loaded first a reference their will not find anything in a modual but a modual can replace the base data at a later point.

Thanks for the bug reports, I'll try to adress that bad bug reporting string in the next patch so people dont get sent on wild goose chases. I'd also like to develop a more detailed tutorial based on your feedback, their are certainly many details I have omited in hast. Also as you grow the number of moduals your using keep track of your loading times, I'm interested in seeing how much it is slowed under large numbers of moduals, if it becomes an issue Lopez's Civilizeditor will bail me out as it alredy has the ability to merge moduals together.

Great :) - Just make sure you let me know which files your changing when you update also - I'd hate to go line-by-line to integrate a change into my source (I custom compiled instead of using your dll 'cause I have some of TheLopez's stuff included as well)

When I release the first version of my prototype in a few days feel free to download it and have a look at the way i'm doing things- did my best to follow your organization as best I could...i'm sure its not perfect though.

I'm currently at 10 Techs, 1 Building, and 2 Civs Modularized. No slow down so far. As a note to that I'm removing all foreign text from my modules since I have no linguistic skills and they make following the files more difficult.

Anything I can do to help count on me :)
 
Great idea, guys! Thank you. I hope it will be widely accepted and used by modding community.
 
Impaler[WrG];4899310 said:
SCHEMAS
The main drawback of this directory flexibility is that every XML file still needs to be validated before loading and the games validation code is inacessable. It demands the Schema be in the same directory as the file its validating, if not the loading fails and a warning is thrown which you can OK past to continue loading but their will then likly be subsequent "ITEM NOT FOUND" warnings as the other files assosiated with the modual are left with dangling references. The nessary Schemas often end up being the largest part of a small modual so if zip size becomes an issue consider ommiting them. But if you do please place a readme in your modual listing what is nessary to make it run. Lastly because each file can be validated by a different schema if your using Modular loading with a DLL that adds new tags be shure to include the updated schema in any content that you want to express that new tag.

Does a schema need to be included even if the XML file should validate against the unmodified schema, i.e. does not need additional tags to "complete" the schema?
 
Yes, a Schema is required all the time though its possible to validate several files with the same schema if their residing in the same folder. The validation code is in the .exe and thus I cant alter it. As I continue to work on the 3.0 release (I'm skipping to 3.0 because its a very deep re-write at the code level even though people wont see much outward difference) I'll see if I can adress this issue and get the file to validate by the original schema if no schema is present in its file.
 
How difficult a process do you think it would be to make a small utility that scans the directories and looks at filenames, and automatically copies the needed files to each directory? It would reduce filesizes for transfers etc. -- Just a thought :)
 
I would LIke to See A new Forum Created for A Plug And Play Compatible Library

*hint hint THUNDERFALL
 
I'll spin off a thread of its own once the CivGold moduals are done, thats the crown jewel after all, I'll also have a lot of new Wonders of the world many of which will use my CCCP effects. Getting a new Forum made would be realy near but probably not essential, we already have forums dedicated to most of the major content types so the moduals could just be found in their respective forums.
 
Top Bottom