MXMLL 3.0 by Impaler[WrG] & Chinese American

DOWNLOAD
GET MODULAR XML LOADING 3.0 DLL, Source Code, Templates and Instructions HERE (http://forums.civfanatics.com/downloads.php?do=file&id=4078&act=down)

INTRODUCTION
Modular XML Loading (MXMLL) is a mod to the Civ4 Warlords DLL which allows the game to load and read multiple XML files of many types. It uses a standardized naming and organizing convention which has the promise of delivering full drag and drop modularity to the games content (almost). Properly configured modules will be able to alter the games default data or add new content such as a Tech, Wonder, building, Unit, Leader or whole Civs with such elements nested inside it. The end of tedious cutting and pasting to assemble Mod content will herald a new age in Civ4 Moding.

NEW IN THIS VERSION
3.0 is a deep re-write of MXMLL 2.0 itself an update and extension of modular loading 1.2 by Chinese American which can be found at http://forums.civfanatics.com/showthread.php?t=184907. Despite this all modules made under 2.0 remain compatible with this 3.0 version which removes the restriction in 2.0 which prevented a base element from referencing a module element. In addition moders can safely reference any XML element on any supported file or within a file rather then just those that were earlier in the loading order. Lastly the ability to read and write modular data to the games cache allows for faster loading both when using modules or not. Future version will seek to address the remaining shortcomings particularly in the area of Audio and Diplomacy text.

HOW IT WORKS
The mod operates entirely during the xml loading portion of the game. First the games "Base" content is added to a list of targeted files. Base contents are the games original XML files as they would otherwise be loaded meaning Custom Assets at the highest precedence, then the Mod if any and lastly Assets from the Civ4 directory. Next regular expression searching is done of the whole XML folder Mod folder and custom assets are both found and adding to the file list. All Tag names on these files are then hashed for future reference and the data indexes assigned. Then a second round of loading occurs and the remaining data is loaded into the predetermined indexes (as opposed to post-determined as was originally done). Modules load after Base content and can step-on aka replace Base data that has a matching Type Tag name, modules will also step on each other with the last one to load being the one seen in game. Module loading order is not affected by Custom Assets vs Mod, either could step on the other so simply avoid ever having two conflicting modules.

WHAT DOSN'T WORK
Unfortunately do the the nature of the LSystem.xml files for Buildings and Improvements its not possible to modularize these files because they don't pass through the SDK. Sound and Audio data has not been addressed yet either and several file types are linked with Enums in the SDK thus any extensions of these files would have no effect without a corresponding modification in the SDK. All other data both XML and art can be made into a module but without LPlot data the building or improvement will be invisible in the game, though it will have every other game play affect and display in the Pedia. A copy of a modified LSystem or instructions on how to modify it should be included in a Building or Improvement module and some notes made on the readme.txt. Similarly custom Diplomacy Text common in new Civs isn't supported due the the complex nature of this file. Both of these files will require traditional cut and paste.

SUPPORTED FILE TYPES
The following types of files are supported, in all instances the game will search and find files matching xml/*_TYPE.xml ware TYPE is one of the following strings.

CIV4ArtDefines_Unit
CIV4ArtDefines_Building
CIV4ArtDefines_Improvement
CIV4ArtDefines_Civilization
CIV4ArtDefines_Leaderhead
CIV4ArtDefines_Bonus

CIV4GameSpeedInfo
CIV4TurnTimerInfo
CIV4WorldInfo
CIV4ClimateInfo
CIV4SeaLevelInfo
CIV4AdvisorInfos
CIV4TerrainInfos
CIV4EraInfos
CIV4UnitClassInfos
CIV4SpecialistInfos
CIV4TechInfos
CIV4FeatureInfos
CIV4TraitInfos
CIV4GoodyInfo
CIV4HandicapInfo
CIV4CivicOptionInfos
CIV4UpkeepInfo
CIV4HurryInfo
CIV4SpecialBuildingInfos
CIV4ReligionInfo
CIV4CultureLevelInfo
CIV4BonusClassInfos
CIV4VictoryInfo
CIV4BonusInfos
CIV4BuildingClassInfos
CIV4BuildingInfos
CIV4SpecialUnitInfos
CIV4ImprovementInfos
CIV4CivicInfos
CIV4LeaderHeadInfos
CIV4ColorVals
CIV4PlayerColorInfos
CIV4EffectInfos
CIV4BuildInfos
CIV4UnitInfos
CIV4CivilizationInfos

CIV4GameText
GlobalDefines

NAMING RULES
The file naming system that must be used to get a modular xml file to load is strict but simple and universal. Because * is a wild card you can place a descriptive name at the front such as Ninja_CIV4UnitInfos.xml also their can be any number of intervening directories under the xml directory so data can be nested to your hearts content. This is in contrast to the games default loading which was and still is very rigid in its structure. Failure to have a file match the regular expression is the most likely reason for content to silently (no load warning) not show up in the game.

SCHEMAS
The main drawback of this directory flexibility is that every XML file still needs to be validated before loading and the games validation code is off limits to me. It demands the Schema be in the same directory as the file its validating, if not the loading fails and a warning is thrown which you can OK past to continue loading but their will then likely be subsequent "ITEM NOT FOUND" warnings as the other files associated with the module are left with dangling references. Their have been odd cross validation errors I cant explain arising from modules using schemas of the same name, thus I recommend that all module schemas be renamed with a simple "module name_" at the front to match the module the module their in. Remember update the header of the files that use that Schema. This is especially true for Modules that are going to use a new XML tag that that interacts with a modified DLL, the local Schema will allow your module to load under a unmoded DLL and will isolate the modified Schema from invalidating other XML files.

ORGANIZATION
Though not required I would strongly recommend content creators and moders use my organization proposals to keep everyone on the same page and make modules easy to use. Within the xml their will be two folders for each of the major data types, one for Modifications and one for new material called Custom. Things like a Civ which have dependent sub elements should have their sub elements nested within their packages so the single package contains everything necessary to add or remove the Civ from the game. In general content should be in the smallest pieces able to standalone such as a single Unit, Tech or Building. Only if multiple elements are inter-dependent should they be packaged together.

TEMPLATES
To help make the creation of modules easier I have included as set of templates for several of the major data types in the core download. These folders contain an empty copy of each fill that needs to be included to get a new element of that type into the game. A modified element only requires a copy of the modified file so for example to change the Pedia of a Unit and Its art Def you would need only a Text and ArtDef file. A typical template looks like this.


<?xml version="1.0"?>
<!-- edited with XMLSPY v2004 rel. 2 U (http://www.xmlspy.com) by Soren Johnson (Firaxis Games) -->
<!-- Sid Meier's Civilization 4 -->
<!-- Copyright Firaxis Games 2005 -->
<!-- -->
<!-- Tech Infos -->
<Civ4TechInfos xmlns="x-schema:CIV4TechnologiesSchema.xml">
<TechInfos>

</TechInfos>
</Civ4TechInfos>


EXAMPLES
As a separate download (at the CCCP Module Library Source Forge repository at http://sourceforge.net/projects/civ4ccp/) you can find a growing collection of modules of all types. These modules are free to download, use, modify and distribute. Moders are invited to submit new modules at the Home thread (see top) or by E-mail to me at impalerwrg@yahoo.com

MAKING MODULES
Because all previous mods have been in the "Monolithic" format, aka one file of each type, most new Modules will be made by extraction of existing content out of these mods or efforts to break them down completely into Modules. Another popular method will be from scratch construction especially with new Art assets that have just been released. Once you've chosen the type of module your making and have the contents at hand begin by select the appropriate Template folder, make a copy and rename it, use a good descriptive name that is unlikely to be used twice. Dump all the art files directly into the folder .nif's .dds's .kfm's .etc .etc. Then if your doing an extraction simply copy and paste the appropriate elements on to the template xml's. Rename each file by adding the Module name before the '_' also add it for the schemas. The ArtDef Info will need to be edited, change what ever file patway was being used before to "xml/TYPEOFMODULE/MODULENAME/FILENAME". TypeOfModule would be things like "Custom units" or "Modified Buildings", you might need a different organization thats more appropriate for your purposes but try to keep it organized. Some file types have Button pathways on them directly (Units) be sure to edit these as well. Pink/Magenta buttons result if the pathway is wrong, wrong .nif and .kfm files will appear blank or generate a python exception if Debugging is on. If your creating from scratch copy and paste from an original object that is a close match for what your making and edit it into what you want. Lastly discard any files you you didn't use, for example if making a Unique Unit you don't need to edit the UnitClass.xml so just discard it. Once your ready load up and look in the Civilopedia or start a game (don't try to run a save made without the module it will be incomputable). You may have to go through a few cycles of tweaking Art scales fixing broken pathways, balancing stats etc etc. When building up a large multi layered Module like a whole Civ its recommended that you start at the top level the Civ get it working and then add Leaders, Units and Buildings as sub-folders, this helps organize them and makes it easier to add and remove them later, be sure to take the parent directory into account when doing the art pathways. If any loading errors occur track them down by activating logging and examine the xml.log generated at load time, it will list each file loaded and what elements were found on them, errors are usually the result of copy paste errors or failure failure to properly reference the Schema. Finally fill out the readme so others who use your module can credit you and contact you, include any special configuration options or dependencies that will be needed.

CACHE LOADING
The 3.0 release includes the ability to load from the Cache like a normal unmodified DLL (this ability had been sacrificed in the 2.0 version). When cache loading is active the regular expression searching will be skipped for each file that can be loaded in this way (the 13 largest files comprising the bulk of the content), other file types will still search and load modules. When the game detects changes in Custom Assets it will conduct the full search and write the contents of all modules it finds into the Cache files, these will load on subsequent games at near the speed of the unmodified DLL's cache loading. It appears from testing that the game can in some case be tricked into falsely trying to cache load when Modules are added or removed so its recommended that you explicitly DE-activate cache loading after adding or removing modules to be on the safe side. If you get an error during Cache loading just reload and it should be fine.

SDK MERGING
This mod is highly mergable with other SDK mods. Not only dose it operate at a narrow time and place unlikely to interfere with any other code the files themselves are easy to combine because of the four files modified, three (CvXMLLoadUtility.h , CvXMLLoadUtility.cpp CvXMLLoadUtilitySet.cpp) are almost never touched in other mods and the third (CvInfos.cpp) has only a few modest additions in the XML readings. You are free to merge this code with your own so long as I am credited and the end user is given links to the SF Module library. Of particular interest is the Civilizeditor by Lopez which has great synergy with with mod.


CHANGELOG


1.0 - By Chinese American, Sep 4th, 2006
Reads partial xml files for *_CIV4ArtDefines_Unit.xml, *_CIV4UnitClassInfos.xml, *_CIV4UnitInfos.xml.

1.1 - By Chinese American, Sep, 2006
Will replace old types with new ones.
Reads partial xml files for *_CIV4ArtDefines_Building.xml, *_CIV4BuildingClassInfos.xml, *_CIV4BuildingInfos.xml.

1.2 - By Chinese American, Sep 9th, 2006
Workaround for bug where buildings get obscene happiness from new buildings.
Added reading techs, improvements, builds, traits, leaderheads, civilizations.
Reads partial xml files for *_Arts.xml, *_Classes.xml, and *_Infos.xml in xml\units\ and xml\buildings\.
Reads partial xml files for *_UnitArts.xml, *_UnitClasses.xml, *_UnitInfos.xml, and
*_Units.xml in xml\units\.
Reads partial xml files for *_BuildingArts.xml, *_BuildingClasses.xml, *_BuildingInfos.xml, and *_Buildings.xml in xml\buildings\.
Reads partial xml files for *_CIV4TechInfos.xml, *_TechInfos.xml, *_Techs.xml, and *_Infos.xml in xml\technologies.
Reads partial xm files for *_CIV4ImprovementInfos.xml, *_ImprovementInfos.xml and *_Improvements.xml in xml\terrain, and xml\art\*_CIV4ArtDefines_Improvement.xml and xml\terrain\*_ImprovementArts xml.
Reads partial xml files for *_CIV4BuildInfos.xml, *_BuildInfos.xml and *_Builds.xml in xml\units.
Reads partial xml files for *_CIV4CivilizationInfos.xml, *_CivilizationInfos.xml and *_Civilizations.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Civilization.xml and xml\civilizations\*_CivilizationArts.xml.
Reads partial xml files for *_CIV4LeaderheadInfos.xml, *_LeaderheadInfos.xml and *_Leaderheads.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Leaderhead.xml and xml\civilizations\*_LeaderheadArts.xml.

2.0 Initial Re-Release by Impaler[WrG] on Dec 20th, 2006
standardizes all file expressions to "xml/*_File.xml"
Adds support for following files
CIV4GameSpeedInfo, CIV4TurnTimerInfo, CIV4WorldInfo, CIV4ClimateInfo, CIV4SeaLevelInfo, CIV4AdvisorInfos, CIV4TerrainInfos, CIV4EraInfos, CIV4UnitClassInfos, CIV4SpecialistInfos, CIV4FeatureInfos, CIV4GoodyInfo, CIV4HandicapInfo, CIV4CivicOptionInfos, CIV4UpkeepInfo, CIV4HurryInfo, CIV4SpecialBuildingInfos, CIV4ReligionInfo, CIV4CultureLevelInfo, CIV4BonusClassInfos, CIV4VictoryInfo, CIV4BonusInfos, CIV4SpecialUnitInfos, CIV4CivicInfos, CIV4ColorVals, CIV4PlayerColorInfos, CIV4EffectInfos, CIV4GameText, GlobalDefines

3.0 Update by Impaler[WrG] Feb 1st, 2007
deep re-write adds caching of module data, universal cross file referencing, fixed incorrect error messages durring loading, many earlier functions by CA/Impaler[WrG] and Firaxis obsoleted and removed.

Modual Library v1.0

DOWNLOAD HERE >>> http://forums.civfanatics.com/uploads/73222/Modual_Library_v1.0.zip

Impalers Modified University - a pin point modification to the University

Sphinx and LeonardosWorkshop - two new Wonders previously seen in Kets composite mod

Medic - a new standalone unit based on resent artwork by asioasioasio

Aborigines - a whole civ from Civ Gold containing 2 leaders a Unique Unit and Unique Buildings

Impalers Terrain Mod - some slight modifications to desert, jungles (less terrible) and floodplains (nerfed but are now in more terrain)

To use place the modular loading DLL in Custom Assets and place a mod folder with its parent folder (Custom Units, Custom Wonders ect) in xml. Placing either inside of a mod folder and running the mod should work as well.

reserved for me

This is an awesome idea. It will make things litterally plug and play.

Beautiful work, I will definitly want to get this incorporated into FfH at some point.

Very cool Impaler... I will definately have to include this in the Civilizeditor and throw in an option to allow people to create modlets :D

Ok great. Time to try this one since I was having issues with chineese americans :) thanks for pointing it out :)... and of course thanks for all the hard work.

Update: I have now fully integrated this into my mod. So far it's working great! The unit I tested it with loads perfectly, and I can use that as a template for future units. The potential for this is incredible. Again Great work and Thank You!

My future plans for this mod are...

Squash any bugs that are found,

Expand to more file types, Formations for one will be included in the next update,

Continue to enlarge the Modual Library, I've been modularizing several of asiox3's Wonders for addition, I'll also see if I can get some Religion moduals made after Christmas. Civ Gold Civilizations files have already been preped by Perplexus and others so these can be processed quickly once he E-mails them to me.

Longer term plans involve a deeper re-organization of file loading which may be able to solve what I call the downhill reference problem. All files are loaded in a very specific order because each file can only reference data on files loaded before it. Techs come early and are followed by buildings and units bring up the rear with lots of other files sprinkled in their too. This puts a lot of fundamental limitations on what kind of mods can be made and how their XML files must be organized. My plan is to map all info tags from all files before proceeding with the remainining data loading, this will allow any file to reference any other file without restriction.

Theirs also potential for a feature that can be of assistence in Lopez's Civilizeditor, a Modual taging system. Basicaly every data opject would be taged with the string of the file pathway from which it originated from. The editor could extract this string to know from what modual or moduals any piece of game data was loaded from and then inteligently manipulate or display this data to the editor user.

Lastly a boolean switch to disable the loading of Base content for thouse total convertion mods out their, or some other method of using a modual to selectivly knock out an element.

;4910848']My future plans for this mod are...

Squash any bugs that are found,

Expand to more file types, Formations for one will be included in the next update,

Continue to enlarge the Modual Library, I've been modularizing several of asiox3's Wonders for addition, I'll also see if I can get some Religion moduals made after Christmas. Civ Gold Civilizations files have already been preped by Perplexus and others so these can be processed quickly once he E-mails them to me.

Longer term plans involve a deeper re-organization of file loading which may be able to solve what I call the downhill reference problem. All files are loaded in a very specific order because each file can only reference data on files loaded before it. Techs come early and are followed by buildings and units bring up the rear with lots of other files sprinkled in their too. This puts a lot of fundamental limitations on what kind of mods can be made and how their XML files must be organized. My plan is to map all info tags from all files before proceeding with the remainining data loading, this will allow any file to reference any other file without restriction.

Theirs also potential for a feature that can be of assistence in Lopez's Civilizeditor, a Modual taging system. Basicaly every data opject would be taged with the string of the file pathway from which it originated from. The editor could extract this string to know from what modual or moduals any piece of game data was loaded from and then inteligently manipulate or display this data to the editor user.

Lastly a boolean switch to disable the loading of Base content for thouse total convertion mods out their, or some other method of using a modual to selectivly knock out an element.

Might I suggest a "base" mod thats simply civ4/civ4 with warlords completly modularized? This will make it easier for people starting new mods to pick up the orginization, as well as make change to stock code.

On another note i put a big push for this on my mods thread - Hopefully other modders will see it and check it out. I'll report any bugs I might find :)

Oh yeah... when you upgrade could you do your best to ensure that people using the older versions wont have to re-do all their files :)

Full back compatability will be maintained I assure you, the core reg-expression code is stable and wont be changed, I'm just manipulationg the structures that it is nested inside so theirs no chance of breaking compatability. Also its possible to modify an existing game element by creating a modual with the same name (XML name that is). The new data will over-write the base data, thus its not nessary to modularize the games base data, also that would be a LOT of work. Example are good to but I will be using all new content for them to give the mod more desirability. The larger the library the greater labor savings for the mod maker, leading to more incorporation in mod DLL's, leading to more content distribution by modual..etc etc...upward network effect spiraling :goodjob:

Sounds good :)

So you know I modularized both the Palace and the Barbarian Civs with no issue (i'm going to be making major changes to both).

I did however find one small (and likely unfixable) bug. When an error message occurs due to the game not being able to find a particular piece of information (in the example where I found it I removed part of the Obolisk but missed one of the declarations) it will report the file to be the last read building file (in my case the palace) instead of the location where the missing data is. Just a heads up :)

Found another bug for ya...

If you have a Modular Technology, techs that aren't modular cannot be Dependant on it (either OR Dependant or AND Dependant --same issue occurs with both) -- just spent an hour tracking this one down due to misrepresented error messages lol :)

Not a huge deal for me since i'm modularizing my whole mod....but could cause issues if someone wanted to throw a tech in without modularizing any existing techs....

Sorry I should have mentioned that limitation, its one of the things the Flat loading I'm working on now would adress. Any other similar dependency should also be affected like Promotions and Improvments. I think I have a work around though, on the TechInfos.xml of the new tech your creating add a second TechInfo element AFTER the tech your adding. The second Info should be a copy of the existing tech you want to make dependent on your new tech with the new dependency added. This should solve the problem as the new tech will process first being at the top of the document and the replacement data will then over-write the original data and the reference should connect. Because Base data is loaded first a reference their will not find anything in a modual but a modual can replace the base data at a later point.

Thanks for the bug reports, I'll try to adress that bad bug reporting string in the next patch so people dont get sent on wild goose chases. I'd also like to develop a more detailed tutorial based on your feedback, their are certainly many details I have omited in hast. Also as you grow the number of moduals your using keep track of your loading times, I'm interested in seeing how much it is slowed under large numbers of moduals, if it becomes an issue Lopez's Civilizeditor will bail me out as it alredy has the ability to merge moduals together.

;4911789']Sorry I should have mentioned that limitation, its one of the things the Flat loading I'm working on now would adress. Any other similar dependency should also be affected like Promotions and Improvments. I think I have a work around though, on the TechInfos.xml of the new tech your creating add a second TechInfo element AFTER the tech your adding. The second Info should be a copy of the existing tech you want to make dependent on your new tech with the new dependency added. This should solve the problem as the new tech will process first being at the top of the document and the replacement data will then over-write the original data and the reference should connect. Because Base data is loaded first a reference their will not find anything in a modual but a modual can replace the base data at a later point.

Thanks for the bug reports, I'll try to adress that bad bug reporting string in the next patch so people dont get sent on wild goose chases. I'd also like to develop a more detailed tutorial based on your feedback, their are certainly many details I have omited in hast. Also as you grow the number of moduals your using keep track of your loading times, I'm interested in seeing how much it is slowed under large numbers of moduals, if it becomes an issue Lopez's Civilizeditor will bail me out as it alredy has the ability to merge moduals together.

Great :) - Just make sure you let me know which files your changing when you update also - I'd hate to go line-by-line to integrate a change into my source (I custom compiled instead of using your dll 'cause I have some of TheLopez's stuff included as well)

When I release the first version of my prototype in a few days feel free to download it and have a look at the way i'm doing things- did my best to follow your organization as best I could...i'm sure its not perfect though.

I'm currently at 10 Techs, 1 Building, and 2 Civs Modularized. No slow down so far. As a note to that I'm removing all foreign text from my modules since I have no linguistic skills and they make following the files more difficult.

Anything I can do to help count on me :)

Great idea, guys! Thank you. I hope it will be widely accepted and used by modding community.

;4899310']SCHEMAS
The main drawback of this directory flexibility is that every XML file still needs to be validated before loading and the games validation code is inacessable. It demands the Schema be in the same directory as the file its validating, if not the loading fails and a warning is thrown which you can OK past to continue loading but their will then likly be subsequent "ITEM NOT FOUND" warnings as the other files assosiated with the modual are left with dangling references. The nessary Schemas often end up being the largest part of a small modual so if zip size becomes an issue consider ommiting them. But if you do please place a readme in your modual listing what is nessary to make it run. Lastly because each file can be validated by a different schema if your using Modular loading with a DLL that adds new tags be shure to include the updated schema in any content that you want to express that new tag.

Does a schema need to be included even if the XML file should validate against the unmodified schema, i.e. does not need additional tags to "complete" the schema?

Yes, a Schema is required all the time though its possible to validate several files with the same schema if their residing in the same folder. The validation code is in the .exe and thus I cant alter it. As I continue to work on the 3.0 release (I'm skipping to 3.0 because its a very deep re-write at the code level even though people wont see much outward difference) I'll see if I can adress this issue and get the file to validate by the original schema if no schema is present in its file.

How difficult a process do you think it would be to make a small utility that scans the directories and looks at filenames, and automatically copies the needed files to each directory? It would reduce filesizes for transfers etc. -- Just a thought :)

I would LIke to See A new Forum Created for A Plug And Play Compatible Library

*hint hint THUNDERFALL

I'll spin off a thread of its own once the CivGold moduals are done, thats the crown jewel after all, I'll also have a lot of new Wonders of the world many of which will use my CCCP effects. Getting a new Forum made would be realy near but probably not essential, we already have forums dedicated to most of the major content types so the moduals could just be found in their respective forums.

I would LIke to See A new Forum Created for A Plug And Play Compatible Library

*hint hint THUNDERFALL

I have to agree with purplexus on this one modules will need there own download database <im thinkign there will be many modules <civilizations, units, buildings, hmmn thats a lot of files then theres terrain etc >

Note as of now there is no way to include buildings into a modular plug and play system :(

Every building will need a cut and paste section to include into the LPlotSystem.xml as well as a quick how to add that section. Perhaps someone will write this and put it at the top of a read me which is included with each and every building for a plug and play system or civilization with a building already included in the civ.

Impaler I have petitioned Thunderfall for a plug and play forum. It will require a sticky for your CCCP add on as well as a how to include the CCCP add-on (don't forget to let every one know the options that the CCCP can do).
categories for your forum may I suggest a separate one for a HOW TO:
and each of the following. CIVILIZATIONS... BUILDINGS... TECHS... UNITS... WONDERS... SOUNDPACKS... MISC...
May I also suggest a Library record be kept in a separate sticky if someone wants to take on this challenge?

The only good think about the LSystem is that it dose not blow up with a dangling reference (Translation: The LSystem can contain references to Buildings that arnt loaded without harmfull affects). We can thus build up a single LSystem with all know buildings on it and keep it in a toped thread for people to use.

Currently the modular loading the CCCP is behind that of the 2.0 release that I made earlier, it dosn't load as many of the smaller more obscure file types. The next release will be Modular loading 2.1 and CCCP 2.07 which will each contain the most advanced loading.

Also I've come across some issues with Schema's it seems that the game will use schemas found in moduals to validate stuff out side that modual, sometimes the base data and sometimes another modual. As long as all the Schemas are the same their were no issues but trying to use any kind of variety screws things up badly so I'm going to include instructions for re-naming schemas as well. Its very trivial, pre-pend the modual name to the Schema and update the schema line at the top of the file.

I'll also try to fix some of the bad error messages that have been poping up when something dosn't load right and try to add a means of skipping Base File content by use of Global Defines.

The modual Forum sounds interesting, it will be tricky though to set them up in such a way that they are not passed over and neglected. What we realy need are more modual makers to create a more extensive library.

Impaler, this is AWESOME work!

Some questions:
1. If I completely throw out the default units in my mod, does my module units file overwrite the original units? EG: default - Unit[0] = SETTLER, my mod - Unit[0] = RIFLEMAN (same with buildings, improvs, etc)
2. If I add a new tag to units, do I need to add the new tag to both the default schema and the module's schema, or just the module's schema?
3. If I have less items in the module file (EG: 21 units) do the default items over that number still exist (EG: units 22+)?

I'm sure I'll think of some others. But that's enough for now. :)

Dale

1) Original (Base) units are over-writen if and only if they they have the same name so <UNIT_SETTLER> in a Modual would replace the original settler without any modification to the Base file, single unit modifications are even easier to do then a full new element as you only need to include the files your modifying (typicaly the Big Info file). To get ride of a unit entirely you would have to use the traditional method of modifing and including a Big Civ4UnitInfos.xml file that has had that unit removed. An alternative is to modify just the UnitClass for that Unit and make the Default unit of the Class None, the unit would then be unbuildable but would still show up in the Pedia.

2) Include the Modified Schema in the modual but rename it "DCM_Artilery_UnitSchema.xml" or some such thing, and have the other XML files point at it, version numbers might also be a good idea if you plan to keep modifying that Schem in the future. It seems the game creates a cach of all Schema file it finds and if the schemas in the moduals have the same names but are different it can mis-apply them. Specific linking each modual with its own schema should allow to be as secure against incorrect validations and any future modifications by yourself or others.

3) The quantity of items in a modual is irrelivent because the data is sorted by Name, if its present in the Base file (which is loaded first) then it over-writes. If not it appends it and increments the total item count. Thus most moduals contain only a single child element on each of their xml files, the bare-bones to add say a new unit or Leader to the game. If multiple elements are present (for example a single Civ4UnitInfos.xml that holds the the Catapult, Treb, Artillery, Battleship witch have been modified to have artilery range) each child element will be processed on its own just as it would be if it was a seperate file so clumping together several elements is a handy way to reduce the amount of cutting and pasting nvolved especialy for modifications to large numbers of existing units. (New stuff is best served ala-cart so people can pick and choose)

P.S. Most of the core functionality I've just explained is realy coming from CA's code, specificaly the AddGlobalClassInfo function and some variations on it for specific hard to handle files. I created a nice wrapper function ProcessClassInfo around it to extend the loading to many many more file types, most importantly Text and Global Defines. The wrapper provides consistent regular expression file enumeration. The Schema stuff was the last major hurdle in getting full plug and play moduals and I figured that out mostly by trial and error. I'd be interesting in what you think of the code and how it could be improved.

I have had a though on this for a bit.... but what if someone were to totally strip the warlords of all civs and also make those modular too? then you have a base in which the game would work with and you just pick and chose which civ's you want in the game. This would make moding for those like FFH where everything is different easier to make.

Just an idea though?

Am I to understand that there are issues with diplomacy xml files?

I have had a though on this for a bit.... but what if someone were to totally strip the warlords of all civs and also make those modular too? then you have a base in which the game would work with and you just pick and chose which civ's you want in the game. This would make moding for those like FFH where everything is different easier to make.

Just an idea though?

Am I to understand that there are issues with diplomacy xml files?

Actually, just provide such a "blank template" CIV4CivilizationInfos.xml file and instruct module makers to include it if and only if they are planning to remove all (or most of) the original civs. The modual would then still use FFH_CIV4CivilizationInfos.xml or whatever, and both files can be in the same folder.

I think it will be very rare for anyone to try to merge two total conversion mods.

Yep that would be the easiest way to knock out all the existing Civs, the game would then load the modual/s and they would be the only data. The Blank file though would need to be in the traditional pathway and file name for examle xml/Civilizations/Civ4CivilizationInfos.xml

Actually, just provide such a "blank template" CIV4CivilizationInfos.xml file and instruct module makers to include it if and only if they are planning to remove all (or most of) the original civs. The modual would then still use FFH_CIV4CivilizationInfos.xml or whatever, and both files can be in the same folder.

I think it will be very rare for anyone to try to merge two total conversion mods.

I was thinking more in a sense such as this that if I wanted to make say for instance a North american map and have the north american tribes only fight it out. I would like to include the Americans the native american tribes and canada with mexico. So in this sense to have the american civ still in the game but as a modular only would be easier for such a scenerio to make. Know what I mean? know what I mean? Nudge nudge... wink wink???

I was thinking more in a sense such as this that if I wanted to make say for instance a North american map and have the north american tribes only fight it out. I would like to include the Americans the native american tribes and canada with mexico. So in this sense to have the american civ still in the game but as a modular only would be easier for such a scenerio to make. Know what I mean? know what I mean? Nudge nudge... wink wink???

No big deal. Now three files in your module:
CIV4CivilizationInfos.xml (blank template in traditional path)
America_Civ4CivilizationInfos.xml (copied America from original version.)
NorthAmercianCivs_Civ4CivilizationInfos.xml

No big deal. Now three files in your module:
CIV4CivilizationInfos.xml (blank template in traditional path)
America_Civ4CivilizationInfos.xml (copied America from original version.)
NorthAmercianCivs_Civ4CivilizationInfos.xml

This would work for sure! But I think what we are trying to accomplish correct me if I am wrong... But take the blank template and then just drag and drop all the civs you want to use in the scenario. it should be that easy although there is still and i think there always will be an issue with the Lplotsystem file.

Wouldn't it be cool if for now on someone creates a civ and if you want it in your game you just drag it into your custom assets folder? As it stands right now I would have to cut and paste every single xml and include them into the original .xml file

I think this is the most powerful tool yet to be built.. IMHO

What I would like to do with my original civ 4 files is set it up as a blank template period. and then just throw in the civ's that I like. The reason this is so valuable to me is this.... Having worked on the Civ Gold team a little bit... I noticed how much friggin work it was to add yet just one more civ. Now it should be just alter one file and VIOLA! done.

Wouldn't it be cool if for now on someone creates a civ and if you want it in your game you just drag it into your custom assets folder? As it stands right now I would have to cut and paste every single xml and include them into the original .xml file

I think this is the most powerful tool yet to be built.. IMHO

Yes, I think it is great too. As long as Impaler can follow the 80/20 rule (do only the work that solves 80% of the problems) and not try to solve every freak edge case of XML mod merging...

Very interesting impaler and chinese american! I will have to check this out when we are ready to merge other mods.

I'm doing several Wonder Moduals from art work made by a number of different artist, I've got most of the wonders from my earlier wonder mod pack converted to moduals along with some new ones too.

In the screen shoot below theirs

Leonardos Workshop, The Brandenburg Gate, The World Trade Center, The Gateway Arch, The CN Tower, BigBen, The Harbor Opera House, The White House, The Apollo Program and the Sphinx

On my to do list, The Porcalin Tower, The Arc de Triomphe, The Mausoleum, The Statue of Zeus, The Code of Hamurabi, The Terracotta Army, Petra, The Leaning Tower of Pisa and Neuschwanstein Castle

All these moduals will soon be uploaded to the CCCP in the Library Branch of the project as freely downloadable and plugable moduals, they will require the use of a modified LSystem file which will also be their.

I'm having a bit of trouble merging this with the FFH SDK.
// Start Addition Extended GlobalDefines by Impaler[WrG]
std::vector<CvString> files;
gDLL->enumerateFiles(files, "xml\\*_GlobalDefines.xml");
int i;
bool failure = false;
for(i=0;i<(int)files.size();i++)
{
if (!ReadGlobalDefines(files[i], cache))
{
failure = true;
}
}
if (failure)
{
return false;
}
// End Addition by Impaler[WrG]

gDLL->destroyCache(cache);
////////////////////////////////////////////////////////////////////////

It says here:
if (!ReadGlobalDefines(files[i], cache))

That:
CvXMLLoadUtilitySet.cpp:206: error C2664: 'CvXMLLoadUtility::ReadGlobalDefines' : cannot convert parameter 1 from 'std::allocator<_Ty>::value_type' to 'char *'
CvXMLLoadUtilitySet.cpp:5297: warning C4129: '*' : unrecognized character escape sequence
:: === Build finished: 1 errors, 1 warnings ===

I think it's having trouble converting a CvString (or whatever that is) to a C string, but I'm not sure. Is there a way to fix this?

The first error is due to not including a change to the functions protype on the .h which I aparently forgot to label (my bad) change the function protype on the .h too

DllExport bool ReadGlobalDefines(const TCHAR* szXMLFileName, CvCacheObject* cache);


and change the function definition on the .cpp too

bool CvXMLLoadUtility::ReadGlobalDefines(const TCHAR* szXMLFileName, CvCacheObject* cache)


The second unrecognized character thing is a harmless warning message caused by the * in the regular expession, I dont know a way to get rid of it but it wont cause any trouble.

Yep, that did it. It seems to work for the most part (see my sig), but a few things aren't quite working.

New promotions don't seem to accept other promotions as prerequisites (non-modular promotions still work fine), which means they can be gained in any order and they don't stack together (ex. Promotion I and Promotion II appear separately, rather than together). Is this fixable?

Its due to the way the game hashes all the names of thouse Promotions, youll experence similar behavior with Techs. Fortunatly my 3.0 code will remove this limitation by Hashing all names first before loading data, this will also allow universal referencing of any object by any other object, currently your restricted to only referencing something thats already loaded and the files load in a very precice order to make shure all these references are valid.

now that sounds usefull <ive been biding my tiem waiting on completion of the 3.0 version and next step in CEP and CIV Gold relearning XML coding and teaching myself python syntax <this old dog isnt going to be attempting SDK alteration and c++ while built on the programming languages i learned is well going to take me a goodly while to catch up on ..ie it will prolly be replaced by somethign else by the time i catch up >

3.0 version released get it on the first page!

;5055544']3.0 version released get it on the first page!

Thanks a heap

Okay, so if I created the file <MODS>/<ModName>/Assets/XML/Units/My_CIV4UnitInfos.xml would it be in the "right" directory? Meaning, would it take advantage of cache loading or not? Would it require a schema in the same directory or not?

Do you think Firaxis is going to include this thing?

Are we now to break it down to the smallest parts such in a civilization for instance there are what Soundpacks...Units... Flags...Unique Units... Cities...Leaderheads???....Etc.???? or Am I reading that wrong???

now that sounds usefull <ive been biding my tiem waiting on completion of the 3.0 version and next step in CEP and CIV Gold relearning XML coding and teaching myself python syntax <this old dog isnt going to be attempting SDK alteration and c++ while built on the programming languages i learned is well going to take me a goodly while to catch up on ..ie it will prolly be replaced by somethign else by the time i catch up >

Bah Fortran, Cobal, and Pascal aren't that far off C++

they are all still compilers

Bah Fortran, Cobal, and Pascal aren't that far off C++

they are all still compilers

haha okay but it has been 10 years since ive written in any of those languages its the newer syntax that im having a bit of difficulty with ...its new terms to utilize ..as to fortran its not a compilier or at least fortran 77 wasnt ..nor was cobal in the incarnation i learned..pascal was a compiler and yea im relearnign as i said its the syntax thats getting me <shortened versions of words instead fo the entire woord ie bool to denote boolean ...as an example not a big deal but takes an old dog like me time to relearn>

Are we now to break it down to the smallest parts such in a civilization for instance there are what Soundpacks...Units... Flags...Unique Units... Cities...Leaderheads???....Etc.???? or Am I reading that wrong???

check the exampel libraries impaler has made and yea its the smallest parts ..ie you have all all the files for a single unit in a small directory ..you can have many of the mini mod directories and they should all run together iot eliminates having to have a huge line by line item file that you have to change to add a unit ..you just add one unit and its done..<note i used units in the example but its just about any item thats doable like this if you dont like leader head a .just add leaderhead b can replace a or use them together ..no sweat>

Okay, so if I created the file <MODS>/<ModName>/Assets/XML/Units/My_CIV4UnitInfos.xml would it be in the "right" directory? Meaning, would it take advantage of cache loading or not? Would it require a schema in the same directory or not?

Do you think Firaxis is going to include this thing?

The DLL would load such a file and write the info to cache during loading but the better pathway would be ...

<MODS>/<ModName>/Assets/XML/Custom Units/My/My_CIV4UnitInfos.xml

As for inclusion we can only pray they pillage the code as it would solve the chicken and egg problem (Cant use a Module if you don't have the moded DLL and why both if theirs no content for it, why both to make your content modular if no one has this special DLL). Early adopters like yourselves will push the envelope by creating the modules that get the ball roling. If it starts to take off and becomes a de-facto standard it has a much better chance of being picked up officially.

Are we now to break it down to the smallest parts such in a civilization for instance there are what Soundpacks...Units... Flags...Unique Units... Cities...Leaderheads???....Etc.???? or Am I reading that wrong???

check the exampel libraries impaler has made and yea its the smallest parts ..ie you have all all the files for a single unit in a small directory ..you can have many of the mini mod directories and they should all run together iot eliminates having to have a huge line by line item file that you have to change to add a unit ..you just add one unit and its done..<note i used units in the example but its just about any item thats doable like this if you dont like leader head a .just add leaderhead b can replace a or use them together ..no sweat>

Morfydd has the right Idea, organize at the level of usable game elements, units, buildings, civics etc etc. Generally when I do a unit its got a single folder with all the file types in it and no sub-directories, I dont both with 'Art' or 'xml' because in total their are only about 8 files. Their might be situations ware it makes more sense to combine two dependent objects. For example I'm working on some Resource Modules and one is going to be for Sevo's Natural gas. Sevo made a new improvement for the gas "Pipelines" and modified the Offshore platform to access the resource as well. I would combine the modified Platform, the new Pipeline and the Gas Bosus all into one folder because none work without the others. That what I mean by "stand alone" you can load just that one module and it will work . When it comes to Civs I recommend that the UUnits, UBuildings and Leaderheads be nested inside the Civ because even though these thing could be made as separate modules and would load fine they cant be accessed/played-with without a Civ that calls on them. Nesting them inside a Civ help make the plug and play process easy and you makes it easier to swap the contents out later, say for example to switch around a Leaderhead or extract a UU and turn it into a standalone unit.

;5055887']The DLL would load such a file and write the info to cache during loading but the better pathway would be ...

<MODS>/<ModName>/Assets/XML/Custom Units/My/My_CIV4UnitInfos.xml

Right, but when I read your new post, it sounded like this "better pathway" would not be cached and would require the schema file.

Im just wondering if this is happening for anyone else.

So I build the directory as described. There isnt anything in the folders. I create a unit to test it out, it loads, everything is fine, yet for some reason The Sphinx, Leonardos Workshop, the Aboringal Civ and the Medic (i.e. all the examples you had) load as well.

This confuses me because the files for them aren't in any of the XML directories, I dont even have the files on my computer.

Anyway I download the examples, i Put them into the correct folders and load again, for some reason it doesnt load, just hangs. Anyway thats whats going on here.

I love the work, it makes things a hell of alot easier for me.

That sounds like your trying to load from cache after changing the Assets, always load un-cached after changing or switching out modules. The other examples were preserved in your cache between loads. Also remember it loads everything in Custom Assets and the Mod folder.

Just curious... how is the project going on the Civ Gold Modularization?
well I hope.!

Slowly at the present time, Wyz is concentrating on getting the new Civs made for CivGold4, making all the separate 1-civ modules is time consuming so I'm now thinking once CivGold4 is ready I will make a single "Super-Module" out of it in which which the files arn't divided into individual civs. This would lack the cherry-picking ability you would have to use all or none but it would be about a 100 times faster to make.

;5132530']Slowly at the present time, Wyz is concentrating on getting the new Civs made for CivGold4, making all the separate 1-civ modules is time consuming so I'm now thinking once CivGold4 is ready I will make a single "Super-Module" out of it in which which the files arn't divided into individual civs. This would lack the cherry-picking ability you would have to use all or none but it would be about a 100 times faster to make.

Time consuming!!!... TELL ME ABOUT IT!

as for making a super-module will it be as easy as dragging the files into a certain civs folder?

OR!!!

will that be even necessary?

I've been thinking of some improvements I could make to Modular Loading code. I've got two big ideas for next version

The foremost is to allow art file pathways to be determined dynamically at runtime. Currently your forced to use names like >xml/TYPE_OF_MODULE/MODULE_NAME/File.DDS< and the person using the module has to put it in that exact file pathway or the art file won't be found. The xml files in the module will load from any location in the xml folder but the art is so restrictive is strangling the inherent flexibility of the xml loading. Dynamic runtime artpathways would simply combine the file pathway of the xml file with then art sting so you would just have >File.DDS< on the ArtDef and so long as you put the art in the same folder as that xml file everything would work fine. Their would of course be a new boolean tag for each pathway that activates this feature so older modules remain backwards compatible and you can continue to mix the games original assets (like animations) with new ones (like NIF and Skins). Modules making use of this feature would have total flexibility in their pathway and the user could create their own directory structure that best fits their mod.

The other idea I'm a bit shakier on, I'm considering adding a new folder dirrectly under Assets called "Modules" and making it the only valid location for modular loading. This is mostly for conceptual clarity, modules are typically art+xml so they don't really fit into any existing directory well. Kicking modular loading out of the xml directory would force some updating of existing modules but it would be rather easy to do and I'm hoping that the run-time pathways help to smooth that transition. The principle advantage would be that mod makers can easily drag out or drag in this new directory to turn their modular content on or off rather then having to root around in the xml directory. I'm considering grandfathering in the xml directory too just to give people the option and maximize back compatibility which so far has never been broken. Any suggestions?

If you are looking for new ideas...

It would also be useful I think to have the possibility of deciding whether a file should be loaded from python. This could allow "meta mods" like Ruff's Cobbled SG Modpack to for example disable an XML mod like the included GP type mod.

There was someone once working on a project to fully expose arbitrary XML to python as well, so as to allow for new XML attributes without SDK modding.

Impaler what would the Load times be on modifying the CCCP DLL to go around searching for art be like. I imagine it would really increase this time as it would have many XML files and a lot of them have Art directions for them. If you take out the Art directions in these files YIKES!!!

I do like the Modules Folder idea a lot actually

I would also suggest to anyone who does use a module that they also have a modules storage folder too.

The run-time pathways aren't going to add anything (noticeable) to the load time, its just combining the string of the XML files directory (minus the xml file itself) with the string in the art tag. This feature will only work if the art is in the same folder as the xml file that references it which is already how all existing modules work anyways (at least the ones I've made), if for any reason you need to separate them just don't use this feature. Their wont be any searching beyond the search thats currently done for XML files via the regular expression search.

It would also be useful I think to have the possibility of deciding whether a file should be loaded from python. This could allow "meta mods" like Ruff's Cobbled SG Modpack to for example disable an XML mod like the included GP type mod.

There was someone once working on a project to fully expose arbitrary XML to python as well, so as to allow for new XML attributes without SDK modding

If were talking Python event triggered mods the Dr Elmer Jiggles Event Manager would be the thing to use, Ruff's mod uses an interesting option control panel to control a lot of python options but its all hard coded. I've been doing some brainstorming on extending or adapting this into some kind of registration based Option Manager much like the Event manager. Many INI file based python mods could relativity easily be switched to work with it in substitution. The Option manager itself would generate a UI for everything thats registered with it and could load the "default" configuration options from any source desired such as the INI parser or XML global Defines.

As for the global exposure of XML values to Python, I've heard this idea before but I'm just now beginning to think of a way this could be done, by a series of Hash maps on CvInfoBase. This could be loaded up with every tag or at least the ones that aren't nested and exposed to Python through some kind of call similar to the GC.getIndexforString call. I think this is doable but would probably result in a rather hefty load slowdown and all calls to the data will be slowed by the string interpretation.

;5188340']If were talking Python event triggered mods the Dr Elmer Jiggles Event Manager would be the thing to use, Ruff's mod uses an interesting option control panel to control a lot of python options but its all hard coded. I've been doing some brainstorming on extending or adapting this into some kind of registration based Option Manager much like the Event manager. Many INI file based python mods could relativity easily be switched to work with it in substitution. The Option manager itself would generate a UI for everything thats registered with it and could load the "default" configuration options from any source desired such as the INI parser or XML global Defines.

Most everything else can be already be done through python, I just need some exposed method call to set an XML file to load or not. I don't expect it to be fully dynamic given the way XML loads, just some way to tell the XML loader not to load a file before the load occurs. Then I just call it from python script before the XML loading takes place. Anyway it is just an idea, you can think about it.

;5188340']As for the global exposure of XML values to Python, I've heard this idea before but I'm just now beginning to think of a way this could be done, by a series of Hash maps on CvInfoBase. This could be loaded up with every tag or at least the ones that aren't nested and exposed to Python through some kind of call similar to the GC.getIndexforString call. I think this is doable but would probably result in a rather hefty load slowdown and all calls to the data will be slowed by the string interpretation.

Actually a good start is just something that is better than CvInfoBase.xmlValue(), so that I don't have to run a ridiculously slow python XML parser or come up with a complicated regex. Perhaps just a few short functions similar to CvInfoBase.getXMLStringByTagName(szTagName). Should be fairly quick if there are no C++ gotchas I've forgotten about.

So, CIV4PromotionInfos.xml is NOT Supported? Damn, thats xml file I use the MOST in my mod.

EDIT: Crap, actually that makes it no point to merge it, too much work for too little gain :( - and I was hoping I could split up skills/equipment/items/buffs/etc

Whats the reason promotions aren't supported?

There is any hope to have Audios XML suport on Module System???

Not at this time, Firaxis did a lot to extend modular ability to new file types in BtS but their wasn't time to do Audio. Its also one of the least frequently moded parts of the game it was rightfully low priority.