1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

[SDKMOD] Modular XML Loading 2.0

Discussion in 'Civ4 - Mod Components' started by Impaler[WrG], Dec 20, 2006.

  1. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    MXMLL 3.0 by Impaler[WrG] & Chinese American

    DOWNLOAD
    GET MODULAR XML LOADING 3.0 DLL, Source Code, Templates and Instructions HERE

    INTRODUCTION
    Modular XML Loading (MXMLL) is a mod to the Civ4 Warlords DLL which allows the game to load and read multiple XML files of many types. It uses a standardized naming and organizing convention which has the promise of delivering full drag and drop modularity to the games content (almost). Properly configured modules will be able to alter the games default data or add new content such as a Tech, Wonder, building, Unit, Leader or whole Civs with such elements nested inside it. The end of tedious cutting and pasting to assemble Mod content will herald a new age in Civ4 Moding.

    NEW IN THIS VERSION
    3.0 is a deep re-write of MXMLL 2.0 itself an update and extension of modular loading 1.2 by Chinese American which can be found at http://forums.civfanatics.com/showthread.php?t=184907. Despite this all modules made under 2.0 remain compatible with this 3.0 version which removes the restriction in 2.0 which prevented a base element from referencing a module element. In addition moders can safely reference any XML element on any supported file or within a file rather then just those that were earlier in the loading order. Lastly the ability to read and write modular data to the games cache allows for faster loading both when using modules or not. Future version will seek to address the remaining shortcomings particularly in the area of Audio and Diplomacy text.

    HOW IT WORKS
    The mod operates entirely during the xml loading portion of the game. First the games "Base" content is added to a list of targeted files. Base contents are the games original XML files as they would otherwise be loaded meaning Custom Assets at the highest precedence, then the Mod if any and lastly Assets from the Civ4 directory. Next regular expression searching is done of the whole XML folder Mod folder and custom assets are both found and adding to the file list. All Tag names on these files are then hashed for future reference and the data indexes assigned. Then a second round of loading occurs and the remaining data is loaded into the predetermined indexes (as opposed to post-determined as was originally done). Modules load after Base content and can step-on aka replace Base data that has a matching Type Tag name, modules will also step on each other with the last one to load being the one seen in game. Module loading order is not affected by Custom Assets vs Mod, either could step on the other so simply avoid ever having two conflicting modules.

    WHAT DOSN'T WORK
    Unfortunately do the the nature of the LSystem.xml files for Buildings and Improvements its not possible to modularize these files because they don't pass through the SDK. Sound and Audio data has not been addressed yet either and several file types are linked with Enums in the SDK thus any extensions of these files would have no effect without a corresponding modification in the SDK. All other data both XML and art can be made into a module but without LPlot data the building or improvement will be invisible in the game, though it will have every other game play affect and display in the Pedia. A copy of a modified LSystem or instructions on how to modify it should be included in a Building or Improvement module and some notes made on the readme.txt. Similarly custom Diplomacy Text common in new Civs isn't supported due the the complex nature of this file. Both of these files will require traditional cut and paste.

    SUPPORTED FILE TYPES
    The following types of files are supported, in all instances the game will search and find files matching xml/*_TYPE.xml ware TYPE is one of the following strings.

    CIV4ArtDefines_Unit
    CIV4ArtDefines_Building
    CIV4ArtDefines_Improvement
    CIV4ArtDefines_Civilization
    CIV4ArtDefines_Leaderhead
    CIV4ArtDefines_Bonus

    CIV4GameSpeedInfo
    CIV4TurnTimerInfo
    CIV4WorldInfo
    CIV4ClimateInfo
    CIV4SeaLevelInfo
    CIV4AdvisorInfos
    CIV4TerrainInfos
    CIV4EraInfos
    CIV4UnitClassInfos
    CIV4SpecialistInfos
    CIV4TechInfos
    CIV4FeatureInfos
    CIV4TraitInfos
    CIV4GoodyInfo
    CIV4HandicapInfo
    CIV4CivicOptionInfos
    CIV4UpkeepInfo
    CIV4HurryInfo
    CIV4SpecialBuildingInfos
    CIV4ReligionInfo
    CIV4CultureLevelInfo
    CIV4BonusClassInfos
    CIV4VictoryInfo
    CIV4BonusInfos
    CIV4BuildingClassInfos
    CIV4BuildingInfos
    CIV4SpecialUnitInfos
    CIV4ImprovementInfos
    CIV4CivicInfos
    CIV4LeaderHeadInfos
    CIV4ColorVals
    CIV4PlayerColorInfos
    CIV4EffectInfos
    CIV4BuildInfos
    CIV4UnitInfos
    CIV4CivilizationInfos

    CIV4GameText
    GlobalDefines

    NAMING RULES
    The file naming system that must be used to get a modular xml file to load is strict but simple and universal. Because * is a wild card you can place a descriptive name at the front such as Ninja_CIV4UnitInfos.xml also their can be any number of intervening directories under the xml directory so data can be nested to your hearts content. This is in contrast to the games default loading which was and still is very rigid in its structure. Failure to have a file match the regular expression is the most likely reason for content to silently (no load warning) not show up in the game.

    SCHEMAS
    The main drawback of this directory flexibility is that every XML file still needs to be validated before loading and the games validation code is off limits to me. It demands the Schema be in the same directory as the file its validating, if not the loading fails and a warning is thrown which you can OK past to continue loading but their will then likely be subsequent "ITEM NOT FOUND" warnings as the other files associated with the module are left with dangling references. Their have been odd cross validation errors I cant explain arising from modules using schemas of the same name, thus I recommend that all module schemas be renamed with a simple "module name_" at the front to match the module the module their in. Remember update the header of the files that use that Schema. This is especially true for Modules that are going to use a new XML tag that that interacts with a modified DLL, the local Schema will allow your module to load under a unmoded DLL and will isolate the modified Schema from invalidating other XML files.

    ORGANIZATION
    Though not required I would strongly recommend content creators and moders use my organization proposals to keep everyone on the same page and make modules easy to use. Within the xml their will be two folders for each of the major data types, one for Modifications and one for new material called Custom. Things like a Civ which have dependent sub elements should have their sub elements nested within their packages so the single package contains everything necessary to add or remove the Civ from the game. In general content should be in the smallest pieces able to standalone such as a single Unit, Tech or Building. Only if multiple elements are inter-dependent should they be packaged together.

    TEMPLATES
    To help make the creation of modules easier I have included as set of templates for several of the major data types in the core download. These folders contain an empty copy of each fill that needs to be included to get a new element of that type into the game. A modified element only requires a copy of the modified file so for example to change the Pedia of a Unit and Its art Def you would need only a Text and ArtDef file. A typical template looks like this.

    Code:
    <?xml version="1.0"?>
    <!-- edited with XMLSPY v2004 rel. 2 U ([url]http://www.xmlspy.com[/url]) by Soren Johnson (Firaxis Games) -->
    <!-- Sid Meier's Civilization 4 -->
    <!-- Copyright Firaxis Games 2005 -->
    <!-- -->
    <!-- Tech Infos -->
    <Civ4TechInfos xmlns="x-schema:CIV4TechnologiesSchema.xml">
    	<TechInfos>
    		
    	</TechInfos>
    </Civ4TechInfos>
    
    EXAMPLES
    As a separate download (at the CCCP Module Library Source Forge repository at http://sourceforge.net/projects/civ4ccp/) you can find a growing collection of modules of all types. These modules are free to download, use, modify and distribute. Moders are invited to submit new modules at the Home thread (see top) or by E-mail to me at impalerwrg@yahoo.com

    MAKING MODULES
    Because all previous mods have been in the "Monolithic" format, aka one file of each type, most new Modules will be made by extraction of existing content out of these mods or efforts to break them down completely into Modules. Another popular method will be from scratch construction especially with new Art assets that have just been released. Once you've chosen the type of module your making and have the contents at hand begin by select the appropriate Template folder, make a copy and rename it, use a good descriptive name that is unlikely to be used twice. Dump all the art files directly into the folder .nif's .dds's .kfm's .etc .etc. Then if your doing an extraction simply copy and paste the appropriate elements on to the template xml's. Rename each file by adding the Module name before the '_' also add it for the schemas. The ArtDef Info will need to be edited, change what ever file patway was being used before to "xml/TYPEOFMODULE/MODULENAME/FILENAME". TypeOfModule would be things like "Custom units" or "Modified Buildings", you might need a different organization thats more appropriate for your purposes but try to keep it organized. Some file types have Button pathways on them directly (Units) be sure to edit these as well. Pink/Magenta buttons result if the pathway is wrong, wrong .nif and .kfm files will appear blank or generate a python exception if Debugging is on. If your creating from scratch copy and paste from an original object that is a close match for what your making and edit it into what you want. Lastly discard any files you you didn't use, for example if making a Unique Unit you don't need to edit the UnitClass.xml so just discard it. Once your ready load up and look in the Civilopedia or start a game (don't try to run a save made without the module it will be incomputable). You may have to go through a few cycles of tweaking Art scales fixing broken pathways, balancing stats etc etc. When building up a large multi layered Module like a whole Civ its recommended that you start at the top level the Civ get it working and then add Leaders, Units and Buildings as sub-folders, this helps organize them and makes it easier to add and remove them later, be sure to take the parent directory into account when doing the art pathways. If any loading errors occur track them down by activating logging and examine the xml.log generated at load time, it will list each file loaded and what elements were found on them, errors are usually the result of copy paste errors or failure failure to properly reference the Schema. Finally fill out the readme so others who use your module can credit you and contact you, include any special configuration options or dependencies that will be needed.

    CACHE LOADING
    The 3.0 release includes the ability to load from the Cache like a normal unmodified DLL (this ability had been sacrificed in the 2.0 version). When cache loading is active the regular expression searching will be skipped for each file that can be loaded in this way (the 13 largest files comprising the bulk of the content), other file types will still search and load modules. When the game detects changes in Custom Assets it will conduct the full search and write the contents of all modules it finds into the Cache files, these will load on subsequent games at near the speed of the unmodified DLL's cache loading. It appears from testing that the game can in some case be tricked into falsely trying to cache load when Modules are added or removed so its recommended that you explicitly DE-activate cache loading after adding or removing modules to be on the safe side. If you get an error during Cache loading just reload and it should be fine.

    SDK MERGING
    This mod is highly mergable with other SDK mods. Not only dose it operate at a narrow time and place unlikely to interfere with any other code the files themselves are easy to combine because of the four files modified, three (CvXMLLoadUtility.h , CvXMLLoadUtility.cpp CvXMLLoadUtilitySet.cpp) are almost never touched in other mods and the third (CvInfos.cpp) has only a few modest additions in the XML readings. You are free to merge this code with your own so long as I am credited and the end user is given links to the SF Module library. Of particular interest is the Civilizeditor by Lopez which has great synergy with with mod.


    CHANGELOG
    Spoiler :


    1.0 - By Chinese American, Sep 4th, 2006
    Reads partial xml files for *_CIV4ArtDefines_Unit.xml, *_CIV4UnitClassInfos.xml, *_CIV4UnitInfos.xml.

    1.1 - By Chinese American, Sep, 2006
    Will replace old types with new ones.
    Reads partial xml files for *_CIV4ArtDefines_Building.xml, *_CIV4BuildingClassInfos.xml, *_CIV4BuildingInfos.xml.

    1.2 - By Chinese American, Sep 9th, 2006
    Workaround for bug where buildings get obscene happiness from new buildings.
    Added reading techs, improvements, builds, traits, leaderheads, civilizations.
    Reads partial xml files for *_Arts.xml, *_Classes.xml, and *_Infos.xml in xml\units\ and xml\buildings\.
    Reads partial xml files for *_UnitArts.xml, *_UnitClasses.xml, *_UnitInfos.xml, and
    *_Units.xml in xml\units\.
    Reads partial xml files for *_BuildingArts.xml, *_BuildingClasses.xml, *_BuildingInfos.xml, and *_Buildings.xml in xml\buildings\.
    Reads partial xml files for *_CIV4TechInfos.xml, *_TechInfos.xml, *_Techs.xml, and *_Infos.xml in xml\technologies.
    Reads partial xm files for *_CIV4ImprovementInfos.xml, *_ImprovementInfos.xml and *_Improvements.xml in xml\terrain, and xml\art\*_CIV4ArtDefines_Improvement.xml and xml\terrain\*_ImprovementArts xml.
    Reads partial xml files for *_CIV4BuildInfos.xml, *_BuildInfos.xml and *_Builds.xml in xml\units.
    Reads partial xml files for *_CIV4CivilizationInfos.xml, *_CivilizationInfos.xml and *_Civilizations.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Civilization.xml and xml\civilizations\*_CivilizationArts.xml.
    Reads partial xml files for *_CIV4LeaderheadInfos.xml, *_LeaderheadInfos.xml and *_Leaderheads.xml in xml\civilizations\, and xml\art\*_CIV4ArtDefines_Leaderhead.xml and xml\civilizations\*_LeaderheadArts.xml.

    2.0 Initial Re-Release by Impaler[WrG] on Dec 20th, 2006
    standardizes all file expressions to "xml/*_File.xml"
    Adds support for following files
    CIV4GameSpeedInfo, CIV4TurnTimerInfo, CIV4WorldInfo, CIV4ClimateInfo, CIV4SeaLevelInfo, CIV4AdvisorInfos, CIV4TerrainInfos, CIV4EraInfos, CIV4UnitClassInfos, CIV4SpecialistInfos, CIV4FeatureInfos, CIV4GoodyInfo, CIV4HandicapInfo, CIV4CivicOptionInfos, CIV4UpkeepInfo, CIV4HurryInfo, CIV4SpecialBuildingInfos, CIV4ReligionInfo, CIV4CultureLevelInfo, CIV4BonusClassInfos, CIV4VictoryInfo, CIV4BonusInfos, CIV4SpecialUnitInfos, CIV4CivicInfos, CIV4ColorVals, CIV4PlayerColorInfos, CIV4EffectInfos, CIV4GameText, GlobalDefines

    3.0 Update by Impaler[WrG] Feb 1st, 2007
    deep re-write adds caching of module data, universal cross file referencing, fixed incorrect error messages durring loading, many earlier functions by CA/Impaler[WrG] and Firaxis obsoleted and removed.
     
  2. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    Modual Library v1.0

    DOWNLOAD HERE >>> http://forums.civfanatics.com/uploads/73222/Modual_Library_v1.0.zip

    Impalers Modified University - a pin point modification to the University

    Sphinx and LeonardosWorkshop - two new Wonders previously seen in Kets composite mod

    Medic - a new standalone unit based on resent artwork by asioasioasio

    Aborigines - a whole civ from Civ Gold containing 2 leaders a Unique Unit and Unique Buildings

    Impalers Terrain Mod - some slight modifications to desert, jungles (less terrible) and floodplains (nerfed but are now in more terrain)

    To use place the modular loading DLL in Custom Assets and place a mod folder with its parent folder (Custom Units, Custom Wonders ect) in xml. Placing either inside of a mod folder and running the mod should work as well.
     
  3. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    reserved for me
     
  4. Edgecrusher

    Edgecrusher Chieftain

    Joined:
    Aug 7, 2006
    Messages:
    915
    Location:
    Massachusetts
    This is an awesome idea. It will make things litterally plug and play.
     
  5. Kael

    Kael Chieftain

    Joined:
    May 6, 2002
    Messages:
    17,401
    Location:
    Ohio
    Beautiful work, I will definitly want to get this incorporated into FfH at some point.
     
  6. TheLopez

    TheLopez Chieftain

    Joined:
    Jan 16, 2006
    Messages:
    2,525
    Location:
    Oregon
    Very cool Impaler... I will definately have to include this in the Civilizeditor and throw in an option to allow people to create modlets :D
     
  7. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    Ok great. Time to try this one since I was having issues with chineese americans :) thanks for pointing it out :)... and of course thanks for all the hard work.

    Update: I have now fully integrated this into my mod. So far it's working great! The unit I tested it with loads perfectly, and I can use that as a template for future units. The potential for this is incredible. Again Great work and Thank You!
     
  8. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    My future plans for this mod are...

    Squash any bugs that are found,

    Expand to more file types, Formations for one will be included in the next update,

    Continue to enlarge the Modual Library, I've been modularizing several of asiox3's Wonders for addition, I'll also see if I can get some Religion moduals made after Christmas. Civ Gold Civilizations files have already been preped by Perplexus and others so these can be processed quickly once he E-mails them to me.

    Longer term plans involve a deeper re-organization of file loading which may be able to solve what I call the downhill reference problem. All files are loaded in a very specific order because each file can only reference data on files loaded before it. Techs come early and are followed by buildings and units bring up the rear with lots of other files sprinkled in their too. This puts a lot of fundamental limitations on what kind of mods can be made and how their XML files must be organized. My plan is to map all info tags from all files before proceeding with the remainining data loading, this will allow any file to reference any other file without restriction.

    Theirs also potential for a feature that can be of assistence in Lopez's Civilizeditor, a Modual taging system. Basicaly every data opject would be taged with the string of the file pathway from which it originated from. The editor could extract this string to know from what modual or moduals any piece of game data was loaded from and then inteligently manipulate or display this data to the editor user.

    Lastly a boolean switch to disable the loading of Base content for thouse total convertion mods out their, or some other method of using a modual to selectivly knock out an element.
     
  9. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    Might I suggest a "base" mod thats simply civ4/civ4 with warlords completly modularized? This will make it easier for people starting new mods to pick up the orginization, as well as make change to stock code.

    On another note i put a big push for this on my mods thread - Hopefully other modders will see it and check it out. I'll report any bugs I might find :)

    Oh yeah... when you upgrade could you do your best to ensure that people using the older versions wont have to re-do all their files :)
     
  10. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    Full back compatability will be maintained I assure you, the core reg-expression code is stable and wont be changed, I'm just manipulationg the structures that it is nested inside so theirs no chance of breaking compatability. Also its possible to modify an existing game element by creating a modual with the same name (XML name that is). The new data will over-write the base data, thus its not nessary to modularize the games base data, also that would be a LOT of work. Example are good to but I will be using all new content for them to give the mod more desirability. The larger the library the greater labor savings for the mod maker, leading to more incorporation in mod DLL's, leading to more content distribution by modual..etc etc...upward network effect spiraling :goodjob:
     
  11. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    Sounds good :)

    So you know I modularized both the Palace and the Barbarian Civs with no issue (i'm going to be making major changes to both).

    I did however find one small (and likely unfixable) bug. When an error message occurs due to the game not being able to find a particular piece of information (in the example where I found it I removed part of the Obolisk but missed one of the declarations) it will report the file to be the last read building file (in my case the palace) instead of the location where the missing data is. Just a heads up :)
     
  12. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    Found another bug for ya...

    If you have a Modular Technology, techs that aren't modular cannot be Dependant on it (either OR Dependant or AND Dependant --same issue occurs with both) -- just spent an hour tracking this one down due to misrepresented error messages lol :)

    Not a huge deal for me since i'm modularizing my whole mod....but could cause issues if someone wanted to throw a tech in without modularizing any existing techs....
     
  13. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    Sorry I should have mentioned that limitation, its one of the things the Flat loading I'm working on now would adress. Any other similar dependency should also be affected like Promotions and Improvments. I think I have a work around though, on the TechInfos.xml of the new tech your creating add a second TechInfo element AFTER the tech your adding. The second Info should be a copy of the existing tech you want to make dependent on your new tech with the new dependency added. This should solve the problem as the new tech will process first being at the top of the document and the replacement data will then over-write the original data and the reference should connect. Because Base data is loaded first a reference their will not find anything in a modual but a modual can replace the base data at a later point.

    Thanks for the bug reports, I'll try to adress that bad bug reporting string in the next patch so people dont get sent on wild goose chases. I'd also like to develop a more detailed tutorial based on your feedback, their are certainly many details I have omited in hast. Also as you grow the number of moduals your using keep track of your loading times, I'm interested in seeing how much it is slowed under large numbers of moduals, if it becomes an issue Lopez's Civilizeditor will bail me out as it alredy has the ability to merge moduals together.
     
  14. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    Great :) - Just make sure you let me know which files your changing when you update also - I'd hate to go line-by-line to integrate a change into my source (I custom compiled instead of using your dll 'cause I have some of TheLopez's stuff included as well)

    When I release the first version of my prototype in a few days feel free to download it and have a look at the way i'm doing things- did my best to follow your organization as best I could...i'm sure its not perfect though.

    I'm currently at 10 Techs, 1 Building, and 2 Civs Modularized. No slow down so far. As a note to that I'm removing all foreign text from my modules since I have no linguistic skills and they make following the files more difficult.

    Anything I can do to help count on me :)
     
  15. NeverMind

    NeverMind Proud to be Russian

    Joined:
    Apr 29, 2005
    Messages:
    618
    Gender:
    Male
    Location:
    Moscow, Russia
    Great idea, guys! Thank you. I hope it will be widely accepted and used by modding community.
     
  16. Gaurav

    Gaurav Chieftain

    Joined:
    May 14, 2006
    Messages:
    402
    Does a schema need to be included even if the XML file should validate against the unmodified schema, i.e. does not need additional tags to "complete" the schema?
     
  17. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    Yes, a Schema is required all the time though its possible to validate several files with the same schema if their residing in the same folder. The validation code is in the .exe and thus I cant alter it. As I continue to work on the 3.0 release (I'm skipping to 3.0 because its a very deep re-write at the code level even though people wont see much outward difference) I'll see if I can adress this issue and get the file to validate by the original schema if no schema is present in its file.
     
  18. Sarke

    Sarke Chieftain

    Joined:
    Jun 1, 2006
    Messages:
    101
    Location:
    Long Beach, CA, USA
    How difficult a process do you think it would be to make a small utility that scans the directories and looks at filenames, and automatically copies the needed files to each directory? It would reduce filesizes for transfers etc. -- Just a thought :)
     
  19. purplexus

    purplexus Chieftain

    Joined:
    Feb 13, 2002
    Messages:
    477
    Location:
    Calgary Alberta, Canada
    I would LIke to See A new Forum Created for A Plug And Play Compatible Library

    *hint hint THUNDERFALL
     
  20. Impaler[WrG]

    Impaler[WrG] Civ4:Col UI programmer

    Joined:
    Dec 5, 2005
    Messages:
    1,750
    Location:
    Vallejo, California
    I'll spin off a thread of its own once the CivGold moduals are done, thats the crown jewel after all, I'll also have a lot of new Wonders of the world many of which will use my CCCP effects. Getting a new Forum made would be realy near but probably not essential, we already have forums dedicated to most of the major content types so the moduals could just be found in their respective forums.
     

Share This Page