New Utility project in the works... YAModXMLEditor!

I am not quite sure where to put this because I want it to be open source so that anyone can work on it but I have never used an open source website before... so I am going to announce it here and get feedback on what to do/where to go. It is in essence a utility but it could also be considered a project.

I have started "Yet Another Mod XML Editor"... (there are a number of them out there... I am currently planning on calling it ModXMLEditor) The goal of this project, however seems a bit different from the ones I have seen. However, if one exists out there with the same goals, I would like to know about it.

I will update this as things flesh out but as of now these are the basic goals (more details to come):

WHAT the ModXMLEditor WILL DO:

1. The editor will be completely expansion independent. In other words it will be usable with Vanilla, Warlords, and BtS mods. As long as the game and it's expansion's have been installed onto the computer using the CD the program will not need any user input to find the location of the game files (It will read those locations from the registry). In the case that a user has altered the location, user input will be required (probably only once) to find the location of the original game. From there, the program should find the expansions (unless they have been moved from the Civ4 directory... in which case you will have to locate each one once).

2. The editor will be XML transparent. You will not need to have a sophisticated understanding of XML to use it (however, you will need to be familiar with any elements you want to modifiy). You will even be able to modify the Civ4 Schemas on the fly without really needing to know how (although the real reason for this is to save time, not to make it ultra user friendly).

3. The editor will not rely on or make use of any hard-coded properties. Instead, it will (heavily) rely on the schemas that are provided by Firaxis and mod developers. (Actually... most of its capabilities will be based on a behavior of tracking and comparing those schemas)

4. The editor will not be limited to which XML files it can alter properties of. ALL XML FILES* will be accessible and alterable by the editor and will be listed within it's menu's. In fact, theoretically it will be possible to add completely new custom made XML files and edit them. (One possible exception may be the text defines... at least for now).

5. XML properties will be color coded by expansion (where appropriate) and a specific color will single out any custom tags that have been added for the mod being worked on, making non-Firaxis tags easy to find.

6. Hard core mods that require adding tags/properties will be able to do so on the fly to all existing items of a particular info type at once (adding a property to units for example). I also hope to be able to allow one to set a default setting that is automatically used for units that already exist. Removal or changing of such tags should also be possible.

7. The program will allow copying items of any particular info type so that easy variations can be made (like with units for example).

8. New items can be made on the fly as well as unwanted items deleted.

9. I hope to also allow commenting of XML files where possible (may not be available at first).

10. Of course original files will be completely left alone.

11. Theoretically, it will require little, if any updating in terms of what is necessary to change for any patches that may still come out. Once again, the schemas themselves will provide the data necessary to make changes to your own files (unless some strange default is necessary).


WHAT IT WON'T DO:

1. It will not provide a graphical interface for any particular info file (such as a tech map). It will simply be a list of all modifiable properties and tags on one side and all infos within the file being edited on the other. (I may eventually however allow it to load dds textures for buttons/icons.)

2. It will not modify anything but XML and XML schemas.

3. It may not be the quickest loading utility. All schemas will be loaded on the fly, just like they are in the game. However, it will only be necessary to load actual data for the current file being worked on.


The project is based on C# and .NET 2.0 so a download may be necessary before it can be used.

When a working version is released, I will post a new thread in the requisite place...
Meanwhile, you can keep track of progress in the second post.


*As of 7/11/2011, all Vanilla files will open except for ones that use the following 2 schemas: Civ4LSystemSchema.xml AND Civ4DetailManagerSchema.xml for the following reasons:

1. LSystem files do not follow the typical schema format and developing reading rules for these particular files will be difficult, particularly for tracking changes.
2. The Civ4DetailsManagerSchema.xml file can be read, but the files using it do not follow the schema even though for some reason that fact slips by the game's own schema manager.

Project status: CURRENTLY INACTIVE - but reconsidering.

Ability to find installed location using the registry: DONE and TESTED
Ability to load and analyze schema structure and create data template for all info files DONE.
Ability to track changes (and additions) in schema in both expansions and mods DONE and TESTED
Ability to use inherited basic types and to define variable types using InfoTypes DONE** (see this post below)
Editor's ability to import an XML file for editing BASIC FUNCTIONALITY COMPLETE, not tested. Wanting to do UI so testing will be easier. See this post for details.
Editor UI: UNDER DEVELOPMENT

As of 7/11/2011 (hmm.... interesting day.... the 7/11 store chain should be holding a major promotion!):
Editor UI is basically working although color coding is not in effect yet. Need to finish support for tracking changes and linking those changes into the file loading engine.

I'll keep an eye on this. It should make my life in modding easier.

By the way... one other MAJOR capability that is theoretically possible with this utility is to automatically upgrade all the XML for a Mod from a lower expansion to a higher expansion without having to do any manual XML modification (for example from Warlords to BtS).

Such an upgrade however WILL require supplying defaults values for any custom Infos added by a mod for variables that are new to BtS if they can't be determined from the already supplied Infos.

Would there be anyway possible for it to work with modular files? Not so important really. Just curious what you think about the idea of possibly being an editor for WoC modular stuff. It would very useful if not regardless.

Technically... you can make the editor create files as if for BTS, and the other versions (Vanilla and Warlords) will just ignore the XML tags that they don't recognise.

johny smith said:

Would there be anyway possible for it to work with modular files?

Click to expand...

I have not worked with modular files yet but if I presume correctly that modular files must follow the format defined in the schema for the particular items being described, then yes. The schema provides the template for all data to be worked with (if my hopes for the editor are realized). If different schema's for the same item within a single mod are permitted by the game (such as two different schemas for unit files) then that could be a problem. I need some more info on this aspect if anyone has it.

Moomanof_ said:

Technically... you can make the editor create files as if for BTS, and the other versions (Vanilla and Warlords) will just ignore the XML tags that they don't recognise.

Click to expand...

Actually... if all goes well it will be able to deal with all versions of Civ. You must specify the edition you are working with in order for it to know which data format to use.

If you are working with Vanilla it will only load Vanilla mods. If you switch to BTS mode, it will only load BTS mods. It will recognize any tag that is defined in the schema.

This is a fairly complex project... a bit more than I realized it would be so don't be surprised if it goes slowly. Handling the schema's is easy enough. (I'm almost done with that part.) Right now though I am worried about how to organize the data from the info files so that it is easily interpreted correctly using the schema.

Once I have something to show, I plan on announcing it and doing a survey of how useful people would find it. If I get it working and enough support... I can make it an open source project where people can create UI plugins, contribute information, etc...

I am "done" programming the schema loading and analyzing capabilities. Now I need to test them. It might take a few days (or weeks if it doesn't work well) but this is the first major hurdle. Only one, possibly two more if this works! (Well... at least only two more before we have a working product to showcase.)

Hey, primem0ver, I sent you an email about a similar unannounced project I've been working on, but it might have gotten lost in your spam filter. Could you either look for it, or PM me and I'll give you my contact details?

primem0ver said:

I need some more info on this aspect if anyone has it.

Click to expand...

The schemas are not different. Nothing drastically different. The one thing that is different is how you would write the xml files. You only need the areas of the xml that are changed in the WoC modular format versus the entire xml type. If you look at the WoC Lite you can get a glimpse of what I am talking about.

For example here is how you could change the xml for the spread factor of a religion.


Anyway it can save some space and work. And again not the most important thing, but I think it could be a time saver if you could add something extra on top as an option to write only the changes in an xml file.

johny smith said:

The schemas are not different. Nothing drastically different. The one thing that is different is how you would write the xml files. You only need the areas of the xml that are changed in the WoC modular format versus the entire xml type. If you look at the WoC Lite you can get a glimpse of what I am talking about.

For example here is how you could change the xml for the spread factor of a religion.

Code:

<Civ4ReligionInfo xmlns="x-schema:CIV4GameInfoSchema.xml">
	<ReligionInfos>
		<ReligionInfo>
			<Type>RELIGION_CHRISTIANITY</Type>
                       <iSpreadFactor>10</iSpreadFactor>
		</ReligionInfo>
	</ReligionInfos>
</Civ4ReligionInfo>

Anyway it can save some space and work. And again not the most important thing, but I think it could be a time saver if you could add something extra on top as an option to write only the changes in an xml file.

Click to expand...

Then yes, it is possible. But we need to be clear on what schema's are. In Civ IV Schema xml files all end with the word "Schema" in the file name.

For instance, your example refers to the schema file "CIV4GameInfoSchema.xml" This file defines all the properties a Religion info can have. All religion infos in the game must follow the format specified in this file. The editor I am programming reads ALL of these schema files first and sets up a property table for all info's. It does this for each edition of the game.

Here is where I am unclear: Does modular XML loading allow multiple schema files to be defined for a single info? Is there for example a CIV4GameInfoSchema.xml file for one optional mod and another CIV4GameInfoSchema.xml for a second optional mod which define different properties? If so, then dealing with modular XML files will be a significant problem. If not, then there will be no problems at all and what you are asking about is possible.

primem0ver said:

Here is where I am unclear: Does modular XML loading allow multiple schema files to be defined for a single info? Is there for example a CIV4GameInfoSchema.xml file for one optional mod and another CIV4GameInfoSchema.xml for a second optional mod which define different properties? If so, then dealing with modular XML files will be a significant problem. If not, then there will be no problems at all and what you are asking about is possible.

Click to expand...

In the WoC modules folder there is a new file that is called MLF that tells the order to load the xml. Each one adds to the same info I believe. I am not the best at understanding the SDK. Sorry.

Regardless just having a plugin that generates xml in the format that removes anything mark with "0"'s or empty lines would be useful. I mean just a good editor for the previous non modular would be very good. I was only hoping only for a bonus a way for it to strip out empty lines and export into the WoC format. I was not thinking for it to read the modules xml really. Sorry again I am not the best person to ask. I hope I helped somewhat though.

johny smith said:

In the WoC modules folder there is a new file that is called MLF that tells the order to load the xml. Each one adds to the same info I believe. I am not the best at understanding the SDK. Sorry.

Click to expand...

load order doesn't affect schemas. My guess is that multiple schema's aren't allowed. It doesn't seem practical from either a programming perspective or a mod perspective. But maybe someone who knows will tell us.

If you use WoC loading you typically have multiple schema files which are different, however their name is also different (so Unitinfosschema, MODULE_unitinfosschema, ANOTHERMODULE_unitinfosschema, etc.). Theoretically you could have mulitle schemas which are different and have the same name, but this can cause XML load errors when starting the game.

It would be nice for WoC Modules if a config file stores the default value of all xml tags(initial value 0 for boolean/int, empty for string) or read it automatically from a cvinfos.cpp file (data is stored in this format pXML->GetChildXmlValByName(&m_iSameReligionAttitudeDivisor, "iSameReligionAttitudeDivisor", 30); , where 30 is the default and iSameReligionAttitudeDivisor is the XML tag)

Then if you flag a xml file as WoC Module you have an option to remove all tags with the default value.

Hmm... ok then. But keep in mind that schema files DO NOT store values of any kind (except how many times a given variable is expected to appear). They only store a required data structure that an info file must follow. If you are talking about simply importing different default values, these values are not obtained from schema files (unless BtS decided to use schema files in an unconventional way... haven't really looked at BtS schema files yet).

From a programming perspective, all extra values that a mod adds must be dealt with by the SDK (such as WoC's version of the dll). So unless we are defining variables in the schema that are not required to be present for the game to work, then there is very little point for including multiple schema's. Also... it is possible to create a variable in a schema and not have it be required, depending on how the schema is set up. So in the end, I am not sure I understand the reason for allowing multiple schema's.

I might be able to add the capability (later) of recognizing and using different schemas. However, two conditions will need to be met in order for this to work:

1. Schemas for the same type of info must have different names (you already do this).

2. The info files that use the altered schema MUST refer to the altered schema file they are using.

This is a feature that will wait for a convenient time if it becomes necessary.

PRODUCT UPDATE:
The schema recognition aspect has been tested and seems to be working properly. Tonight I will test the ability of tracking changes made by an expansion.

I just want to say anything WoC would be nice as an extra. It is not necessary. I do not want to get you sidetracked with too much WoC xml stuff.

johny smith said:

I just want to say anything WoC would be nice as an extra. It is not necessary. I do not want to get you sidetracked with too much WoC xml stuff.

Click to expand...

I fully agree with this. I guess there are already many waiting for this tool

WoC Modules basically work just like any other XML, so you could easily add any special support when you are finished.

In a typical WoC Module only information is listed, which is different from the default value defined in the SDK to allow the copynondefaults for WoC. If I understand johny smith correct, what we would like to see is a feature that removes information (XML tags with default value) from an XML file. You can't read the default value from schema or infos file, but you can read it from the SDK source.

Sephi said:

I fully agree with this. I guess there are already many waiting for this tool

WoC Modules basically work just like any other XML, so you could easily add any special support when you are finished.

In a typical WoC Module only information is listed, which is different from the default value defined in the SDK to allow the copynondefaults for WoC. If I understand johny smith correct, what we would like to see is a feature that removes information (XML tags with default value) from an XML file. You can't read the default value from schema or infos file, but you can read it from the SDK source.

Click to expand...

Yes that is all I would like to see.

And again WoC should come after a working editor if at all. Just wanted to add it as a note for perhaps for something extra later down the line.

Perhaps a plugin would be best for dealing with certain aspects of WoC then.

I have tested the application's ability to build a table of and track all variable types defined within all schemas within Civ4 and it's expansions. So far it seems to be working for the most part. Only two files seem to have troubles that don't really matter much at this point (but I will fix the problems eventually):

Civ4LSystemSchema.xml (gee... what a surprise there!)
Civ4DetailManagerSchema.xml

Next on the agenda is refining the variable typing system. (Easy hurdle compared to the last step... see descriptive post below).