[WIP] Mod editor (XML)

[Senap]

Release 2009-05-04

CivSmith v0.3 has been released! Please check out this thread for more information:
http://forums.civfanatics.com/showthread.php?p=8049627

The offical site can be found here:
http://thirdpilot.net/projects/civsmith


Original post:

Hi!

First, I'd like to introduce myself. I am, as many others, a long-time lurker but have been coming here from time to time to check the status of the different mod (or xml) editors that have been in development. I have tried to get into Civ4 modding several times but found it boring since there are so many files to edit and some of them are really huge.

So I decided to try and make one myself. I'm eager to show you a screenshot of a mod editor that I'm currently working on. It is currently in its early alpha stage but I'm hacking away, focusing on finishing a unit editor. It may not look like much at the moment, but most the time so far has been spent planning and building the "under the hood" functions



As you can see in the screenshot, I have only put a few controls on there to edit some values. This is because most of my time so far has gone into making the application work through plugins, made in Python. Almost all the logic in the editor will be written in Python scripts and will allow users with experience in Python to change, add or remove features. Using Python as a script language has lead to longer development time for me since I could've done this way faster in the programming language/IDE that I'm using, but I wanted to get away from making a fully hard-coded editor, which is harder to maintain and update. This way, users can contribute code and/or make changes to the editor to fit their own needs.

Anyway, I don't want to get too technical here and keep it short. I just wanted to give you guys a heads-up about this project as it may be very useful for newbies and hopefully for advanced users as well

Here's a list of features:

• No installation required. (Yes, I'm counting this as a feature )
• It automatically grabs the paths to the base (game) folders from the registry, but allows users to customize the base path if they need to.
• If the program tries to read from a particular XML file, but the mod doesn't have it, it will read (only read) the base's original XML file.
• When the user changes a value in the editor, the target XML file will be automatically copied to the mod folder (if it doesn't exist already) and save the changes on-the-fly (i.e. no save button)
• It is fully built on Python plugins. For instance, the Units tab in the screenshot, and everything in it, were dynamically created in Python. The preview version will include the Unit Editor plugin and examples on how to write custom plugins in Python.
• The editors will be as dynamic as possible. For instance, default values in boxes and so forth will be read from XML - if possible - instead of providing hard-coded default values that won't suit all mods. Still, since everything is written in Python, possible hard-coded values can be modified by the user.
• A help system is in the works which will allow newbies to learn what all the data means. The balloon hint in the screenshot is a part of this help system. It may even be open for user modification (wiki style).
• Easy access buttons are available. One opens the mod's folder in Explorer and the other launches the mod for testing.

Phew. The current release date is set to April 2nd and it will be a beta version. Most of the unit editor should be done by then. Other editors will be added as time goes on. Thanks for reading this post and I hope you'll provide constructive feedback and/or suggestions

 

[Ajidica]

Looks cool. I'll look forward to seeing it done. Out of curiosity, does it look at the schema to see what new xml tags were added? (EG: In Dales Combat mod, a few new tags were added, if I were to use this editor with that mod, would it allow me to edit those new tags?)

 

[cybrxkhan]

Hmm! Very interesting. It'd make the work of us hard-working modders completely, well, unneeded, at least on the XML level, but that's quite fine. I hope to seeing the final version of this thing being completed.

 

[TheLastOne36]

Welcome Senap to cfc! it seems like you'll soon be a well known name in the modding area with a program like that! Hope things go to plan, it seems really promising.

 

[Senap]

Thank you for your comments

Out of curiosity, does it look at the schema to see what new xml tags were added? (EG: In Dales Combat mod, a few new tags were added, if I were to use this editor with that mod, would it allow me to edit those new tags?)

Warning, there's some technical mumbo jumbo ahead

In early development, I wanted it to read the schema but seeing how limited it is in describing what values each tag needs (for some tags we have the description "textonly" but you need specific values for "textonly" tags like Domain), I still had to do everything manually in Python. Using the schema ended up being extra work that still required me to manually place an edit box through code for each tag, figure out how each tag should be interpreted and which default values I should provide for the tag. So the short answer is no, I won't make use of the schema. Still, I am open for suggestions (as to how to make it work), since the code is pretty flexible and can be changed at any time.

Though, this is exactly the kind of problem I wanted to solve by making it depend on scripts. The user has the ability to modify the script to read the new tags. I am trying to make the Python scripts as documented as possible to allow users to add new controls and features with just a few lines.

I'll go through the schema again though, to see if I can come up with a solution, as it would make the editor even more flexible.

 

[Agent327]

Looks cool. I'll look forward to seeing it done.

I agree.

 

[Ajidica]

Perhaps instead of having to add a value to the tag, just include a function where you can select a 'insert tag' function that looks at the schema to see what new tags there are, and you choose from a list of those. I know little of python though, so this may be a pain to code.

 

[mujambee]

I'll go through the schema again though, to see if I can come up with a solution, as it would make the editor even more flexible.

You can extend the schema to hold metadata about the fields, and your editor could use that info. If no metadata is present just use a default behaviour.

I've done a quick test, adding a field to the tech schema, and the game loaded with no errors.

Standard CIV4TechnologiesSchema.xml

<Schema xmlns="urn:schemas-microsoft-com:xml-data" xmlns:dt="urn:schemas-microsoft-com:datatypes">
	<ElementType name="Type" content="textOnly"/>
...
...

Modified version:

<Schema xmlns="urn:schemas-microsoft-com:xml-data" xmlns:dt="urn:schemas-microsoft-com:datatypes" xmlns:ex="urn:schemas-microsoft-com:office:forms">
	<ElementType name="Type" content="textOnly" ex:contentclass="id"/>
...
...

In this case, the ex:contentclass attribute could tell your editor that the field is an id for the data.


Modders need only add the metadata to their modified schemas and your editor could use them.


Now, for version 2, you can also add a schema editor

 

[Ajidica]

Its easy to edit the schema, but to make the new tag do something, you need to edit the SDK. A bit beyond an XML editor.

 

[mujambee]

Mmmmmm

I propose to add tags to the schema to tell Senap's editor what is in those fields. It has nothing to do with the SDK.

 

[Senap]

Yes, adding custom tags to the schema might work. I'll look into it.

Just so you know, Python isn't the problem here (even though I've only coded small things in it up until now). The problem lies more in the controls (dropdown boxes, edit boxes, text etc) that must be created dynamically, they must have x,y,width and height values for the position and size on the screen. Making that happen automagically is a bit of a pain. But still, using your suggestions, the user could get a prompt and be asked to simply place the new control somewhere on the window. That's one idea.

The other idea is to complicate things a bit and make an XML document for the editor, which will hold the data (as in position, size and where to look for default values) of these pre-defined controls. Python could parse it and create the controls when needed. The user would need to add new tags to the editor's XML to tell the editor how to interpret custom tags. Using XML here would also allow me to make a simple dialog in which the user can just type in this data, to avoid the hacking... maybe even allow her/him to "paint" it on the window.

 

[Ajidica]

@mujambee: Sorry, I misunderstood what you were saying.

Both methods sound good, I dont know which would be better though.

 

[mujambee]

The problem lies more in the controls (dropdown boxes, edit boxes, text etc) that must be created dynamically, they must have x,y,width and height values for the position and size on the screen. Making that happen automagically is a bit of a pain. But still, using your suggestions, the user could get a prompt and be asked to simply place the new control somewhere on the window. That's one idea.

No need to go through all that pain.

Civ XML files can only hold certain data types: booleans, integers, data references, text references and text strings. You can just put all controls in a big column, one at a row, adding dynamically to your dialog box as you read from the schema. Subtags may be a bit more of a pain, but you should be able to handle them.

Also, there are three types of xml file: schema, data and text. Your editor should be able to determine which type is it reading and treat it differently.

Think of it as a database, for a database it is. Just that you don't have an SQL engine hanging around.

This are the metadata fields I would create:

• Description: Name of this field, as should be shown in the interface
• Type: Tells the editor what special field you have there:
  • ID: Used to identify records of a certain entity type (used with link fields, see below)
  • Name: Name of entity, should be displayed in dropdown lists
  • Reference: When a field refers to a different entity (i.e.: techs needed in units)
• Links to: Entity that this field refers to

When parsing a schema file, you look for those fields. If no description is present, use tag as field name. If a text field has no type tag telling you it is a link, assume it is a link into the text XMLs.

Optionally, if you want data to be tabbed, you can add an additional tag to create a new tag.

Sorry

What for?

 

[Senap]

I am currently testing a few scenarios with your solution, mujambee. One thing that bugs me though is that custom tags in the CIV4UnitSchema.xml file would require me to either deploy a custom version of this file with my tags or edit the user's existing file, which may not be a popular choice with modders.

Either way, I think it would be better to deploy an editor-specific XML that would allow the user to add any custom fields to the app. That way, I don't need to increase the XML size of the mod with custom tags.

Sure.. it takes away a bit from the automatic process, but in the long run, I think it's good to keep the editor files separated from the Civ/mod files. For the average user, it will still feel like an automatic process.

I decided to follow your advice about having the controls in a column to allow for easier customization. I tried this method in the first version of the app but failed terribly. After revising the code, I figured out that my first code was doing a poor job, that's all Unfortunately that happens sometimes when you code around 3-4 AM. As soon as I have something visual to showcase, I'll post some screens.

Thanks for your support so far, I really appreciate it

 

[GoodGame]

Sounds like a great idea, a community-customable editor. I take it even the UI can be edited in Python?

 

[zahgrim]

SWEET MOTHER OF ALL THAT IS HOLY!!!
Fantastic idea... as a newbie modder, I have just recently spent hours trying to find a single character typo, or not aware that I need to add/modify an entry is xyzArtDefinesblahblah... very frustrating! I am getting better at it, but this would be a god send!

One suggestion... I have seen other editors for Civ4, and other games for that matter... they are understandably heavy on the editing function... and often woefully underdeveloped on the adding new items/units/buildings side of the equation;

That would be a great help... it's not too hard to go into the XML and change the AirCombat value to make a Jet Fighter more powerful... adding a wholly new Fighter, and not knowing where you need to add entries is MUCH more difficult, and much more frustrating to the uninitiated.

Tabs that would allow you to set the right values, point to your .Nif models, and then go out and automagically write the correct XML for example; Perhaps some options around if the unit or building in question is unique to a particular vanilla civ, or custom civ, or available to all civs?

THAT would be serious good stuff!

I will keep my eye on this thread... can't wait to see the beta! Thanks for the work so far, and keep it up!

Cheers,
Z.

 

[Senap]

Sounds like a great idea, a community-customable editor. I take it even the UI can be edited in Python?

Yes and no.. but mostly yes . I suspect that a plugin (mine or community made) might need to have custom menus so that functionality might be added. As it is right now, a plugin must "register" itself with the program, through Python, and a custom menu item is added under the Plugins menu in which the user can activate/run it.

The space of a plugin is basically confined to the tabbed area in the screenshot. Each plugin can create its own tab here and create any supported control in it, like edit boxes, dropdown boxes, checkboxes, buttons, more tabs... The rest is up to the Python script. The scripts can be used to do simple things or complex tasks.

Plugins don't need to be editors with hundreds of controls though, it can be a script that simply executes and does something "in the background". The only limitations here are really Python's own limitations.

I'll provide some Python examples with the release to show how one can get started on writing a custom plugin.

One suggestion... I have seen other editors for Civ4, and other games for that matter... they are understandably heavy on the editing function... and often woefully underdeveloped on the adding new items/units/buildings side of the equation;

Yes. I plan on making the addition of units possible. But, as with all editors, I must get the editing aspect down first before I can do the addition part.

That would be a great help... it's not too hard to go into the XML and change the AirCombat value to make a Jet Fighter more powerful... adding a wholly new Fighter, and not knowing where you need to add entries is MUCH more difficult, and much more frustrating to the uninitiated.

Yes, I tried to add some ships myself and felt like a child lost in the forest so I know where you're coming from. I found this to be so troublesome that I decided to make this editor. So naturally, I will try to make it happen

 

[Ajidica]

Any progress on it?

 

[Senap]

Yes, I'm actually working on it "as we speak". It's just taking longer than I expected because I was first writing it as a "normal" application without any Python and then I decided to add the Python functionality about a week before posting here. Basically, it turned out to be a mess - codewise.

So I am working on a rewrite that is better integrated with Python and therefore more customizable. The application is turning out to be extremely flexible.

 

[Ajidica]

It can now edit python? Cool! Sounds nice to hear that it is coming along. To many editors stop with one xml file.