XML GUI progress

[Nightinggale]

A bit off topic, but I'm actually about to look into unit art now. Multiple people have asked for a modcomp for BTS where you can set the art for each era, instead of the 3 predefined settings. Looking at the code I think I have nailed how to code that and plan to do it soon (actually today, but something came up). Once I get it working, I plan to move it to M:C. This will provide the bonus that I get some insights on how the DLL handles this.

If it is an area, which needs simplification, then we need to simplify it, either by perl GUI or DLL or both. I think we should look into that problem once we get there.

Another simplification I plan to do is to add a text xml file for each field, which needs one. Something like TXT_KEY_AUTO_*_* where first * is the Type and the second is the tag, like TXT_KEY_AUTO_UNIT_SERF_DESCRIPTION. This will then be placed in the unit description xml file. With the GUI knowing this, The string you enter for Description is written for English in the text file. This provides full translation support while we have direct, quick and easy access to write the text while editing in English only.

 

[Nightinggale]

I ran into issues regarding this editor. It turned out that moving it from one computer to another is really tricky. Since it turned out to be broken by design (library dependencies), I decided to stop working on this.

Thinking hard what to do to be able to get the editor out to all the people who needs it, I realized that the one thing everybody have in common is the mod itself. Everybody can run the mod just fine (or it won't be the editor, which is the main concern). As a result I decided to see how far I can go with the game GUI.

First I investigated the XML interface in the game EXE. It can load the XML files for the mod, which would make it natural to use it to load the same XML files for modding. However it turns out that it can't read attributes and without those it's impossible to figure out what the XML schema say about the different tags.

I found tinyxml2, which is a C++ library for reading and writing XML files. It's designed to be easy to handle and have low impact on memory and CPU as the creator planned to use it on Android. It consists of a cpp and a header file. The last part makes it perfect, because it allows adding the library source at compile time and compile the library itself into the game DLL file and not need extra DLL files at runtime. It had to be modified slightly to be compiled and linked with the DLL though, but only regarding header includes and defines.

I used this to create a few new C++ classes, which are python exposed. Much of the real workload is done in the DLL, such as finding the correct element in the schema to match a certain tag because such work is based on pointers to tinyxml2 objects and as such C++ can do it really fast compared to python. I even added cache to the interface to ensure a virtually instant response time.

Next I built a new python screen from scratch (first time ever I built python from scratch).

Current status is that I can select a file, get the tags from that and clicking either opens a new column with the children of said tag or a string editor. When a new string is inserted using the editor, it's saved into the file right away. No need to remember to do that and no lag because the save code is fast.

I plan on adding types as dropDown menus because typing those correctly is a time consuming task and prone to typos. Also by adding a menu, people no longer enter Unit instead of UnitClass and and so on. The DLL collects all the strings for such menus, but I haven't implemented it in python yet.

Screenshot of progress (the green/red dots are placeholders for bools)

Spoiler :

 

[Nightinggale]

It's working (though not complete yet). Now it's possible to edit strings, add more strings to lists tags missing due to minOccurs=0. Clicking on a bool toggles the setting. It's possible to right click on a tag and it will bring up a setup menu, which currently only contains the ability to assign the tag to point to types in a specific file. Once set to a file, instead of a string editor, it will bring up a list of tags in target file and you can select one (see screenshot). Once set up for a tag, it will work like that for all tags with the name in question and using a schema file from the same location.

Spoiler :

 

[Kailric]

I user tested this and I have to say what you already have is brilliant. When you implement all that you have planned they will have to invent a new word for it This is a modders dream come true.

What about the ability to click a unit and have options available to edit its XML file? And other objects on the map as well. That could be handy too.

 

[Nightinggale]

What about the ability to click a unit and have options available to edit its XML file? And other objects on the map as well. That could be handy too.

Interesting idea. I guess it would be possible to add two strings (or char pointers) arguments to opening the editor. One is a type for the file (like UnitInfo) while the other is getType() of the unit you want. The editor will then open, set file to whatever file is picked and open the element containing the type. This should be generic enough to open everything and it will just be a matter of add the opening code whereever you want it.

I'm not sure how useful it would be, but adding an edit button to the pedia pages would be a nice touch. I have planned an on/off switch for the editor, meaning the edit buttons will vanish in a release build. It would likely be a good idea to wait until I have that working before adding buttons everywhere and then go back and edit all of them later. For now I will focus on perfecting the editor rather than providing more ways to opening it.

 

[Kailric]

Yeah, it just seems like it would be neat for editing units. Say you are play testing unit combat abilities and you think a unity is a bit to powerful, you could click an edit unit button somewhere and change its starting xml stats. When I start play testing I'll keep that in mind.

For buttons in the pedia, yeah, that could be useful too. What about say a shift click on a pedia entry brings open the editor to that entry? That would be very useful. Right now for anyone without the mod if you see you need to make a change you have to go to the exact xml file and search for the entry, so a neat feature like that would save much time.

 

[Nightinggale]

Say you are play testing unit combat abilities and you think a unity is a bit to powerful, you could click an edit unit button somewhere and change its starting xml stats. When I start play testing I'll keep that in mind.

That sounds quite useful. Perhaps it can be an extra command meaning the button row will contain an edit button. Perhaps right clicking on a promotion can open that one. Also it might be useful to have a jump to promotion from a free promotion in the editor. It should be generic to allow jump to UnitClass and so on. Pages are however not stored and it would make it tricky to add a return button. I wonder if I can get clever and add layers of editors

I do see a number of potential issues with editing units/promotions in a running game. The main one being that the old stats are more likely cached, meaning it would have to be combined with a flush cache to be really effective. A player gaining/losing a civeffect, which allows promotions or even grants free promotions will loop all units and force some cache recalculation. It might be as simple as calling this function, but it's something, which needs to be checked.

Adding a new promotion to XML or something like that will break a running game and I can't recommend actually playing a game after editing XML. However I have added code to M:C to "upgrade" savegames on load to match a new xml layout, meaning it should be possible to add a new promotion, restart the game and then load the savegame should work. Having said that, I do agree that it could be beneficial to alter something while running a game to get some trial and error until it looks right. Tech tree placement and stuff like that really benefits from not having to restart the game

 

[Nightinggale]

I added support for help text. Right clicking on a tag brings up a menu where different properties for the tag type is set. One of them is help text. This means it's now possible to tell what each tag does without messing with either code or pref files.

Spoiler :

 

[Nightinggale]

Another update. Now it should be possible to make a mod using the editor only without actually opening the XML files and edit them directly. There are still features to add to make the interface easier/faster to use, but there should no longer be issues with something it can't do at all.
New additions:

1. Writing in a tag not present in XML will create it (like due to MinOccurs=0)
2. Lists always ends with an empty tag, which allows adding more items to the list
3. Added delete button to remove optional tags (like list elements)
4. Added up/down buttons for quickly change int values

Spoiler :

 

[Nightinggale]

I added an icon to list elements (uses Button child tag if possible) and drag-n-drop support to reorder lists. If you for some reason want the elements to appear in a different order, just drag the icon to the line where you want it to be.

It turns out that there is an unexpected issue with this. It uses addDragableButtonAt to create the dragabe buttons. It turns out that this function or anything similar to it only exist in Colonization. This means drag-n-drop can't be moved to BTS.

Spoiler :

 

[JosEPh_II]

This is fascinating and is this available for use? I see it's still a wip, but did not see a D/L link so I ask.

I can see where this would save the Caveman2Cosmos Team sooo much work! We are going thru some extensive Tech tree changes and adding new Eras. And a whole new set of Leader traits as well.

Adding a new promotion to XML or something like that will break a running game and I can't recommend actually playing a game after editing XML. However I have added code to M:C to "upgrade" savegames on load to match a new xml layout, meaning it should be possible to add a new promotion, restart the game and then load the savegame should work. Having said that, I do agree that it could be beneficial to alter something while running a game to get some trial and error until it looks right. Tech tree placement and stuff like that really benefits from not having to restart the game

Agree 100%!

I do hope you will release this.

 

[Nightinggale]

This is fascinating and is this available for use?

M:C is using a public git repository, meaning you can always get the latest version if you can compile yourself. It is being developed in the branch called "develop".

https://sourceforge.net/p/colonizationmodcollection/Medieval_Conquest/ci/master/tree/

Guide on how to use git and compile. Highly recommended reading because there are some unique conditions, like git submodules and the modified makefile assumes perl to be installed.
https://sourceforge.net/p/colonizationmodcollection/wiki/GIT/
(note that there are 4 pages if you follow links)

I can see where this would save the Caveman2Cosmos Team sooo much work! We are going thru some extensive Tech tree changes and adding new Eras. And a whole new set of Leader traits as well.

The same thing goes for Medieval Conquest. In addition to that Colonization 2071 is going to start over using the same DLL file, meaning more or less all XML files will need to be redone. I figured that if I'm ever to make an XML GUI, I better do it now before those huge tasks start.

 

[Nightinggale]

Progress of the last week.

File selector has been split into two menus: dir and file. When changing dir, file changes to contain all the files in the selected dir. This way a menu will only contain max 10-15 lines instead of one containing almost 100.
Files are now sorted alphabetically. This is quite useful because if you press a key, the menu engine will make it jump to a line starting with that character.

Added the ability to assign a GameFont character to an int element. Needless to say this could require some DLL work outside the editor to make it work with anything other than Medieval Conquest. Interestingly enough GameFont is easier to mod in BTS because it has CvGameTextMgr::assignFontIds() (which is inside the exe in colo)

Spoiler :

I have added more code to use button art. Now an element with children can select which one to use for button art (before it was hardcoded to Button). Elements, which are type selectors (contains a string, which is a type in some file) will automatically get the button from the element setting that type. Combine those two and you can effectively make it go through multiple files before it ends up with a Button element, which provides what is displayed.

Why this is really useful is best told with a screenshot. Adding the buttons makes it a lot easier to navigate the menu. If it's a list (and not BTS), the buttons can be dragged-n-dropped to reorder the list.

Spoiler :

 

[Nightinggale]

A new update on progress. I will only mention major changes (not adjustments and fixes)

Type selection is now able to select strings from multiple files. It might only be useful for M:C (CivEffects) and it might be useful for modules.

All art used in the editor is now written in XML instead of hardcoding it in the code. This makes it easier to update as well as getting the (unmodified) code to work in a different mod.

Spoiler :

<Icons>
   <BoolOn>Art/Interface/Buttons/Units/Medallion_CanMove.dds</BoolOn>
   <BoolOff>Art/Interface/Buttons/Units/Medallion_CantMove.dds</BoolOff>
   <Clone>Art/Interface/MainScreen/CityScreen/hurry_commerce.dds</Clone>
   <DefaultButton>Art/Buttons/Tech_Categorys/NativeCat.dds</DefaultButton>
   <DeleteElement>,Art/Interface/Buttons/Actions_Builds_LeaderHeads_Specialists_Atlas.dds,8,4</DeleteElement>
   <DirClosed>Art/Interface/Screens/City_Management/ArrowLEFT.dds</DirClosed>
   <DirOpen>Art/Interface/Screens/City_Management/ArrowRIGHT.dds</DirOpen>
   <IntUpArrow>Art/Interface/Buttons/up_arrow.dds</IntUpArrow>
   <IntDownArrow>Art/Interface/Buttons/down_arrow.dds</IntDownArrow>
</Icons>

Added a button to write all XML files. This converts all files into the layout written by the editor meaning svn/git will provide more readable diff files and changelogs.

Added a clone button to drag elements to. This makes it fast to make a copy and then change the 2-3 tags you want to keep different. Interface use Colonization only code, but the actual cloning doesn't.

Added a table to provide an overview of the contents of GameFont, complete with big and small version of each as well as the int value of the IDs. It doesn't do anything, but in some cases it's really useful to know the IDs.

Spoiler :

 

[Nightinggale]

Imagine this scenario: You notice the following line in your code:

<Type>TECH_GLASSS</Type>

You want to fix the typo, but don't because you know that it will break loading all the files, which copy pasted the typo and you don't know if it's 0 or 30 places or in which files. Just confess that you have been in that situation because we all have.

Now I updated the editor to help in precisely that case. Whenever a tag of type Type gets a new string, the editor will loop all files, compare the string of each element to the old Type value and if they match, replace it with the new Type string. Nothing is changed in the user interface. It's all behind the scene, but it really reduces the workload in some cases. It also helps keeping your mind at ease because you know there is one less way to screw up your files.

 

[Nightinggale]

I added the ability to create new entries in other files than the current one. For instance if you make UNIT_SCOUT, set the class tag to be of UnitClass type and click the new button, it will clone the first entry in unit classes, add it to the end and set the type to UNITCLASS_SCOUT and then it will set the class tag to UNITCLASS_SCOUT. It can be used for all tags pointing to other files, but it makes more sense to do it with some files than with others.

I improved support for international keyboards. It seems that the game assumes everybody to use a US English keyboard, which means comma, period and similar doesn't match the locations if you have a non-US keyboard. I added a "calibration" where you press all the keys in turn and the editor saves the result. Those saved calibrations are stored under the name windows use for the keyboard, meaning it will only have to be used once for each layout and once a number of people have calibrated, chances are good that people will no longer have to do this themselves. Still I don't like the fact that KB_ keys aren't all placed on the correct keys on all keyboards. Maybe this should be explored more as this is likely also an issue for ingame hotkeys.

Due to a limitation in the code (I didn't feel like messing with conversions), only ASCII characters can be used, as in 0-0x7F. This shouldn't really be a problem because using non-ascii in XML could lead to issues. It's an editor for xml data, not TEXT XML. For the most parts, text and keyboard input would be limited to typing Type strings and possibly art paths.

Other than that, the editor is near completion. It's currently in the beta stage, meaning it's usable, but as it is being used/tested, bugs are encountered and fixed.

ETA for completion is still unknown. It's when all bugs have been fixed, all issues with "this is too tedious to use for each entry" issues have been solved and the editor documentation/help text has become good enough for people to use all features without asking me how they work.

 

[Nightinggale]

Bug hunting is going well. The bugs have all been minor and easily fixable. There are also some improvements to make it easier/faster to do common tasks, like when writing a new Type string, it detects what the other types in the file starts with (like UNIT_) and sets the string to that before accepting input, meaning this will not have to be written each time. Also Types are now uppercase even if shift isn't used.

GlobalTypes.xml is now used as types strings can point to, just like Type in other files. This file can't be edited in the editor, but it's rarely edited anyway.

Setup for a tag type can now be file specific. This deals with nameclashes, like say two files contains a tag called Class, which points to two different files.

Each cloneable entry now has a clone button. It's easier to use than drag-n-drop and it works in BTS. Reorder entries in a list is now the only feature not available in BTS.

Added a lot of help text. Now all checkboxes should have a mouseover text explaining what they do.

Currently I'm working on gaining awareness of text xml files. The idea is to make one file for each xml file and then automatically add the entries. The editor can't be made into a replacement for editing the text file, but just filling out the entries and deal with the TXT_KEY setup should speed up adding more text strings. This is the last planned feature for the editor.

 

[Nightinggale]

I managed to get the editor to autogenerate TXT_KEYs. I added a button, which generates a text xml file for each xml file with TXT_KEYs and then it copies the text from other files, or generates new entries if none exist already. It deletes the moved strings from the original files to prevent duplicates. If somebody just wrote say <Description>Warrior</Description>, then the editor will move that string to a text entry, which in turn allows translators to translate the name. Works for all TXT_KEY entries.

The plan is to add the ability to click on a TXT_KEY block in the editor and then write what you want it to say and it will be saved in the txt xml file and the TXT_KEY will be autogenerated and set correctly both for the current file and the text file. I still recommend using a proper text editor for writing long strings. This is just to generate entries and add names etc as programs like Notepad++ will always do a better job for long text.

I managed to unlock the full characterset, not just ASCII. Sadly it's a tradeoff since it turns off keywords like [NEWLINE]. This means extended characters will only work in some strings. If disabled, the key input is just ignored.

This is the last major planned feature for the editor, meaning it's hopefully done really soon. Still no support for modules through. Still not sure what to do about those.

 

[Nightinggale]

Does that xml editor work on Regular Edition?

It's a good question and it got me thinking. My plan is to finish for M:C and then make a modcomp for BTS, but if more versions are still in use, then what?

Thinking about a proper answer to this question made me realize something really important. Right now the editor is hardcoded to read the path of the DLL and then use that one for reading and writing files. What if it is possible to add an XML file, which tells the path of the mod? This would allow making an editor mod, which can be used on all mods even without modifying the code in those mods. If exe paths are set as well in that new XML file, then editor and mod will not even have to use the same exe file.

The result from this approach is that I can make a standalone editor as a colonization mod and it can be used by everybody regardless of which version of civ4 they mod. It shouldn't be hard to make an editor mod for BTS as well, but using the Colonization version is highly recommended because only the Colonization exe supports drag-n-drop and the editor use that. This means the BTS version will be a worse editor and there is nothing I can do about that.

The only issue I can see with editing a "remote" mod is the button preview. However this can be solved by just copying the Art folder. Most likely a symbolic link will be a better solution, though it takes more skill to set up. Either approach should work.

This requires some thinking and some testing, but I think I'm on to something really interesting here. Making this standalone setup will not prevent mods from including the editor in their own source code. This is just added flexibility.

 

[AbsintheRed]

Just read through the thread. Simply wonderful!