XML GUI progress

[Nightinggale]

I will use this thread to keep track of my perl GUI work. As mentioned elsewhere I managed to find wxPerl, which allows usage of wxWidgets in perl scripts. In other words it is a library, which allows easily created GUIs, which works cross platform (windows, mac, linux and some other platforms). However it turns out that easy only applies once you truly figured it out. wxPerl is mostly undocumented and wxWidgets is ok documented for reference, but I haven't found anything useful for "get started" except for how to open a window.

I made this window (as already mentioned) and committed it as TypeRename.pm inside GUI_scripts in the source module. It's a fairly simple interface to rename types in xml and it renames both the type itself and all references to that type.

My new addition called Editor.pm. It reads the xml files, makes the list to the left and when one is clicked, it makes a list of all types and writes those in the right list. Currently it does nothing else and it lacks a concept of types as it forgets about all of them as soon as it has added them to the list.

The goal is to be able to edit xml files through this interface, hopefully by using the schema files as templates to ensure everything is written in the right order, even when MinOccur=0. I have no idea if that will ever be possible because there is currently so many unknowns, but I do know the only way to find out is to keep trying. Right now it seems like the obvious goal is to make something, which opens a schema file and then writes a bunch of text fields based on the xml layout. Once that is working, fancy stuff like checkboxes for bools can be implemented. My biggest concern is lists as the one tag, one text field seems fairly simple.

Actually the next step is to make a hardcoded "schema file" with some tags to write text for. Nothing about file I/O, just draw the stuff on the screen.

 

[orlanth]

Woww you are making an XML GUI editor from scratch? That sounds like an epic undertaking worth over 500 lightbulbs of research and 1000 Science FF points.

As I bet you know there are a variety of XML parser modules including XML:arser and XML::Simple which could hopefully help.
http://perl-xml.sourceforge.net/faq/
http://www.techrepublic.com/article...l-increase-your-xml-perl-development/5987871/

I hadn't been able to figure them out well unfortunately; in retrospect my approach to making the Perl XML generator by hardcoding the printing of XML files was stupid, because it will break after any changes to the XML schemas of which there have now been many. Hopefully a more general solution that could read schemas and generate appropriate tags could work better.

 

[Nightinggale]

XML reading libraries.... somehow it sounds like a good idea, but it isn't part of my plan.

My concept is fairly simple. I select a file, then a type and it will open the xml file, save it in memory as start, "interesting" and end. "interesting" is then placed in the gui and when the user clicks save, it will write the file start + "interesting" + end. This has reduced the writing problem to a single class instance. However now that you mention it, maybe it would be better to use a proper xml library.

Building the GUI for the specific file is what I'm working on right now. I have figured out the code to handle strings, enum values and bools. Presumably ints are equally easy, but I just realized I forgot about those.

Currently I only have one big problem: lists. I can easily handle anything with one tag->one value. MinOccur=0 should be possible to handle too. I just need to figure out something about default values. The problem is when there is more than one of the same type, like a profession using two different kinds of yields. I know how to handle that in perl code, but how to draw that in the GUI and make it editable?

Right now I'm thinking something like a list will only be a button and clicking that one will bring up some list editing window. That will solve the problem of the propertyGrid being one tag, one value, but it leaves the other part of the problem unsolved: how should the window be designed and how should it work?

 

[Nightinggale]

If your application really is sensitive to element order, don't use XML::Simple (and don't put order-sensitive values in attributes).

There goes that idea. XML::Simple is unable to save xml files suitable to the civ4 engine. XML:arser isn't recommended for new projects anymore because it is based on an old interface. XML::LibXML seems to be the recommended one, but I don't like the look of that learning curve.

I think I will stay with my original plan of generating a GUI with all the default settings for each setting (schema based), then go through the file and overwrite the GUI value with whatever the tag contains. On write, write order based on the schema and then for each tag, fetch the value from the GUI. Our requirements is fairly simple, which mean simple to code, yet it is too complex for XML::Simple.

Maybe it would make sense to use XML::Simple to read the xml file, but not to write it

Brainstorming on the list problem might have solved the issue of having multiple items using the same tag. I will aim for adding a button to each, which will add a new default element to the list if clicked. Each element then contains a delete button, allowing the modder to remove the elements again. The internal unique name is then autogenerated to ensure it's unique. It likely need some sort of sort to ensure it isn't saved in a random order, hence making changes to that part of the xml without having actual xml changes.

EDIT: seems like adding a button to the list requires adding a custom Wx:GEditor. This can be done in C++ (particularly poorly documented, but possible), but it seems that this particular feature isn't supported in the perl frontend
This mean either I restart in C++ and make all the xml read/write code much more complex or I will drop adding custom features to the list.

One alternative idea is to make a two part window (like the files and types screenshot) where the property list is in the left. Lists (like required yields) are then added as a non-changable string. If such a line is highlighted, the right part turns into an editor for the list. This should be doable using only standard features like button placement and such and should be fully supported. It would require triggering on the event when a new tag (line) is selected, but it should do that anyway if we are to have/support a help text for each tag. Also that event is used in the wxPerl demo, meaning there is working code to copy paste

 

[orlanth]

Hmm ok maybe using XML::Simple is not so simple

One cool feature of some XML editors is to enable an XML Grid View; in this case top-level tags representing the "object" is one row (e.g. each <UnitInfo> unit entry in Civ4UnitInfos.xml); while each tag within that "object" (e.g. <iStrength>, etc) is one column.

http://stackoverflow.com/questions/...itor-with-grid-view-similar-to-that-of-xmlspy
http://www.stylusstudio.com/xml-grid-view.html

For dealing with lists, the list could become a table within a cell of the grid as in some editors, but it's probably much more straightforward to highlight or click the list entry to edit it in whatever way is most feasible.

 

[Nightinggale]

I can make a gridview just fine and that's the plan. The problem is adding/removing rows on the fly. In some cases it could be done by adding all possible. For instance adding a yield modifier could be done by adding one for each yield, each with an int field default to 0. Not ideal, but it would work. In other cases that approach would require something like numUnitClasses * numPromotions. That is quite a lot of lines. There are also cases where the max number of lines can't be predicted by perl. The problem isn't to make the gridview, but rather changing the number of children where the schema says MaxOccur=*.

I think I have the GUI worked out now. However I noticed one feature in the wxPerl demo that I would like to copy. Say we use the editor screenshot in the first post. It is possible to add a line between the two posts and then let the user click and drag the line between the lists to resize the lists relative to each other. If I figure out how that is done, then I can make all "blocks" resizable, which mean I can just rely on the automation to set the size of each. After all if the automation gets it wrong, the user can just resize as needed. That makes the design phase easier and the window should be able to fit on any resolution. If it turns out that I need to do something fundamentally different, then I better figure it out before making the entire GUI.

I figured out how to make the help text for the tags. For each schema file (that is, for each directory in xml) there should be one tag->one text. When you click on a tag, the help text block (not yet drawn) should switch to the tag in question, telling what the tag does. Filling out all the text seemed problematic until it occurred to me that the easiest solution to both code and use would be to just make the help text editable and add a save button to it. No new GUI to do that or weird file format. Internally it will be stored as a hash. Since order doesn't matter, the easiest way to save it on the disk is to use XML::Simple and make a help text xml file in each directory. The same file can be used to store features for each tag, such as type of input. Type of input can then be used for say assigning that it is of type yield, in which case the GUI will only accept strings used as types in yield.xml and it will recommend based on that list.

There is a bit hardcoded for M:C in the code right now, like the path to the xml files. I have a plan for a setup menu, which can remove the need for hardcoding and if it works as intended, it mean it should be able to work on any mod without code modifications, including FTTW.

I'm still not 100% sure this will work though. There are still some blanks and untested parts of my concept, but the blanks get fewer and fewer and the task no longer seems epic and/or impossible. It might actually be doable with much less code than CivEffects.

 

[Lib.Spi't]

Intriguing!

 

[Nightinggale]

I figured out how to make resizable "blocks" in a window. Mysteriously it works in the example and in my proof of concept. However when I try to add it to the existing editor code (first post screenshot), it messes up the window completely We can do without it and if somebody figures out how to use it eventually, then it would be great. If not, then it will still get the job done.

I added a Setup button to the editor and if it is clicked, a new window opens (yeah, multi window application ) and that window currently has two options: path setup for XML for mod and vanilla. More can easily be added later.

Next step is to use XML:Simple to save Setup settings and automatically open the window if the setup file is missing. Once the application has both paths during setup, it will be able to generate a list of files and types as well as gaining access to all schema files. With this foundation in place, the path is clear for making code, which actually reads and writes the xml files.

 

[Nightinggale]

-The save button in settings does what you would expect the button to do
-Settings are loaded at startup
-If settings aren't valid after load, the settings window will open and the load will repeat when the window closes
-The path from settings is now used rather than a hardcoded path
-settings.xml is ignored by git to prevent user defined paths to vanilla from being committed

I got the xml filehandling to work. It turns out that XML::Simple is really simple and quite useful for reading xml files. It's absolutely useless for writing them, which mean I went searching for an alternative and found XML::Writer, which writes easily, but can't read. This mean the code now relies on both. However being extremely simple, I might remove XML::Writer and just write my own code to handle this. That is, my own code would be able to work for writing game xml files as well, which I'm not 100% sure the library can handle. It indents incorrectly, but I have a script to fix xml indents because humans tend to indent incorrectly as well

Next step shouldn't be that hard. Now that I have the paths available, I need to read the files to build a database of them in memory. This doesn't require any GUI work and no xml actions I haven't already done. This mean no research, just code writing. That will be nice for a change

Once the "memory database" is complete, I can start to write code to display files and once that work, I should write the actions of the save button and it should be good to go. It suddenly doesn't seem to far into the future anymore. Sure I can add all sorts of useful features, like switching order of elements in a file and add the type rename code, but reading and writing basic info seems like a good place to start to do something useful.

 

[orlanth]

Wow I can't believe you can code an xml editor that fast

For the grid view, how can you make it know which are the top level tags representing an "object" to be portrayed in rows? I.e. Understand that each <unitinfo> entry is an object, ignoring the irrelevant tag levels like <civ4unitinfos> present in some files?

For showing grid view where some tags have a variable number of children, I suppose some potential approaches could be
1) make the column header for that tag a button that can show/hide all children of that tag
2) make the contents of that column each contain a button, which when clicked opens a table (or plain xml text editor) to edit the list contents

 

[Nightinggale]

Wow I can't believe you can code an xml editor that fast

I haven't actually made an editor yet. I have made a propertygrid displaying two paths, one for vanilla and one for the mod. Whatever the user writes there is stored using xml and I can read the xml and use that as default values at startup.

I can generate a list of xml files based on the paths and for each I can make a list of types, but that's it. There is no real editor yet. Right now I figure I better slow down and do the foundation correctly or it might hurt later.

For the grid view, how can you make it know which are the top level tags representing an "object" to be portrayed in rows? I.e. Understand that each <unitinfo> entry is an object, ignoring the irrelevant tag levels like <civ4unitinfos> present in some files?

XML::Simple can handle this. If there is one unitinfo, it will add unitinfo to the hash as whatever that tag contains. If there are 3 unitinfos, it will add unitinfo to the hash as an array of length 3, each element in that array containing one unitinfo.

I plan on making some sort of file editor. Left is a list of children, displayed as their type names or index for type less files.
In the center block is the propertygrid, as in setup for each tag for whatever is selected to the left.
To the right is a list editor, which is used when the center selects a tag containing a list.

PropertyGrid has support for a "folder structure" like the files in the screenshot in post #1. It looks different, but the principle is the same with a line with an open/close children approach. I intend to use that to display children just like they are displayed in the xml schema.

Renaming types and resorting children in a file is problematic due to technical issues and we might end up with a dedicated window for each to allow those actions without corrupting cache. For now I will ignore that issue and just read the existing xml files. I will not try to implement fancy features like "new unit" buttons until I can edit existing ones and save them.

 

[Nightinggale]

I made some progress again. On startup, the script now loads all vanilla xml files, then all mod xml files and overwrite in memory if the same name exist. It doesn't actually store the file content itself as that would be a waste of resources. Instead it stores metadata like name, location, types and so on.

The window now relies on the cache rather than disk access. The result is that the lag from constant disk access is removed and the GUI feels quite fast. I added a show/hide button for vanilla files, meaning you can display xml files not already present in the mod. The plan is if you do that, edit and save such a file, it will save to a new file in the mod. The only non-automated task for doing this would be to add the new file to git.

There is also some behind the scenes improvements regarding locating schema files, which was my main goal. Now all it takes to load a schema file (or any of the xml files) is to know the name of the file, not the location. In other words I can now write code to load schema files without caring if it should read the one from the mod or vanilla.

Next step now is to actually load a file into the GUI and I will start simple with a basic file containing only strings. Using my "think ahead" approach, I think it would be best to add all the needed tools to those simple files, as in read/write, reorder, rename and all that while the files are still simple to handle for humans. After that bools, ints and floats should be added and the editor can be used for even fairly big files

Last will be the dreaded lists. However now I think I have an idea on how to handle those.

 

[Nightinggale]

Finally I finished the savegame code (sort of, it's stable now) and did what I promissed other mods I would assist with and it's back to the xml editing

I made a perl gui branch to keep the git log somewhat orderly. I also added a new window, which is supposed to be the one you use when you edit a single file. However right now it is pretty empty, but it does display the file you selected as window title. In other words the new file selection code and button works as intended.

I decided to keep the files in memory anyway. They are stored as arrays of strings(one string -> one line). Turns out storing them is instant and it only use 5 MB of memory, which makes the combined usage 61 MB. The less filehandlers needed, the faster the program

I tried using XML::Simple to store the xml data in a data structure. It added 60 MB of memory and startup time went from 0.9 seconds to 21.6 seconds. I want the opening time to feel instant and dropped the all files processing approach. One interesting thing is that included in the 0.9 seconds is reading all the files, searching them for lines containing <Type> and making a database of types for each file.

 

[Nightinggale]

Progress again. Now all schema files are read into a data structure, which allows quick access, meaning perl is aware of which tags to place where and which order. It is also aware of min/max occur.

Currently it's not really used though, but it's getting close to actually do something now. I just need to add support for... well the actual xml files
Still it looks good that it tells me that if I ask it to edit CIV4CivicInfos.xml, it tells me that the tag it will use to create the list of elements is CivicInfo. I didn't tell it about this file at all. Instead it can figure it out based purely on the schema reading code. In fact it doesn't know about any files at all. It starts by scanning for xml files, making it really adaptive to whatever we can throw at it.

It doesn't detect this correctly for all files. For instance it picks CivilizationInfos. My code works, but the schema is wrong and is using MacOccur="*" when max should be 1. Even worse: vanilla has issues with correct MaxOccur and works just because xml modders only adds one even though the schemas allow multiple. I'm not sure if it would be wise to force all schema files to be correct or code to allow "buggy" schema files.

EDIT: I decided to find the branch by declaring it to be the first tag not ending with s. It's a dirty hack, but it will get the job done right now and I can carry on working on the script. At some point it should likely be very strict on xml schema errors as a lax policy will likely cause problems at some point.

 

[Nightinggale]

I finally managed to make a file editor window (see attachment). It assumes everything to be strings and there are some glitches. Also it can't handle "directories", which is why Yields has a weird value. Some files crash when this window is opened, but it looks pretty good considering I just added support for strings and it tries to see how well it can go by assuming everything to be a string

Also at some point it would be nice to make the window resizable. While I can click on each tag setting to write something else, the changes are not saved, which would be another useful feature to add. In fact the change isn't even written back into the database, just in the window and is lost the instant the window is closed or even a new item is selected in the left panel.

It should be noted that I told the perl code absolutely nothing about routes. It autogenerated everything based on reading the xml file and the schema file it points to. The only file it has special awareness about is globalDefineALT where it will use DefineName instead of Type when naming the items in the left panel. If not DefineName or Type is found, then it will use a fallback, which is to give each one a number. It doesn't really care what it writes as it works purely on the index anyway. The names are just for human readability and we could call all of them the same for all the code cares.

 

[Lib.Spi't]

Skynet is becoming self aware you will kill us all you fool!!!!

 

[Nightinggale]

Skynet is becoming self aware you will kill us all you fool!!!!

He doesn't look that harmful

I keep wondering if I have done something stupid. I keep running into "do this" for wxWidgets in C++ and it's not implemented in the wxPerl. Also if I start over in C++, I can compile an exe and distribute that one. The install instructions for perl becomes more and more complex. On top of that I suspect i have encountered perl only bugs regarding resizing, which isn't good either.

Last but not least I do not feel in control of the code like I usually do in C++. Perl is very helpful in producing working code where you don't have to consider lots of stuff, but that's not good right now. In C++ I use a pointer to an object if I want to change the original. In perl that is done with a reference, but I don't feel in control of the reference system and isn't completely sure if I edit the original or a copy. Naturally the changes in a copy will not go back into the original and hence the change is lost when the window close.

I will try to see if I can convert the code into C++. The window drawing code is more or less copy paste, which leaves the XML code. It shouldn't be hard to make better code than what I came up with as for once I just went with a trial and error approach.

 

[Nightinggale]

I'm giving up on a C++ solution, at least for now. There is something very odd going on regarding wxWidgets. It seems that even though I link to the library files, it can't see the functions in the libraries and link to those. It includes the headers just fine, but not the library files themselves. Quite odd. I have never seen anything like that before

Looking at wxWidgets vs wxPerl, I do have to say that wxWidgets is even more powerful in comparison than I thought. You can make custom classes for everything, including brand new widgets. It's like modding the civ4 engine. wxPerl is as powerful as xml modding while wxWidgets (C++) is as powerful as modding the dll. I would love using the C++ code, but if I can't even get it to compile the included examples, then it really isn't of any use to us.

 

[Nightinggale]

Added support for tree like XML structure. Right now it assumes MaxOccur=1, which is the value used if it isn't mentioned. I know what the calls to wxPerl should be to handle a max=* and the data I need is available in memory. I just need to figure out how to write a recursive/loop structure to generate the wxPerl calls. Mixing MaxOccur and the nested tag structure turns out to be quite complex and great care should be taken to avoid bugs and unreadable code

I will explain the plan from a XML modders perspective using AllowsBuildingTypes as an example.

All AllowsBuildingType will be added. One blank will be added as well. When one entry is edited, it will check for blanks and if none are present, it will create one. Removing is impossible, but blanks will not be saved. This mean removing existing ones is done by just deleting BuildingType or iChange (or both).

When something else is selected in the left column, all lines in the right side will be regenerated and empty ones will be thrown out. You will not get stuck with 20 empty.

Once the multiple instances of the same name is fixed, the next step is to save changes to memory. It's already working at root level, but "subtags" needs to be saved as well, meaning the save back to memory feature needs awareness of the nested layout.

After that a save button is required to dump the memory content of the xml file onto the disk in a game readable format.

Once save to disk works, the editor should be usable. I plan on further improvements, like checkboxes for bools.

I also have a plan to make some user defined database, which is saved in the mod. The idea is that I can select the line with BuildingType and then click on some button. This will open a window where properties can be entered. It would be an ideal place to enter help text, but it could also be used to tell that it is a variable of type BuildingClass. This should then force it to be a select menu displaying all BuildingClasses instead of just a text field. By placing it in an xml file, it can gradually be filled out by xml modders and it will not require me to do all the work. Other interesting stuff like default value would also make sense in such a setup window.

On top of that, there are glitches I would like to fix, such as making the window resizable. It can be resized right now, but the two lists will not resize and it will just add a bunch of gray background. Being able to use most of the screen would be good. If I can't get it to work, the alternative could be to add size to user settings. Right now I can set it to whatever size I want when the window opens, but I can't change it after it has been opened.

EDIT: oh and for the record, I still haven't told the code anything about the xml files. The layout is still 100% generated based on xml file+schema.

 

[Kailric]

It has been a while since I done any XML work but the one thing that annoyed me the most was having to add all the UnitMeshGroups in UnitInfos. I think with the CIV4UnitArtStyleTypeInfos mod this was simplified a bit, but still it is a pain to deal with. Just think how a mod like World History would be like. Anyway, perhaps this xml editor will help out there.