Civ4 Moders we need you!

[Hydromancerx]

Hello I am from the Civ4 mod called "Caveman 2 Cosmos" our mod has been around for a long time now and we have one feature that has been hanging in limbo, "multimaps". Basically the ability to have more than one map in our game. One of our previous modders named Koshling started making it but failed to finish it. Our lead modder Thunderbrd has tried to take it on but it is beyond his skills. So we are looking to other modders from beyond C2C. Perhaps your new eyes can help us achieve this long sought after feature in our mod.

Thank in Advance!

Multi-Map Overview
https://forums.civfanatics.com/threads/modders-documentation.441325/page-3#post-12199585

Trouble We Are Having
https://forums.civfanatics.com/threads/multiple-maps-and-mapscripts.460023/page-9#post-14685203

 

[Nightinggale]

You can just post like this and get better programmers

It sounds really interesting, but for the time being I'm way too busy with Medieval Conquest to look into new problems. Maybe later (and later is not next week). You could link to something, which gives even better insights into what has been done so far, like the source code itself.

From what I get, plots are now x,y,z and looking at a different z means marking everything dirty in the drawing engine, which forces it to reload all data from the DLL and redraw everything. Then comes all the other problems with AI, pathfinder and stuff. It sounds complex, but doable.

I'm not sure if it should be like Colonization where you go to the edge of the map and then leave the map to show up in another part of the world (which would be another map) or if it would be like Master of Magic where it has gateways (towers) to link to a plot on another map. I suspect the latter is the case and the first can then be added by adding the jump plots on the edge of the map.

 

[Thunderbrd]

You can just post like this and get better programmers

lol... you can't blame a fella for tryin' right? I've worked with you before and if this interests you, I'm very confident you'd be the guy to pull it off. Truth be told, I'm surprised you haven't gravitated towards C2C already.

It sounds really interesting, but for the time being I'm way too busy with Medieval Conquest to look into new problems. Maybe later (and later is not next week). You could link to something, which gives even better insights into what has been done so far, like the source code itself.

I would be more than happy to give you full SVN access and thus access to look through the code. C2C has had a bewildering amount of modding done, some incredibly genius, other's a little mad scientist, a lot of it somewhere in between. That said, it's not got a whole lot of the original coding left. You might really love some of the wide varieties of examples of cleverness shown here.

From what I get, plots are now x,y,z and looking at a different z means marking everything dirty in the drawing engine, which forces it to reload all data from the DLL and redraw everything. Then comes all the other problems with AI, pathfinder and stuff. It sounds complex, but doable.

Sorta. It's XY and Map but Z will also be added soon which is a height/depth measure on the given map - for submerging and hovering units.

As for how Koshling set things up to be able to switch between maps and so on and define them... understanding and using some of this will be why we need your help.

I'm not sure if it should be like Colonization where you go to the edge of the map and then leave the map to show up in another part of the world (which would be another map) or if it would be like Master of Magic where it has gateways (towers) to link to a plot on another map. I suspect the latter is the case and the first can then be added by adding the jump plots on the edge of the map.

The moving from one map to another should be through missions so should be a variety of ways but nothing so unrealistic as to go to the edge of the map. Predefined 'portals' may exist from city exit points but those would just be qualifiers to perform the mission that takes the unit from spot A on X Map to spot B on Y Map. There's a lot of ways units may end up moving from one map to the other.

Our challenges are in figuring out how to define and run new mapscripts and to switch displays from one map to the next and how to manage all this data. Also how to get the units on one map to pretty much all resolve themselves before we go to loading the next map to resolve what happens there (because flipping between maps willy nilly will likely have to cost a slightly annoying amount of processing time...) things like that. When to trigger the generation of new maps, what happens to generate them. And worst of all is resolving numerous potential graphic issues that naturally come from this. The EXE is, as we know, a big problem here. But some proofs of concept have taken place with the work Koshling did on viewports soooo... this should be something that is really very close to being a complete system already.

We'd love to have your help Nightinggale... As I said... you've already earned my deep respect for your knowledge in previous discussions.


Also... keep in mind that although Koshling may not be willing to work directly on C2C anymore, he's always been very good at being willing to get into depth with answering questions. I have not wanted to hound him too much on this matter because I know my limited realm of understanding is probably going to frustrate. But he's very good at answering and someone who knows their stuff can probably get a lot of information very quickly from him.

And if you are the one that pulls this off for us, you'll be a godlike figure on this forum... we've been wanting to achieve this for something like 7 years and there are hundreds, if not thousands of fans that are praying for this solution to happen sometime soon.

We ask at this juncture because we've had some very interesting breakthroughs in XML and art preparedness for this step... extremely prolific modders have been working dilligently on buildings and terrains and features and improvements and have managed to make this sorta work with special hand designed maps that have regions for all the different map types... but it's a really pathetic proxy for what we know we should eventually be able to do. In short... we're ready to make this happen if we can get the skill onboard to bring it to fruition.

 

[Nightinggale]

Truth be told, I'm surprised you haven't gravitated towards C2C already.

I started out modding Religion and Revolution, then Medieval Conquest, then Fallout: Tame The Waste. After that Colonization 2071 was in dire need of my modding skills and I was like I can't actively work on 4 mods at once. I then made the decision to mod M:C into a "universal DLL", which means it should support both Medieval Conquest and Colonization 2071 as well as whatever other mod would like to use it. I can't abandon this undertaking, particularly not now that it's near completion. Not even for C2C.

And if you are the one that pulls this off for us, you'll be a godlike figure on this forum

Before the forum upgrade, the forum acknowledged me as a deity. Seems like it has really gone downhill lately Luckily I have a plan to fix all that. It involves 3 units of veteran missionaries... oh wait, wrong plan. My plan is the attached screenshot (of work in progress)

Spoiler :

 

[Thunderbrd]

I then made the decision to mod M:C into a "universal DLL", which means it should support both Medieval Conquest and Colonization 2071 as well as whatever other mod would like to use it. I can't abandon this undertaking, particularly not now that it's near completion. Not even for C2C.

Completely understood... very similar for me the whole reason I've continued working on C2C for so long. Too much invested to stop now before completion.

Before the forum upgrade, the forum acknowledged me as a deity. Seems like it has really gone downhill lately Luckily I have a plan to fix all that. It involves 3 units of veteran missionaries... oh wait, wrong plan. My plan is the attached screenshot (of work in progress)

That's right... I needed to come up with a whole new title myself. Conveniently, this one was dubbed me by a rival during one of our long going MP games (Play By Email adapted to Play by SVN).
Wow... is that just for Colonization or would the xml editor apply for Civ IV in general as well?

 

[Nightinggale]

Wow... is that just for Colonization or would the xml editor apply for Civ IV in general as well?

You partly said it yourself.

The EXE is, as we know, a big problem here.

The XML interface through the EXE is useful when you know the file layout. You can't ask the EXE to provide the schema or layout, nor can you ask for attributes. It seems the EXE was once used to write the files in the first place though as it can write files and write data to XML. Also if you snoop inside the EXE with a hex editor and read the plain text, it says tinyxml is compiled into the EXE. The vanilla xml files states they have been created with tinyxml.

Since I need the schema attributes I gave up on the EXE before figuring out if it had any other limitations. What I have done is added tinyxml2 (a complete rewrite for optimization), modified it to link with the DLL and compiled it into it (no need for more DLL files). I then was a bit creative and wrote code to look in both the mod and vanilla (windows has some functions, which can be used in a creative way to find both locations of EXE and DLL). When opening a file, it will try MOD, then vanilla, just like the EXE. I added 3 C++ classes to keep track of all the XML data (file locations, read file contents etc). Two of them are exposed to python. Each tag object given to python contains two schema pointers and two pointers to the xml file itself (the tag and the parent). This is what is required to access all needed info from both the file and the schema as well as the ability to write data back to the file. The XML file is automatically saved each time a string is updated and is so fast that it feels instant even with a debug DLL. In fact I put as much as the work in the DLL as possible because it's extremely optimized and well integrated with the highly optimized tinyxml2.

The result is that I ended up with 5 C++ files, which I put in a directory of it's own (Makefile 2.x supports source subdirectories) and none of it uses code from outside (except code present in all civ4 games, like boost python). There is one python file for the actual GUI, which is also self contained and should work on all civ4 games. It needs a few modifications to vanilla files, such as the python screen getting the DLL interface pointer from gc. No files have been hardcoded either. The easiest way to tell an XML interface where the xml files are is by adding a new xml file to list them. My way to open the interface is to enter the main screen in pedia where I added a button. This way it should be possible to open the editor even if xml setup is borked and you are unable to start a new game (though I haven't tested that yet). The pedia code should also be the same for all versions of civ4.

The only issue regarding moving to BTS is the location of vanilla. For vanilla it would check BTS, but the file might be in warlords or original. Not sure what to do about that, but it's most likely fixable. Having xml files in different locations/names is just a matter of editing the xml file and might be needed even when moving between two mods using the same exe.


My planned goal for this feature is more advanced than can be seen in this screenshots. I plan to add checkboxes for bools, dropDown menus when selecting a type (like UnitClass in a unit) and so on. However it turns out that the python GUI system is not just unfriendly, but downright hostile to make something this advanced. Time will tell how much I can manage to add. I hate manual xml editing enough to write this and I hate it enough to not stop until I'm done or learned that something is simply not possible.

 

[Thunderbrd]

However it turns out that the python GUI system is not just unfriendly, but downright hostile to make something this advanced.

Completely agree. Py is not my favorite for that reason.

You're definitely doing something seriously impressive there! It sounds like the limitations of the DLL to EXE in Colonization is a little different than it is in BtS.

Good luck with the project but I have a feeling it won't take you as long as you think if you dedicate deep focus to it. I usually find I psyche myself out with certain things and once I get in and do it it's not AS difficult as I thought. THIS multimaps project though is just out of my range at the moment. I'm also quite patient for help there... So get done what you want to get done and if you can find a break in the weather, we'd love your help!

The only issue regarding moving to BTS is the location of vanilla. For vanilla it would check BTS, but the file might be in warlords or original. Not sure what to do about that, but it's most likely fixable. Having xml files in different locations/names is just a matter of editing the xml file and might be needed even when moving between two mods using the same exe.

Yeah and we'd probably complicate things for you by having multiple modules and xml files for the same class type in the same folder, a few tricks you don't see normally. Plus we load XML with Xerxes... all this stuff was setup by guys who know a lot more than I, that's for sure.

 

[Nightinggale]

having multiple modules and xml files for the same class type in the same folder

I think it would not be such a big issue if each file is added to the xml list. I already planned for collecting types from multiple files due to civeffects, a concept I coded from scratch in Medieval Conquest.

Plus we load XML with Xerxes

The editor load the files independently from the mod's xml loading code. How much you modded that part shouldn't really matter. All it needs is the xml file itself and the schema file for it. If you ended up with a different schema file with different keywords, then it could be a problem.

 

[Thunderbrd]

If you ended up with a different schema file with different keywords, then it could be a problem.

Differnt to what exactly? I mean, obviously we've added a ton of new tags to nearly every schema, but we keep the all files throughout the mod compatible with the schemas in the core and usually, unless it hasn't been updated for a while, a module's schema file for a particular set of info types should be a replica of the one in the core (even if renamed perhaps).

 

[Nightinggale]

Differnt to what exactly?

There are multiple standards for xml schema files. The one vanilla uses is some Microsoft standard and as long as you keep that standard, it should work. It has nothing to do with adding more tags. It just has to be able to understand the schema to know what is added. Odds are that it is unchanged, but I still wanted to mention this when it was mentioned that the xml reader is modded.

 

[Thunderbrd]

There are multiple standards for xml schema files. The one vanilla uses is some Microsoft standard and as long as you keep that standard, it should work. It has nothing to do with adding more tags. It just has to be able to understand the schema to know what is added. Odds are that it is unchanged, but I still wanted to mention this when it was mentioned that the xml reader is modded.

I see. There are some schemas that use a slightly different method, where the one schema manages all files of that info type regardless of where that type is located BUT I'm not sure that wasn't reverted. Alberts2 is the one that was implementing that and IIRC, it may have presented issues elsewhere but I don't know for sure. We could ask him.

So, from what I'm gathering, you are creating a GUI for working with any XML file and the intention is to be able to let you work with XML like a form in Access allows you to manipulate data in a microsoft DB right? I mean... that's damned cool. Especially if you can then add in some ways to generate some reports as well.

The tricky thing about dropdowns is you'd have to be able to connect to multiple XML files... a dropdown list would usually need to know what XML types you're referencing and in what XML file right? So are you trying to make a structure that connects all XML files within a batch of folders? Modular stuff that edits other type entries in other files (WOC) would be a real pain with that I'd think.

If you do it right though, you could have a marketable program on your hands. That would apply to the modding of numerous other games as well wouldn't it?

 

[Nightinggale]

So, from what I'm gathering, you are creating a GUI for working with any XML file and the intention is to be able to let you work with XML like a form in Access allows you to manipulate data in a microsoft DB right? I mean... that's damned cool. Especially if you can then add in some ways to generate some reports as well.

Actually I use the low tech approach to just open the schema file as an xml file. I just code based on the MS standard keywords, like eltOnly. What would you want to get in a report?

The tricky thing about dropdowns is you'd have to be able to connect to multiple XML files... a dropdown list would usually need to know what XML types you're referencing and in what XML file right? So are you trying to make a structure that connects all XML files within a batch of folders? Modular stuff that edits other type entries in other files (WOC) would be a real pain with that I'd think.

My idea is that once it's complete, you can right click+control (or something) on a tag to open a menu. In this menu you can set that the tag in question is of type (some file). This setting is then saved in an xml file next to the schema file. Fairly easy to set up, can be added to a mod to apply to modded schema files and should be fairly easy to set up for all mods.

Each "file" is stored in the DLL as a vector of CvString. There are several approaches to solving this issue. The simplest is to append to the list instead of adding a new one if the type in question already exist.

 

[Thunderbrd]

What would you want to get in a report?

It would be nice to be able to query things like, What are the withdrawal values and tech prereqs on all throwing units. Things like that. Things to help with analysis.

My idea is that once it's complete, you can right click+control (or something) on a tag to open a menu. In this menu you can set that the tag in question is of type (some file). This setting is then saved in an xml file next to the schema file. Fairly easy to set up, can be added to a mod to apply to modded schema files and should be fairly easy to set up for all mods.

Each "file" is stored in the DLL as a vector of CvString. There are several approaches to solving this issue. The simplest is to append to the list instead of adding a new one if the type in question already exist.

So is this basically a way to mod from in-game? I was thinking this might be a side app.

 

[Nightinggale]

So is this basically a way to mod from in-game? I was thinking this might be a side app.

You missed the cleverness of this solution. I did start out with a crossplatform standalone application and it was working ok, but the amount of stuff people would have to install just kept on growing and growing. After having problems even moving it from one computer to another, I scrapped it and started over in C++ to allow compiling everything into an exe file. That idea never really took off due to some other unforeseen issues. I then started thinking rather than considering how to make it, then figure out how to distribute the editor to the people, who can benefit from it. The next idea was then to add it to the game DLL itself and not add any external library dependencies. The result is that if somebody can play the mod, they can use the editor without any other setup. While it limits the ability to set up the GUI (the civ4 python display engine is rather limited), it does solve the distribution problem entirely and it allows me to really learn the display engine, which can be used later for modding.

Screenshot of the first attempt.

Spoiler :

 

[Leoreth]

If I too may complain, I'd actually prefer just having a sane human editable data format (i.e. YAML) over any external editing tool.

 

[Nightinggale]

I view the biggest problems regarding XML editing to be getting the tag names correctly, the order correctly and if the tag is a Type from another file, then getting the string correctly and from the right file (like not mixing Unit and UnitClass). Switching to YAML will only solve the order issue, though arguably the order issue is in the EXE and not XML itself. The goal of the GUI editor is to solve all those issues.

Besides I didn't even consider switching away from XML. Too much work rewriting the XML reader and all the XML files. Having said all this, I generally agree that a proper text based system is to be preferred, but I think the amount of data in some of the XML files is just too big to be properly dealt with without assistance. I know the screenshot is from a simple file, but the goal is actually to deal with the complex files. I intentionally decided not to support globalDefineALT because it's easy to edit manually and doesn't suffer from the issues mentioned before.

 

[Leoreth]

Personally the most important issue is syntactic boilerplate, having to close every stupid XML tag and to never forget a pointy bracket or slash. Likewise, having to read the middle of any given entry (which is like 5% of the entire line for numerical data) to get the pertinent information. Having humans interact with machine optimised data formats like XML or even JSON is very, well, 2005.

Input validation is a different subject imo. XML has schemas, but they're almost completely useless in Civ4 because they are not aware of how the EXE is actually parsing the files, so I'd prefer not to have them at all.

I'm not sure if there actually is an order issue given by the EXE with the base game. At least in some XML schemas I've adapted stuff to make all tags optional and order agnostic, which is already a huge help. Aside from list fields (e.g. all yields in order), the EXE does look up by tag name, not order.

 

[Nightinggale]

I moved my postings about the GUI editor into the thread I already had for that purpose. It's here: https://forums.civfanatics.com/threads/perl-gui-progress.561743/page-2#post-14687829
It seems more fitting than hijacking this thread (though it's likely too late now)

I've adapted stuff to make all tags optional and order agnostic, which is already a huge help.

What do you mean by order agnostic?

 

[Nightinggale]

The link in the previous post has more or less turned into a development blog. I recommend taking a look if you want to know what the xml editor can do. It's full of screenshots. M:C has started using the editor for xml editing and it's both a lot faster and easier than editing the xml files manually. It has revealed some places where it could be better and I'm currently working on those. There are also some planned features, which haven't been coded yet.

Notes regarding a possible BTS port:
Actual work will not start until it's fully implemented in M:C. I don't want to fork this into two different versions and regularly (near daily) update both.

I have come up with a solution for modules and is currently adding support for those. Still haven't figured out how to locate the extra files, but if anything else fails, just opening modules, loop all files recursively to locate files of the right name should work. I have made a standard well working code structure for that in perl and it will likely work in C++ as well.

I added the ability to reorder list elements. It's really easy and intuitive to use as each list element has a button (currently contents of child Button or a default if it's missing). Just use the mouse to drag the button to the place you want the list element to be and it will move. The problem is that it relies on using addDragableButtonAt. It turns out that this function is added to the colonization exe file. As with other exe internals, there isn't much to do about it and this particular feature can't be used in BTS. Luckily it's not a critical feature.