[We the People] Development "diary"

[ArchdukeLiet]

As said my knowledge of modding is quite limited.
Ray provided guidance and training - - and as such I was able to finish adding a quest on colonial growth.

 

[raystuttgart]

I was able to finish adding a quest on colonial growth.

Congratulations and great job.

The Python Event System is really a great system to add flavour and small new game content.
It is quite powerful and versatile and also easier to use than most people believe.

The effort vs. benefit of that system is way better than almost anything else we have - which explains why we have so many Python Events and Quests.
DLL Diplomacy Dialogues may look bettter and can use all DLL capabilities but also easily take 4 to 5 times as much effort compared to a Python Event.

 

[raystuttgart]

Hi guys,

one of our community members also started a nice YouTube series on the mod with newest release WTP 4.1.

So if you want to know what WTP 4.1 is about, maybe give it a look.
Also it would be nice to support such initiatives from community members by giving them a like.

Best regards
WTP team

 

[HiRezAudio]

Much appreciated, and thanks a lot for this.
Off to download it & check it out on the big screen

 

[Nightinggale]

I finished my ingame xml editor. Well it might never be 100% finished (like any mod project), but it can be used as it is in the current state and it is now part of the develop branch.

Dating back to even before WTP, I noticed how modders have a tendency to use a subset of the available tags. In particular optional tags became a pain to use because you have to look them up in the schema and because people don't do that, optional tags tend to end up overlooked. Another issue is that it can be hard to determine which type goes into which tag, like is a tag UnitTypes or UnitClassTypes. To make matters even worse, vanilla xml reader won't care if you use the wrong type in a tag. It will just end up making a game other than what the modder wanted.

To fix this, I decided to make an xml editor of my own, one which will present all tags as they are written in the schema. I did a few attempts, which ultimately ended up in nothing (though I reused code from the last attempt). You can read about that here.

XML GUI progress

I will use this thread to keep track of my perl GUI work. As mentioned elsewhere I managed to find wxPerl, which allows usage of wxWidgets in perl scripts. In other words it is a library, which allows easily created GUIs, which works cross platform (windows, mac, linux and some other platforms)...

forums.civfanatics.com

Now I made an editor, which is integrated into We The People itself. It's working and I merged it into the develop branch. It does require some setup to know what each tag is supposed to do, but that's easily done as the setup for that is in the right click menu on each tag. That setup is saved meaning once somebody set it up, other people can use it. Also it allows adding mouseover help to each tag.

It's accessible ingame in the same menu as world builder. No setup needed to enable this.

FAQ

Can I use this with my mod?
Yes. The menu has an option to create a settings file, which makes it work on the xml files of another mod.
Can I use this with Beyond the sword?
Yes. As long as the schema file obeys vanilla rules (support for XSD/W3C schemas might happen)

Do I need to use We The People/Colonization to get this to work?
Currently yes. I have no plans to create or maintain a standalone version of this editor. I'm aware that there are BTS modders, who doesn't have Colonization and I wouldn't be against the idea of a standalone BTS editor mod. However I do not plan to make one myself. Most of the source files are standalone, so just copying them into a BTS mod shouldn't be a major issue.
There is one major difference between Colonization and BTW. Only colonization supports drag and drop widgets. A BTS mod would have to have code to disable any feature relying on this. I did add COLONIZATION_EXE and CyXMLEditor.isColonizationExe() so it should be possible to create a single source code, which supports both. That is if somebody is willing to spend the time making it. Obviously it would be in the best interest of everybody if a fork would stay so compatible that getting an update from the other one is just copying the files.

 

[keldath]

wow that looks superb, where where you 10 years ago ?

so im considering to go ahead and try to make it into bts,
where should i get only code of it? or better yet, where should i start?

 

[Nightinggale]

wow that looks superb, where where you 10 years ago ?

I was starting out modding. One of my regrets is not starting earlier than I did.

[mod] Religion and Revolution Extended (modmod of Religion and Revolution)

Out of date!!! Use Religion and Revolution 1.6 instead As previously announced I decided to make a mod of the mod Religion and Revolution. Here it is: the modmod Religion and Revolution Extended. This is based on Religion and Revolution 1.5, which in turn is based on The Authentic...

forums.civfanatics.com

 

[Nightinggale]

so im considering to go ahead and try to make it into bts,
where should i get only code of it? or better yet, where should i start?

First of all, we do have the We The People discord. Questions might be answered faster there.

As for the source code: https://github.com/We-the-People-civ4col-mod/Mod/tree/develop

The C++ source files all start with CyXML with the exception of tinyxml2.cpp/h. The last is lightly modified to compile into the dll without any modifications to the makefile. Also I changed space indentation to tabs. CvButtonPopup opens the screen.

For python, it's just assets/python/editor. It adds a screen enum value and code to open the screen, but other than that I don't think it touches any python file outside of this new folder.

It requires the following widgets to be python exposed:

        <!-- xml editor start -->
        <WidgetType>WIDGET_XML_EDITOR</WidgetType>
        <WidgetType>WIDGET_XML_EDITOR_BOX</WidgetType>
        <WidgetType>WIDGET_XML_OK</WidgetType>
        <WidgetType>WIDGET_XML_CANCEL</WidgetType>
        <WidgetType>WIDGET_XML_DELETE</WidgetType>
        <WidgetType>WIDGET_XML_CLONE</WidgetType>
        <WidgetType>WIDGET_XML_MENU_ITEM</WidgetType>
        <!-- needs to be in order down, zero, up because it makes widget - zero = +1 (up) or -1 (down) -->
        <WidgetType>WIDGET_XML_DOWN</WidgetType>
        <WidgetType>WIDGET_XML_ZERO</WidgetType>
        <WidgetType>WIDGET_XML_UP</WidgetType>
        <!-- xml editor end -->

Yeah I added those to GlobalTypes because then it's scripted to be exposed to both dll and python without any other modification. That is indeed a WTP only feature.

 

[keldath]

was starting out modding. One of my regrets is not starting earlier than I did.

i know my friend, was kidding around,
im here since 2006 or so.
i have followed your work since then.

--
cool thanks for the details and the source code,
ill get to that very soon i hope.

cheers

 

[Nightinggale]

I started to investigate the code, which handles multithreading. I added a compile time switch to bring back single threaded code and as such I have unlocked the ability to compare the performance impact from using more than one CPU core. I started out getting multithreaded pathfinding to take 0.06 seconds longer, an increase in wait of 4.3%. Well that's off to a bad start and I need to do some more digging.

Getting hold of a more proper late game savegame, I started to see some completely different numbers. Reduction in wait time compared to single threaded:

• Pathfinding: 10.2%
• Moving colonists to best professions: 26.3%
• Both: 36.6%

(Numbers doesn't add up completely due to rounding and run to run variance)

It would seem that the pathfinding is actually slower in early game while it becomes a not insignificant boost in late game. The colony handling of colonist professions seems to be faster in all cases, through the boost in percentage increase as the game progresses. It seems that this is more or less like we expected. The difference is that now we have numbers to not only proof the speed boost, but also get an idea of how much the boost is from each part of the code.

It should be noted that the numbers given are unlikely to be what you will experience. How much the game will be faster will heavily depend on number of units, map size etc as well as your CPU. The important part here is that even if somebody is unlucky and only get half of this boost (fewer CPU cores etc), then it's still a significant boost and worth the hassle of using multiple threads.

 

[alberts8]

Especially on large maps the K-Mod pathfinder itself spends alot of time on performing memset.

Reverting these changes

Changed the core of KmodPathFinder to use a single block of memory. · karadoc/Civ4-K-Mod@c8a3e19

node_map is now a multi_array<FAStarNode, 2>, and the path finder no longer uses any shared_ptr. This change will (usually) use a bit more memory than the previous implementation, but I expe...

github.com

and using a memory pool for the FAstarNodes instead of new/delete can improve performance.

 

[Nightinggale]

using a memory pool for the FAstarNodes instead of new/delete can improve performance.

I have been wondering about adding a memory pool for EnumMap for quite a while by now. The main reason I haven't touched it yet is due to my idea not being thread safe, but I think I solved that problem a few days ago.

A memory pool in a multithreaded setup will likely have to have some locks to make it thread safe, but at least those are faster than memory allocation.

 

[Nightinggale]

Turned out that we had an issue where computers with Chinese locale (and likely most non-European languages) could not load our text xml files. I rewrote the code to read all the text into the game to have it in the dll file rather than relying on the exe for xml access. Problem solved and now that part of the code is completely under our own control.

A nice side effect is that the time spend reading the text is now reduced by 83%. This makes me wonder how vanilla messed up xml access in order to make it that slow. Even more interesting what will happen if all xml loading is done without using the exe? I have done a bit of work towards writing a custom xml reader, but it's incomplete and not in use at all yet. The goal was to make the code cleaner and less prone to bugs, but if we can boost speed at the same time, then it would be great. Time spend reaching the main menu doesn't seem to be the right location to spend a lot of effort into optimization. It's better to spend time optimizing the code, which executes between turns to reduce that wait so the xml optimization isn't top priority.

 

[alberts8]

Even more interesting what will happen if all xml loading is done without using the exe?

I don't remember which but a few xml files are being read by the exe itself.

The rest of the xml loading process can be done without involving the exe. Even switching to something like json should be doable.