[We the People] Development "diary"

As said my knowledge of modding is quite limited.
Ray provided guidance and training - :thumbsup: - and as such I was able to finish adding a quest on colonial growth.

Colony growth.png
 
I was able to finish adding a quest on colonial growth.
Congratulations and great job. :thumbsup:

The Python Event System is really a great system to add flavour and small new game content.
It is quite powerful and versatile and also easier to use than most people believe.

The effort vs. benefit of that system is way better than almost anything else we have - which explains why we have so many Python Events and Quests.
DLL Diplomacy Dialogues may look bettter and can use all DLL capabilities but also easily take 4 to 5 times as much effort compared to a Python Event.
 
Last edited:
Hi guys,

one of our community members also started a nice YouTube series on the mod with newest release WTP 4.1.

So if you want to know what WTP 4.1 is about, maybe give it a look.
Also it would be nice to support such initiatives from community members by giving them a like. :)

Best regards
WTP team
 
Much appreciated, and thanks a lot for this.
Off to download it & check it out on the big screen
 
I finished my ingame xml editor. Well it might never be 100% finished (like any mod project), but it can be used as it is in the current state and it is now part of the develop branch.

Dating back to even before WTP, I noticed how modders have a tendency to use a subset of the available tags. In particular optional tags became a pain to use because you have to look them up in the schema and because people don't do that, optional tags tend to end up overlooked. Another issue is that it can be hard to determine which type goes into which tag, like is a tag UnitTypes or UnitClassTypes. To make matters even worse, vanilla xml reader won't care if you use the wrong type in a tag. It will just end up making a game other than what the modder wanted.

To fix this, I decided to make an xml editor of my own, one which will present all tags as they are written in the schema. I did a few attempts, which ultimately ended up in nothing (though I reused code from the last attempt). You can read about that here.


Now I made an editor, which is integrated into We The People itself. It's working and I merged it into the develop branch. It does require some setup to know what each tag is supposed to do, but that's easily done as the setup for that is in the right click menu on each tag. That setup is saved meaning once somebody set it up, other people can use it. Also it allows adding mouseover help to each tag.

It's accessible ingame in the same menu as world builder. No setup needed to enable this.

FAQ

Can I use this with my mod?
Yes. The menu has an option to create a settings file, which makes it work on the xml files of another mod.
Can I use this with Beyond the sword?
Yes. As long as the schema file obeys vanilla rules (support for XSD/W3C schemas might happen)


Do I need to use We The People/Colonization to get this to work?
Currently yes. I have no plans to create or maintain a standalone version of this editor. I'm aware that there are BTS modders, who doesn't have Colonization and I wouldn't be against the idea of a standalone BTS editor mod. However I do not plan to make one myself. Most of the source files are standalone, so just copying them into a BTS mod shouldn't be a major issue.
There is one major difference between Colonization and BTW. Only colonization supports drag and drop widgets. A BTS mod would have to have code to disable any feature relying on this. I did add COLONIZATION_EXE and CyXMLEditor.isColonizationExe() so it should be possible to create a single source code, which supports both. That is if somebody is willing to spend the time making it. Obviously it would be in the best interest of everybody if a fork would stay so compatible that getting an update from the other one is just copying the files.
 

Attachments

  • ints and bools.png
    ints and bools.png
    236.3 KB · Views: 266
  • list.png
    list.png
    224 KB · Views: 257
  • UnitClass selector.png
    UnitClass selector.png
    230.7 KB · Views: 224
wow that looks superb, where where you 10 years ago ? :)
I was starting out modding. One of my regrets is not starting earlier than I did.
 
so im considering to go ahead and try to make it into bts,
where should i get only code of it? or better yet, where should i start?
First of all, we do have the We The People discord. Questions might be answered faster there.

As for the source code: https://github.com/We-the-People-civ4col-mod/Mod/tree/develop

The C++ source files all start with CyXML with the exception of tinyxml2.cpp/h. The last is lightly modified to compile into the dll without any modifications to the makefile. Also I changed space indentation to tabs. CvButtonPopup opens the screen.

For python, it's just assets/python/editor. It adds a screen enum value and code to open the screen, but other than that I don't think it touches any python file outside of this new folder.

It requires the following widgets to be python exposed:
Code:
        <!-- xml editor start -->
        <WidgetType>WIDGET_XML_EDITOR</WidgetType>
        <WidgetType>WIDGET_XML_EDITOR_BOX</WidgetType>
        <WidgetType>WIDGET_XML_OK</WidgetType>
        <WidgetType>WIDGET_XML_CANCEL</WidgetType>
        <WidgetType>WIDGET_XML_DELETE</WidgetType>
        <WidgetType>WIDGET_XML_CLONE</WidgetType>
        <WidgetType>WIDGET_XML_MENU_ITEM</WidgetType>
        <!-- needs to be in order down, zero, up because it makes widget - zero = +1 (up) or -1 (down) -->
        <WidgetType>WIDGET_XML_DOWN</WidgetType>
        <WidgetType>WIDGET_XML_ZERO</WidgetType>
        <WidgetType>WIDGET_XML_UP</WidgetType>
        <!-- xml editor end -->
Yeah I added those to GlobalTypes because then it's scripted to be exposed to both dll and python without any other modification. That is indeed a WTP only feature.
 
was starting out modding. One of my regrets is not starting earlier than I did.
i know my friend, was kidding around,
im here since 2006 or so.
i have followed your work since then.

--
cool thanks for the details and the source code,
ill get to that very soon i hope.

cheers :)
 
I started to investigate the code, which handles multithreading. I added a compile time switch to bring back single threaded code and as such I have unlocked the ability to compare the performance impact from using more than one CPU core. I started out getting multithreaded pathfinding to take 0.06 seconds longer, an increase in wait of 4.3%. Well that's off to a bad start and I need to do some more digging.

Getting hold of a more proper late game savegame, I started to see some completely different numbers. Reduction in wait time compared to single threaded:
  • Pathfinding: 10.2%
  • Moving colonists to best professions: 26.3%
  • Both: 36.6%
(Numbers doesn't add up completely due to rounding and run to run variance)

It would seem that the pathfinding is actually slower in early game while it becomes a not insignificant boost in late game. The colony handling of colonist professions seems to be faster in all cases, through the boost in percentage increase as the game progresses. It seems that this is more or less like we expected. The difference is that now we have numbers to not only proof the speed boost, but also get an idea of how much the boost is from each part of the code.

It should be noted that the numbers given are unlikely to be what you will experience. How much the game will be faster will heavily depend on number of units, map size etc as well as your CPU. The important part here is that even if somebody is unlucky and only get half of this boost (fewer CPU cores etc), then it's still a significant boost and worth the hassle of using multiple threads.
 
Especially on large maps the K-Mod pathfinder itself spends alot of time on performing memset.

Reverting these changes
and using a memory pool for the FAstarNodes instead of new/delete can improve performance.
 
using a memory pool for the FAstarNodes instead of new/delete can improve performance.
I have been wondering about adding a memory pool for EnumMap for quite a while by now. The main reason I haven't touched it yet is due to my idea not being thread safe, but I think I solved that problem a few days ago.

A memory pool in a multithreaded setup will likely have to have some locks to make it thread safe, but at least those are faster than memory allocation.
 
Turned out that we had an issue where computers with Chinese locale (and likely most non-European languages) could not load our text xml files. I rewrote the code to read all the text into the game to have it in the dll file rather than relying on the exe for xml access. Problem solved and now that part of the code is completely under our own control.

A nice side effect is that the time spend reading the text is now reduced by 83%. This makes me wonder how vanilla messed up xml access in order to make it that slow. Even more interesting what will happen if all xml loading is done without using the exe? I have done a bit of work towards writing a custom xml reader, but it's incomplete and not in use at all yet. The goal was to make the code cleaner and less prone to bugs, but if we can boost speed at the same time, then it would be great. Time spend reaching the main menu doesn't seem to be the right location to spend a lot of effort into optimization. It's better to spend time optimizing the code, which executes between turns to reduce that wait so the xml optimization isn't top priority.
 
Even more interesting what will happen if all xml loading is done without using the exe?
I don't remember which but a few xml files are being read by the exe itself.

The rest of the xml loading process can be done without involving the exe. Even switching to something like json should be doable.
 
I have been neglecting this thread, but that doesn't mean I have been idle.

Currently I'm working on a new project I started, which I call autogenerate.exe. It is a C++ project, which is compiled when the makefile starts and it outputs C++ code and makefile files. This aids programming and compiling WTP.

XML and code awareness
It scans both the mod's source code as well as all the xml files. A new files.xml makes it possible to configure how files are handled.

New enums
It creates classes, each containing an enum. For instance UnitType contains all types from UnitTypes and has a single variable, so it is intended to be interchangeable. The difference is that a class can hand member functions. For instance looping would be
PHP:
for (UnitType eUnit; eUnit.next();)
This will loop through the entire list and no typecasting needed. Other features are on the drawing board, but this part is still in the early days and isn't used yet.

EnumMap rewrite
EnumMap is the concept of having arrays of length matching xml files (not a new concept). It has gone through a few iterations, each fixing the mistakes of the previous one. Currently it's based on template classes and class inheritance. Now autogenerate.exe is generating them, so it's still templates, but no class inheritance and all the code is specialized functions. This makes the code way more readable and it's also easier to maintain and upgrade. This has reduced the memory usage so now all EnumMaps will only use 4 bytes. Well 8 for bools if they are in the range 33-64, which is useful for say CvPlot storing a bool for which teams have seen the plot. Since it doesn't have to use the precompiler to configure anything automatically and instead relies on C++ to generate text, the need for creative coding is gone. It also allows for using faster code as each specialized function can be optimized to the specifics of said xml file, like is length known at compile time, which may or may not be used to reduce the code enough to inline functions.

EDIT: forgot to mention, one major change to EnumMap is that I now consider thread safe coding. Since I wrote the original, we have split off to make the mod multi threaded, but the current EnumMap is optimized for low memory usage and sacrifices multi thread support in the process. The new version aims at being thread safe for 4 byte variables, such as ints. It's not a flaw in the original, but rather the requirements have changed.

Fastdep replacement
Fastdep.exe is a program used to list which C++ files includes which headers recursively and that way will only need to be recompiled if an included header is changed. However it's not a super good program as crashing is an issue, at least for some people and the output is not configurable. I let fastdep.exe do the job instead and it is custom designed for our use case. For instance usage of the precompiled header has become optional on a per file basis. If a cpp file doesn't include CvGameCoreDLL.h, then the resulting makefile will just not use the pch file, which is an improvement over the current makefile, which will cause it to make a compile error. Since the output isn't depending on the pch file, those files can be compiled in parallel with generating the pch file, which just happens to take forever and only use a single CPU core. This will at least eventually make compiling faster.

InfoArray rewrite
This is a write once, read many times array of a non-fixed length. It's designed to read an array from xml, like list of UnitCombatTypes a promotion can apply to, through it can have much more complex data structures, like <FeatureTypes, TerrainTypes, int> entries in the list. Like EnumMap, this class has seen a few iterations and is currently a hybrid of template class inheritance, which in turn inherits the pre-template version as protected so the type unsafe version can't be called from outside the class. It works, but it's certainly not reader friendly and not expansion friendly.
Work on InfoArrays have not yet started. I want to get the other parts to work flawlessly before including yet another project.


So work is progressing into having much better code, which in turn will make programming both faster and more fun. It will also hopefully make it harder to introduce bugs as the new code is way more restrictive on type correctness as well as proper use of const.
 
Last edited:
I came up with an idea yesterday. Simply put add game options, which are only visible in scenarios. This way we can have extra options in games where the map options can't be used, which more specifically will be all the premade maps.

I instantly used this idea to move the colony catchment radius from the txt file into the game GUI. Even better future changes will allow us to add them as both map and game options, hence available to be set for both random and premade maps. Next up for that treatment is selection of new vs old movement cost system.

image.png
 
Hi

I am in the year of 1808, basically the whole world map is revealed, i am playing in a big map, and i need to wait almost 4 minutes until the next turn.
Is there any way to make it faster?
I have attached the save game file too, thanks.

AMD Ryzen 5 5500U with Radeon Graphics, 2GHz. Memory: 16GB
 

Attachments

Last edited:
Hi

Thanks for looking into this. I tried on a smaller map (Huge), and now i am again at 3 minutes of loading time.
What is the recommended map size for my specs?
 
Only you can know what your pain threshold is. The next smaller map size is certainly faster, but whether that's still fun is another matter ...
 
Back
Top Bottom