Dev diaries: Retrospective and code base changes

[Leoreth]

It's almost two months since the last one so I thought I should write another dev diary. The last days have been quiet, but I've been busy. I was also on vacation, but that doesn't preclude being busy

Code base changes

This paragraph is going to be a little technical, but I want to emphasize how important that work was going forward. It's not pushed to the Git repo yet, but I am going to after a few more test runs tonight. What exactly has happened?

• All settler maps in CvRhyes.cpp have moved to SettlerMaps.py
• All war maps in CvRhyes.cpp have moved to WarMaps.py
• The region map and regional religion spread modifiers in CvRhyes.cpp have moved to RegionMap.py
• All civilization modifiers in CvRhyes.cpp have moved to Modifiers.py
• All constants influencing AI behavior (e.g. city placement) have moved to AIParameters.py
• Every definition of civ specific information in CvRhyes.cpp (loading time, star ratings) have moved to CIV4CivilizationInfos.xml
• The way text keys for interface stuff such as UHV texts are handled has been rewritten, the tedious specification of XML tags is now unnecessary, and they are automatically generated from the three letter key (i.e. EGY for Egypt) in CIV4CivilizationInfos.xml
• Removed all starting era constants in CvRhyes.cpp. They were used to determine which era screens to show at the start of the game (i.e. only the most recent one), which happens dynamically with smarter code now, to control the starting era of cities (influencing initial size and buildings), which is now always one less than the player's current era, and for some effects on the scaling of tech costs, which is determined dynamically at the start of the game from the defined starting techs
• Removed birth year constants from CvRhyes.cpp, they are initialized from the Python definition in Consts.py now
• This leaves only the following constants defined in the DLL: religion persecution preference, religion persecution weights against other religions, era modifiers for tech cost rubberbanding. I have turned them into static variables and moved them to CvRhyes.h. CvRhyes.cpp has in turn been deleted. That is a huge unwieldy source file completely eliminated.
• Made code that handles which civs have autoplay and which immediately start for a scenario dynamic, so explicit definition for every civ is not required anymore.
• Other parts of the C++ code have been rewritten to remove manual array initialization and the like. This hopefully means that adding a new civ only requires one DLL change: upping the number of civ slots by one!
• All previous DLL constants that have moved to Python are now persistently stored in either their player or plot object. Their value can be changed during the course of the game at any time. That means that none of the previous "constants" are actually constants anymore and instead be dynamic.
• This allowed to remove tons of special cases baked into the DLL code (respawns, civs with changed modifiers after the discovery of Astronomy, scenario scaling ...), making the affected code much cleaner and easier to understand. I have also removed previously commented out legacy code (Rhye's) to further improve readability.
• All constants that have moved to Python are now stored by civilization, not slot id. That made it much easier to store information for respawns, and no unnecessary double entries for respawned/unrespawned players is required. Respawns simply use the methods to change constants as soon as their civ changes (e.g. Aztec modifiers to Mexican modifiers).
• All area related constants in Consts.py (capital, flip zone, core, normal and broader area) moved to Areas.py and changed from slot based to civ based. This to handle respawns in a similar way as above, removing duplicate entries. I have also rearranged their definition so that the TL and BR tuples are always right next to each other.
• Streamlined and generalized different capital locations (start, relocation for AI, respawn) so that accessing code does not have to be aware of those differences, this should solve a couple of bugs with that.
• Moved starting technologies from RiseAndFall.py to Civilizations.py to separate program logic and information, and to have all technologies in one place. This also initializes the starting era as mentioned above.
• Implemented Areas.py in a way that a region can be queried and is returned as a plot list so that outside code does not have to be aware of the TL/BR rectangles or exception lists that may or may not exist. In turn, most Python code has been refactored to accept input as plot lists instead of TL, BR, exceptions triples.

If you have never modded RFC/DoC, you might not understand most of that. If you have only passing knowledge, I have highlighted what I feel are the most important parts.

In practice, this has the following benefits:

• Almost no DLL work required to add new civs. This is such a huge step up from vanilla RFC where this was a chore with relevant code scattered all over the source. It will help me later on and make the life of modmodders much easier.
• Also, much less overhead in repetitive work that contains no information where new civs are added (text keys, empty entries in irrelevant tuples (such as respawn), duplicate entries for rebirths ...)
• Existing civs can be modified without DLL changes. For me, that means I can make quick tests or adjustments without compiling the DLL. Saves time and makes it much more likely to bother with it. For contributors it means they can play around with modifiers and see what works better. Stuff that I would need to do myself (stability map fixes etc.) can now be done by a contributor even if they cannot compile DLLs.
• Modifiers can change during the game. I am not using this yet, but this might allow us to handle balance better, or enforce historicity.
• Tile stability can change during the game. This allows us to make stability editable in the world builder.
• Encapsulation of many repetitive tasks (give me the plots in this rectangle!) at the place where they belong saves work, makes the code more robust and avoids bugs, both existing and future ones.
• Modularization of huge constant files. CvRhyes.cpp is gone and now spread out over half a dozen Python files, hopefully making it easier to find the right place to edit and navigate it. Stuff from Consts.py has moved to new files (I plan to continue this later, but that is enough for now). Starting technologies defined in RiseAndFall.py also have their own file now.

I know this means the adding a civ tutorial needs to be rewritten, I will get to it in a while.

But seeing this written down is really a lot and I hope you can see that the time spent was worth it, even if nothing of it is noticeable in the game.

Git retrospective

I am really happy about the participation on Github. I did not expect that many pull requests, especially not with that degree of complexity and quality. Thanks again to all contributors.

I hope that nobody is put off by my tone in those discussions there. This is a little bit closer to professional programmer interactions so I naturally tend to be more critical and blunt. In the end, I am managing this project and I need to have both overview and control over its direction. This sometimes results in me being fussy over every detail.

Feedback welcome in that regard.

Feature retrospective

Let's look back at my stated plans for 1.14 from the last dev diary, along with my updated notes on them in bold:

Religion:

• new mechanism for religion spread: stronger historical and geographical impact (no Hinduism in Europe), disappearing religions -- I will probably use the K-Mod implementation for this, it is more efficient than BtS and uses the concept of religious grip on a city, which I can adapt for my purposes
• New Christian schism: for a more logical and rewarding relationship between Catholicism, Orthodoxy and the Apostolic Palace
• Minor religions (may require new mechanics for spawning or branching off religions)
• Rewards for religious unity over religious diversity
  The latter two likely won't make the cut for release, I will explain why further down.

Utility improvements:

• better city name manager with less maintenance: auto-discovery of reverse name translations, auto-discovery of native tile names from other languages -- done!
• more dynamic names with less maintenance overhead: e.g. separate conditions for title (like "Empire of %s" or "Protectorate of %s") and used short name (e.g. "England" or "Britain"), increasing available combinations while reducing maintenance on game text -- done!
• better world builder: I'm currently looking at adapting the one from History Rewritten -- which is Platybuilder, as merijn pointed out. Then he went ahead and incorporated it, so: thanks, and done!
• better civilopedia: I like the one HR uses here as well
• Xyth also did a great espionage advisor recently (he's just very good with interface stuff)
• RFC specific world builder features: display and edit tile stability, output stability maps. Hopefully this will allow us to edit stability/settler maps in world builder -- merijn has already worked on it and I need to take a closer look at what he's done. In the meantime, I have provided the DLL groundwork for changing settler maps and the like.
• New solution for dynamic Great People names: keep the code in the DLL but remove the need for duplicate GP unit types. If possible, even make the names era specific to avoid inappropriate names for respawns or later scenarios -- I have been thinking about this but no elegant solution presented itself, I will likely keep this as is for now.

Going forward, I will first resolve the remaining pull requests and bug reports, and update the documentation on the forum in a couple of places. Then I will finish the rest of the utility section, which should be rather quick. I really like the pace of progress right now.

Future plans for 1.15

In the meantime, I have changed what I have in mind for 1.15 quite substantially. I will make some fairly major changes to the mod in that version, and it therefore makes sense to delay some other plans until those are through, otherwise I would have to do a lot of work twice. I will go into more detail once 1.15 draws closer.

 

[zinedine]

Blessings be upon you

 

[DC123456789]

Excellent.

 

[clanky4]

Almost no DLL work required to add new civs. This is such a huge step up from vanilla RFC where this was a chore with relevant code scattered all over the source. It will help me later on and make the life of modmodders much easier.

A week long chore which is nigh impossible to figure out if you messed something up. Simply amazing work here.

 

[Leoreth]

Yeah I'm very happy about that too.

After some thinking, I lied though: you also have to add a PlayerTypes constant in CvEnums.h for your new slot, and probably increment the buildings constant if your new civ comes with a new UB.

 

[Novicenoble]

Oh huge one D:

 

[Leoreth]

Another thing I forgot to mention: I removed the huge borders table that had a 1-5 proximity value for every combination of civs. With around 50 civs in the game you had to add 100 entries there for every new civ, the values were often arbitrary and the size made the table itself hard to handle.

It turned out that it was only used in only two places: vassalization rules and AI war target priorities. Instead of the predefined values I changed it so that proximity was calculated from their actual city distances, which reflects the actual state of the game more closely anyway.

Now there are three proximity values:
- civs are CLOSE if they share a border (increases AI war target weight)
- civs are DISTANT if their closest cities are more than half the maximum map distance away from each other* (reduces AI war target weight, cannot be vassals)
- no special effects if neither category applies

*approximately, explanation:

Spoiler :

I don't look up all cities to determine the actual closest pair, because that is computationally more expensive. Instead, for player A, I search the closest city of player B to A's capital. Then I search for the closest city of player A to that city of player B and calculate their distance. The same in reverse for B and A (since it's an approximation there might be edge cases where a different pair is found). The I use the minimum of those distances, if different. In most circumstances that should produce the correct result without having to compare every combination of cities.

 

[ezzlar]

Thats a great improvement as the AI likes to waste its time by declearing war on Tibet whilst in Europe.

 

[Hightower]

Another thing I forgot to mention: I removed the huge borders table that had a 1-5 proximity value for every combination of civs. With around 50 civs in the game you had to add 100 entries there for every new civ, the values were often arbitrary and the size made the table itself hard to handle.

It turned out that it was only used in only two places: vassalization rules and AI war target priorities. Instead of the predefined values I changed it so that proximity was calculated from their actual city distances, which reflects the actual state of the game more closely anyway.

Now there are three proximity values:
- civs are CLOSE if they share a border (increases AI war target weight)
- civs are DISTANT if their closest cities are more than half the maximum map distance away from each other* (reduces AI war target weight, cannot be vassals)
- no special effects if neither category applies

*approximately, explanation:

Spoiler :

I don't look up all cities to determine the actual closest pair, because that is computationally more expensive. Instead, for player A, I search the closest city of player B to A's capital. Then I search for the closest city of player A to that city of player B and calculate their distance. The same in reverse for B and A (since it's an approximation there might be edge cases where a different pair is found). The I use the minimum of those distances, if different. In most circumstances that should produce the correct result without having to compare every combination of cities.

How does this work with the AI's decision (or not) to declare war on the Independents, who don't have a capital?

 

[Leoreth]

Independents are completely excluded from the original war target AI (like barbarians). The AI will just declare war on them if they decide to take one of their cities.