[mod] Religion and Revolution Extended (modmod of Religion and Revolution)

[raystuttgart]

The issue is not really slow code, but more the fact that the AI calls it over and over. It calculates yields used to make certain professions, like scouts uses 50 horses, 0 lumber etc.

There is nothing touched more often in Civ4Col logic than the logic about professions.

That is why I was extremely reluctant about having new features that add new computation intensive logic about professions.
(e.g. Inventions that change the professions allowed)

I wasn't too happy with the vanilla code to begin with and this certainly didn't improve my impression of it.

Well, I guess all DLL modders share that impression.

The number of professions is now hardcoded into the dll as I needed that info at compiletime. Adding more professions will need a new dll file.
...
I plan to do this with more data from the XML files as knowledge of the numbers at compiletime provides faster code and open up some more coding features.

Well, as a DLL modder, it would probably be quite easy to change that.

But it will be a really bad design decision for others that don't do DLL modding and might be interested to use your modmod for their own private version.
The majority of people that occassionally do modding does not know anything about the DLL.

Thus you might lower interest of others to participate in your project ...
Maybe you might reconsider this.

Would it be possible to have it directly identify the # of professions on the initial load of XML data then store that in a global variable.

Things like that are usually a good idea.

I tried to do that with many XML settings to avoid reading from XML or the relatively slow FVarSystem.
This is one of the main points by which we have increased performance in RaR.

 

[Nightinggale]

Would it be possible to have it directly identify the # of professions on the initial load of XML data then store that in a global variable (or if it's needed to declare an array size when compiling, set a comfortable max threshold that most mods won't exceed?)

It needs to hardcode the array length at compile time. However now that I think about it, nothing really bad will happen from using a bigger number. The array contains pointers and if coded correctly those pointers will not be accessed and will remain as NULL pointers. This mean each player will use 4 bytes for each extra pointer, which is within acceptable waste. It will slow down player init a little bit though, but that's so little and so rarely called that I don't really care about that.

It would then have to check if the allocated size is the same or bigger than expected by the XML. It would also mean I should review my additions to remove the new hardcoded number from some other places.

For me as a modmodder trying to make a conversion with limited c++ knowledge, turn speed wasn't as much an issue as many hard to find crashes caused by various bits of hardcoded content buried throughout the vanilla DLL

Right now the XML loading ends with a check and it will make a popup like this if you add a profession

XML has 70 professions while CvEnums.h:NUM_PROFESSION_TYPES is set to 69.
Recompile DLL to avoid bugs/crashes.

CvEnums.h has a line saying

NUM_PROFESSION_TYPES = 69,

If you get that error popup (not assert, popup), then change that line to 70 and recompile. It doesn't take much C++ knowledge to do that.

 

[raystuttgart]

It doesn't take much C++ knowledge to do that.

That is true, but a huge part of the modders does not even know how to compile a DLL.

It is your choice of course.

 

[Commander Bello]

That is true, but a huge part of the modders does not even know how to compile a DLL.

I have to assist Ray in that.

Many people are shy of meddling with the DLL.
Furthermore, as mods have a tendency to be copied and re-used and changed and copied again, you'll never know which way your DLL will go. It could theoretically end in a mod "XYZ" being appealing to a new modder who just doesn't make use of it because of the (theoretical) need to change the DLL.

So, this idea of yours

It needs to hardcode the array length at compile time. However now that I think about it, nothing really bad will happen from using a bigger number. The array contains pointers and if coded correctly those pointers will not be accessed and will remain as NULL pointers. This mean each player will use 4 bytes for each extra pointer, which is within acceptable waste. It will slow down player init a little bit though, but that's so little and so rarely called that I don't really care about that.

It would then have to check if the allocated size is the same or bigger than expected by the XML. It would also mean I should review my additions to remove the new hardcoded number from some other places.

sounds like a good idea.

 

[Nightinggale]

That is why I was extremely reluctant about having new features that add new computation intensive logic about professions.
(e.g. Inventions that change the professions allowed)

Luckily the cache mean the actual calculations are done quite rarely. I don't mind making the calculations more complex, as long as it works with the cache, which mean it profession X for player Y needs the same each time and isn't affected by say buildings in the colony. I completely agree that it really is a dangerous function to make just a bit slower without the cache.

Would it be possible to have it directly identify the # of professions on the initial load of XML data then store that in a global variable.

Things like that are usually a good idea.

That would actually be easy to do. I have considered inlining some of those small often called global functions and reducing the return value to a cached int would make it even better. I think that would be just as fast as hardcoding the value (well almost) except that it involves a risk of cache misses. However the more often it's used, the more likely it is to be in the cache. I will also place such caches next to each other in memory, which mean the most likely scenario would be to have all or nothing in the cache, in which case it would be unlikely to get a cache miss.

I will implement this and do the same to some other XML generated arrays (unit types and so on). There is one way to find out if the impact is great enough to make us able to tell the difference.

I decided to remove the hardcoded numbers. Maybe I should allocate memory for 84 professions? That would leave 15 free slots, enough for most people. Sure I could set it to something insane like 200 and it would never run out, but that sounds like overkill to me.

Btw the number of yields is hardcoded and I have no intension of changing that.

 

[Nightinggale]

I learned something new. Turns out that the player constructor is called before reading the XML files while player init is called after reading the XML files. I'm not 100% sure, but it's likely what caused me to look for a hardcoded value in the first place. Either way I managed to find a way to remove the hardcoded value and the DLL is once again open for XML changes.

I also got rid of the hardcoded value for number of units, though the price for that one is adding 4 bytes to each city (colonies+native settlements). I couldn't find a clean way to get around that.

To my knowledge there is no longer any hardcoded values like that except for the number of yields.

 

[Commander Bello]

Thanks not only for what you are doing, but for your explanations as well!

I think a good explanation for the reasoning behind the doing is worth nearly as much as the doing itself.

 

[orlanth]

Great! My thanks as well for your modding contributions. Reducing hardcoding will really enhance the versatility for modmods, adding a few bytes seems a small price to pay for the added flexibility.

Turns out that the player constructor is called before reading the XML files while player init is called after reading the XML files.

IIRC, one other remaining bug in RaR/TAC was something about the French civilization trait not applying correctly during player creation - Ray could you refresh my memory about that, I wonder if what you found there could be related to that issue?

 

[raystuttgart]

Ray could you refresh my memory about that, I wonder if what you found there could be related to that issue?

It is related to the trait "Cooperative" which gives a small relationship bonus to Natives.

To problem occurs only for human player when he chooses the French.
(It does not occur to AI.)

It is related to initialization order of Players.
(Human Player is initialized before Native AI players have been.)

Discussion starts here.

 

[Nightinggale]

I managed to get profiling to work with the stack. Now I can see how long a function is used including the time it spends waiting for the functions it called itself.

I noticed calculating warehouse capacity took 1 minute 54 seconds. This changes when buildings are added/removed and on colony creation/load. Fairly simple to cache, so I did.
It actually took another two seconds from the next turn events.

I merged it with the cache for teaching level as they should update at the same time. I made a function to call those two functions meaning if some future cache needs to update on building change it only needs to be added in one place and it will hook into the code at the right places.

IIRC, one other remaining bug in RaR/TAC was something about the French civilization trait not applying correctly during player creation - Ray could you refresh my memory about that, I wonder if what you found there could be related to that issue?

That's unrelated. My issue was about the order stuff done inside one player while the French issue is the order of players. Two completely unrelated issues even though they sound similar.

 

[Kailric]

That is why I was extremely reluctant about having new features that add new computation intensive logic about professions.
(e.g. Inventions that change the professions allowed)

This is an interesting thread, I'll have to go back and read from the start. I myself am no programmer but rather a tinker. I completed my first major project, the Privateer mod, not even knowing anything about Debug. It was nuts, I know. I guess I just have a knack for "if this, then do that", and just knowing that for the most part the computer only reads one line at a time, so you have to keep order in mind always. I am, however, enrolled in college and will eventually take some programing classes.

One thing in my Medieval mod that I had to add that gets called a lot is a for loop of the Civics(I used Civics for Techs). It has to loop through nearly every civic to test if say, a profession is allowed or disallowed. Then checks to see if the Player has learned that tech. I started thinking on ways I could do this without having to cycle through the Civics each time. Creating arrays for player allowed Buildings, Professions, Units, Promotions, Yields, etc was one idea. I made sure though to that if a Tech disallows a Profession that it allows a replacement or upgrade. Then the code just cycles through the players units and forces a profession change. This fixed a lot of errors I was getting for professions not allowed.

Anyway, like I mentioned to ya, a lot of my code needs overhauled. I create stuff in a hurry just to test if I liked it or not or it would even work then if I liked it I left it as is for the moment... things like that.

Thanks for all the info you are sharing, I'm loving it.

 

[orlanth]

Originally Posted by raystuttgart View Post
That is why I was extremely reluctant about having new features that add new computation intensive logic about professions.
(e.g. Inventions that change the professions allowed)

Originally Posted by Nightinggale
Luckily the cache mean the actual calculations are done quite rarely. I don't mind making the calculations more complex, as long as it works with the cache, which mean it profession X for player Y needs the same each time and isn't affected by say buildings in the colony.

One thing in my Medieval mod that I had to add that gets called a lot is a for loop of the Civics(I used Civics for Techs). It has to loop through nearly every civic to test if say, a profession is allowed or disallowed. Then checks to see if the Player has learned that tech. I started thinking on ways I could do this without having to cycle through the Civics each time. Creating arrays for player allowed Buildings, Professions, Units, Promotions, Yields, etc was one idea. I made sure though to that if a Tech disallows a Profession that it allows a replacement or upgrade. Then the code just cycles through the players units and forces a profession change. This fixed a lot of errors I was getting for professions not allowed.

Anyway, like I mentioned to ya, a lot of my code needs overhauled. I create stuff in a hurry just to test if I liked it or not or it would even work then if I liked it I left it as is for the moment... things like that.

Thanks for all the info you are sharing, I'm loving it.

Yes very interesting discussion - even for someone with the bare rudiments of Perl like me
If Nightinggale is using a cached array that stores yield requirements of profession X for player Y, I wonder if it could be possible to extend that for your tech requirements as well. (eg a cached value of -1 means that player can't currently access that profession.) Then you could just update the cache when a tech is gained/lost, and again get a big savings.

 

[raystuttgart]

Until now, most discussions about performance were by private messages between a few modders interested and "skilled" enough to understand what it was all about.
If there is really a lot of interest for a public discussion about improving performance, than maybe a separate thread could be opened for it.

Isabelxxx for example was also very interested in that topic.
(I am not sure however, if she is currently still modding Civ4Col.)

Generally however, there are a few major topics involved:

1. Logging
2. Caching / File Access
3. Graphics
4. Game Logic
5. Structuring of XML

Each of those we tried to improve in RaR.

We had spent a complete release on the topic performance.
We simply had to do it due to the incredible size of the mod (Mapsize, Number of Players, Number of Graphics / UnitArtStyles, ...)

But of course, a lot more can be done.

Basically it comes down to a very simple summary:

If you keep your mod tidy and well structured, you will have good performance.
You need to take care of everything. DLL, Python, XML and graphics.
DLL alone will not solve all your performance issues if your graphics are messed up.

 

[Nightinggale]

I completed my first major project, the Privateer mod, not even knowing anything about Debug. It was nuts, I know.

I have done stuff like that too. Modifying your limits to fit the task rather than reduce the task to your limits can be a good thing. However it's a fine balance between making something great and making something slow which is impossible to test for bugs.

One thing in my Medieval mod that I had to add that gets called a lot is a for loop of the Civics(I used Civics for Techs). It has to loop through nearly every civic to test if say, a profession is allowed or disallowed. Then checks to see if the Player has learned that tech. I started thinking on ways I could do this without having to cycle through the Civics each time. Creating arrays for player allowed Buildings, Professions, Units, Promotions, Yields, etc was one idea. I made sure though to that if a Tech disallows a Profession that it allows a replacement or upgrade. Then the code just cycles through the players units and forces a profession change. This fixed a lot of errors I was getting for professions not allowed.

While I agree that it sounds like it could be really slow, I would like to profile before judging it. Sometimes the slowdown is in the place you least expect. The slow looking code might be fast enough while the fast code is called 7000 times instead of 5 because of a typo in a loop condition.

1. Logging
2. Caching / File Access
3. Graphics
4. Game Logic
5. Structuring of XML

I tried looking at the logic, or rather recode how certain stuff is calculated. It turns out that RaR is actually decent in that regard and "improvements" turned out to make no difference, or in one case made it slower. In fact I considered all 5 of those and found 2 to be the only place where I could make noteworthy performance improvements.

If you keep your mod tidy and well structured, you will have good performance.
You need to take care of everything. DLL, Python, XML and graphics.
DLL alone will not solve all your performance issues if your graphics are messed up.

Yeah. Slowdowns can happen in all kinds of code. Luckily the stack based profiling looks into this as well because it measures the time the DLL waits for external stuff, like python. Profiling will not help fix the problem, but it will tell where it went wrong.

 

[Nightinggale]

I cached the profession IDs for fishing and whaling boats to avoid reading the XML every time. This reduced the wait around 1.4-1.5 seconds (according to profiling). The savegame which originally took 40 seconds to get to next turn is now down to 30 seconds.

I'm currently getting this assert over and over and over:

Assert Failed

File: CvCity.cpp
Line: 4685
Expression: iValue >= 0 || eYield == YIELD_FOOD
Message:

I don't get it

Can I get a savegame where this happens? Preferably one where all I have to do is to hit end turn and then it happens every time.

I looked into CivilizationIV.ini and there is a line saying

AutoSaveInterval = 4

It's likely a good idea to set it to 1 to make sure you have a savegame just before the assert took place.

 

[Nightinggale]

I committed project files and makefile. No change to the DLL, but much easier for people who wants to compile themselves.

The makefile is the new one mentioned in my signature. I really should make a more interesting looking signature, but it will have to do for now.

 

[agnat86]

Hello Nightinggale. Great that you are working on this mod!

I'm playing R&R a lot, and I'm currently trying to learn how to make modifications myself. I've already done a lot of balancing changes in XML, and also a few in cpp.

I've done the following things:

- Changed the order and abilities of Founding Fathers. In R&R, lots of great FFs were added, but in my opinion some of them appear too late in the game, while others are underpowered. After playing a few games, I noticed that I never took some founding fathers (like Orellana), while other appeared at a point when they weren't very useful anymore (Villegagnon).

- Changed the traits and attitudes of many native civs, to make them more diverse.

- Rebalanced the negative influence of population growth on native attitude.

- Allowed units with mountaineer to enter peaks.

- Many minor balancing changes to units, resources, improvements.

I still have a long list of things I still want to change, like adding events, further diversifying native civs and rival colonies and improving the AI of rival colonies, but these generally take up more time, and first I would like to get a clearer idea of who else is working on R&R, and what their plans are.

Please let me know if you are interested in these things for your Modmod, then I'll post my changes here.

 

[Kailric]

Please let me know if you are interested in these things for your Modmod, then I'll post my changes here.

I was told Nightinggale is out of pocket at the moment (you better not be reading this while I'm typing!) but from the sounds of it you guys need to talk. It can be tough trying to do things as a team though because everyone has their own ideas on what they do and do not want. So I would suggest to anyone out there trying to work together to concentrate on what you do agree on and then make Modmods for the things you don't

 

[Nightinggale]

Hello Nightinggale. Great that you are working on this mod!

Sadly I'm a bit stalled right now. However I haven't stopped and will commit stuff again soon.

Please let me know if you are interested in these things for your Modmod, then I'll post my changes here.

All of what you wrote sounds interesting. I like the sound of the result and it's something I haven't even considered modifying meaning I wouldn't have done it myself.

I was told Nightinggale is out of pocket at the moment (you better not be reading this while I'm typing!)

I'm reading it now

It can be tough trying to do things as a team though because everyone has their own ideas on what they do and do not want. So I would suggest to anyone out there trying to work together to concentrate on what you do agree on and then make Modmods for the things you don't

I will deal with one problem at the time. First problem is getting potential team members with skills and ideas. That appears to be a far more serious problem than disagreements. However at least a bit of agreement will always be needed and it looks to me like we fully agree so far.

 

[agnat86]

All of what you wrote sounds interesting. I like the sound of the result and it's something I haven't even considered modifying meaning I wouldn't have done it myself.

Great! I'm planning to add my changes to the newest version of R&R (probably including Schmiddie's changes). Please let me know it when you want some of them, then I'll share them.

I'm mainly occupied with changing the natives right now, and after that, I think I'll look if I can make other Europeans more interesting. I've already added new traits and abilities for natives. The next thing I want is to change some native civs into others. I think that currently, the American midwest is overrepresented with 7 civs. I'm planning to remove the Comanche, Navajo and Crow, and add the Haida (Alaska/Western Canada), Ohlone (California) and Timucua (Florida). Of course, this is all rather ambitious for a novice modder, so I don't know if I'll ever get there...