CivEffects

[Nightinggale]

This is mainly a reminder for myself telling me what I should do. Sometimes I think of ideas I have to postpone implementing and then later I forget about them

Obviously people can reply here and ideas can be debated and evolve.

Activation status: 83/88
I rewrote the XML reading code and the cache. Activating the cache mean to inject the cache into the relevant places in the code in a way that the code acts as we would expect it to work.
Done. However a number of tags were skipped as they have never been implemented in the DLL.

1. Add all trait tags to CivicInfo
2. Cache all new (trait) tags in CvPlayer The only tags, which aren't cached will likely have to be redesigned
3. Apply cache to actually work
4. Move CivicInfo to new directory
5. Split CivicInfo into multiple files
6. Generate an effect civic, which sets +1 to all allows, which aren't enabled in XML
   (this can simplify CvPlayer cache generation)
7. Make reduced JIT, which stores only none-zero values ticket link
8. Make all getBool functions return an int of value 0 or 1.
   This will allow adding all "bools" of owned CivEffectInfos and storing the count. This mean if the player loses one of those, the count would go from 2 to 1 and nothing happens. No looping to figure out if any other CivEffectInfo has the bool set.
9. Change CvPlayer cache to store allow score rather than bool.
   Allows cache updating based on the changed civic instead of looping all like now
10. Reduce/remove non-CivicInfo from the cache
    Stuff like making a native trait, which enables the yields, which are now enabled by being native
11. Cache every single effect in CvPlayer (gifts excluded). There should be no reason to loop CivEffectInfos except for cache generation or display.
    That will allow a setup where adding a massive amount of CivEffectInfos will have little or no performance penalty on the game.
12. Rename tags to follow naming convention and write wiki text regarding the naming convention.
13. Make a vector of CivEffects, which are applied to players when setting up the cache
    This vector should read an XML file with one CivEffect and then it should add a CivEffect where bIsInit is set in the constructor
    This will allow XML setup of default player behaviour as well as copying the current setup where everything is allowed if nothing enables it
14. Fix discover sea trade route by entering access plot
15. Add an InfoArray in CvPlayer telling which trait he has. If it is empty, the get function will return the InfoArray from leader XML (which will have to be converted to InfoArray)
    The only place to actually set traits in CvPlayer would be in scenarios. This mean scenarios can set custom traits, but if no traits are set, it will rely on XML
16. Clear researched techs when start reading a scenario. This will remove the normal starting techs and only provide those, which are in the scenario file
17. Implement research cost modifiers in handicap and gamespeed
18. Implement yield bonus for connected missions and trading posts
19. Add the ability to completely clear promotion effect cache in units and recalculate (useful when new promotions are allowed)
20. Add techs for nation states in CvPlayer::init() and generally clean up assigning techs in that function

1-4 is important before next release. The rest is DLL internals, which will improve performance and can wait.
I have come this far. I might as well finish it all before next release.

Possibly add bools to a trait saying it's free for all players who aren't native and one for non-natives. That will allow a quick setup for difference between native/non-native without having to edit every single leader. Also it removes the risk of forgetting to add it to a specific leader. This would be good to have in next release.

List of files:

1. Civics
2. Inventions
3. TradePerks
4. Censures
5. Traits
6. Effects

Move support files for those files into the new dir as well, such as CivicOptionInfos and TechCategoryInfos.

Figure out if DllExport can be removed from CvCivicInfo. If not, then renaming could be an issue. Solution:

Spoiler :

Looks like the exe is aware of CvCivicInfo and can get the vector from the DLL. However it never calls any function beginning with CvCivicInfo::
The next question is if the call in the exe can be reached. It makes little sense to get the vector if it can't access any of the functions related to it. I would say it's likely leftover code from something, which didn't make it into the final exe, such as possibly debug code.

Rename CvCivicInfo to CivEffectInfo (or whatever)

Last in the header file, add

class CvCivicInfo : public CivEffectInfo
{};

That way we can store CvCivicInfos in the vector and give those to the exe and it will be happy. In the DLL, we can use CivEffectInfo. This will work because we can always "revert" a class to the parent class and only use parent functions and variables. In this extreme case the child class will add nothing other than the name the exe expects.

Maybe we should have a class for each type and add type specific functions to each, meaning we will have CvTechInfo::getInventionCategory().

Rename CivicInfo to PlayerEffects or CivEffects and assign CivicInfo to those civics, which are actually civics.
All XML files provide data to info classes. I think it would be best with CivEffectInfo.

Make CvGlobal functions to access each type of CivEffectInfo.

for (int i = 0; i < GC.getNumTechInfos(); i++)
{
    CivEffectInfo kTech = GC.getTechInfo(i);

GC will then take care of the offset issue and deliver the correct one from the shared vector.

effect civic
On save:
disable civic, save, enable civic. This can be done by just changing the bool array, which would skip cache updating.
Reason: BoolArray (like JIT array) only saves from index 0 to the highest non-default. By clearing the rear, less data is saved and filesize is reduced.

On game start/load:
postLoadGameFixes() should call a cache set function for each player. It should do for alive players:

1. Enable civic effect (skip cache update for performance reasons)
2. Clear CivEffectInfo cache
3. Loop all CivEffectInfos and assign cache
4. Era restrictions (post 100)

This will make the cache valid for all games, be it new game, scenario or loaded.
This function should be written in a way, which allows it to be called at any time without breaking anything. It will most likely only need to be called from postLoadGameFixes(), but writing it to just work in other cases is good, should the need show up.

Python
It will be a whole lot of work to add all the CivEffect get functions to all the affected classes, especially if the number of classes keep increasing. It will also make it harder to use shared code.

Solution 1: figure out python class inheritance. This might be tricky as the class is returned from the DLL and as such is not pure python.

Solution 2: generate a CivEffect class, where IDs represent a fake single vector and expose this class to python.
This is like Tech ID 4 is CivEffect numCivics + 4. The get function in CvGlobals will be slower than the get functions for each type. However it is only used for python, high performance isn't an issue.

This needs further investigation, but I think solution 2 will be the one with the cleanest implementation in python. It will be like making individual pedia pages and then to fill out the effect, it simply calls a function to do that and this function takes getCivEffectIndex() as argument.

The DLL will not really be affected either way as class inheritance is pure C++ and works well. Looping all CivEffects is not really needed due to the cache, though it might be useful during init. In fact init will need to ask "is any CivEffect allowing this" (for everything in allows group), which really is the only time where looping all would make sense.

Solution 3: Don't expose CivEffects to python
This one is the most simple one to code. If we need to print some text, we can make a getText function, which python use to print everything. If help popup use the same getText function, then we will only have to write the text stuff once.

It isn't unlikely that we will have to make multiple functions to get text and each screen then merge the replies as needed or print them in different boxes or whatever we come up with.

Sea trade route discovery
This is disabled for now. The trigger mechanism is in CvUnit::setXY(), but it is hardcoded and performance isn't great for a function called this often.

Possible solution: Add a CivEffect to EuropeInfo in XML. In setXY(), if the new plot has access to a trade screen, which the owner doesn't know about, then the owner is given the CivEffect from XML. The movie name can be set in Europe XML and be triggered by enabling the trade screen with any CivEffect. That way the DLL supports movies for any trade screen, regardless of how it is invented.

Maybe add an InfoArray to CvPlot and store which trade screens it can access. This saves some looping in setXY. If it is added in addition to the current access code (instead of replacing), it can be a bool array of screens, which can be discovered by moving into them. That way setXY only has to check the (often empty) InfoArray and owner's CvPlayer::canUseTradeRoute().

AI
We might have to make a CivEffect in XML, which tells special rules for the AI. If trade routes are enabled and disabled here, which could be used to set vanilla Europe for the AI. The DLL code will not take AI into account and the vanilla Europe will then not have a special meaning in the code.

 

[Kailric]

I'd vote CivEffects, the Civs may appreciate that more, Yo Player, get with it!! I got your "Player" boy, I'm a Civ, bring it!!

There is also two new player, uhh hurmm Civ, types: Minor Civs and Nation States.

 

[Nightinggale]

I'd vote CivEffects, the Civs may appreciate that more, Yo Player, get with it!! I got your "Player" boy, I'm a Civ, bring it!!

All XML files provide data to info classes. I think it would be best with CivEffectInfo. Not naming it info might cause confusion if it is an XML data class or not.

 

[Nightinggale]

I finished grouping all the tags, both civic and trait and have merged them into one big CvCivicInfo. The tags are added to XML, are read by the DLL and all have accessor functions. However nothing calls those functions yet meaning they still have no ingame effect, not even for the display.

Updated wiki guide on the tags:
https://sourceforge.net/p/colonizationmodcollection/wiki/Civic_XML/
Descriptions starting with NA are those, which aren't read yet. That will help finding the "missing" ones when adding them to CvPlayer.

 

[orlanth]

I'd vote CivEffects, the Civs may appreciate that more, Yo Player, get with it!! I got your "Player" boy, I'm a Civ, bring it!!

I propose splitting this into PlayerEffects.xml and HaterEffects.xml

There is also two new player, uhh hurmm Civ, types: Minor Civs and Nation States.

What's the difference between these atm?
I wonder what I should plan to do for the aliens in 2071. Ideally I'd want them to behave pretty much like normal "colonial" player civs, except that most of the "natives" effects can still apply to them (for example the native border system, not popping Goody Huts, gift giving, the anti-native combat bonus, etc).

Maybe if each of the vanilla features of "natives" (borderless, no goodyhuts, etc, etc) were each controllable independently by a civicinfos tag, this would be total moddability to the max, playa! You could even let primitive tribes start out with all native-like characteristics, and then gradually discover new Civics/Techs to advance to become more full features civs.

 

[Lib.Spi't]

I THINK the current difference is:

Nation State: Popey Rome, starts advanced and with multiple developed towns. Being planned for use with other major powers like Byzantium.

Minor Nation: These are 'natives' that have many of the same abilities as normal civs, they have either developed to this point through techs, etc. or started like that. (I don't know how far Kail has implemented it at this point)

 

[Kailric]

Minor Nation: These are 'natives' that have many of the same abilities as normal civs, they have either developed to this point through techs, etc. or started like that. (I don't know how far Kail has implemented it at this point)

Kai can't remember all that was done either. Minor Civs still have the isNative Trait, but in some instances this is over written to make them more Civ like. I would have to go back and look at the code for specifics. I should actually document the differences. (Perhaps I did in the Minor Civs thread, duh)

Maybe if each of the vanilla features of "natives" (borderless, no goodyhuts, etc, etc) were each controllable independently by a civicinfos tag, this would be total moddability to the max, playa! You could even let primitive tribes start out with all native-like characteristics, and then gradually discover new Civics/Techs to advance to become more full features civs.

This is a good idea. Natives and Minor Civs both don't respect territory but perhaps this could change as you say at some point, perhaps with treaties. We can introduce a new difficulty where you can play as a Minor Civ or a lesser Native type. Then you really work to raise your status. The AI can be doing this as well and you may at some point hear a trumpet blast declaring, "Arpad has become a new Rival!", then his borders appear on the map and THEN he starts beating you with his pimp cane.

 

[Nightinggale]

I should actually document the differences. (Perhaps I did in the Minor Civs thread, duh)

Write it to the wiki as it will be easier to locate later. Also we can all modify it to improve on it whenever we (re)discover or create new features.

I'm all for making the native stuff individually modable. I would say look at the first post for a valid place to add stuff like that

It would actually make sense. Something like a native trait, which enables a bunch of native features and then inventions and stuff to disable them one by one. Modders will also be able make "hybrid" civs by just adding a few native features, but not all of them.

I'm not sure what to do about treaties from a coding perspective though, but adding a more complex relationship between civs could be interesting.

 

[orlanth]

It would actually make sense. Something like a native trait, which enables a bunch of native features and then inventions and stuff to disable them one by one. Modders will also be able make "hybrid" civs by just adding a few native features, but not all of them.

Yeah, suppose there were Techs or Civics like Coinage, Feudal Code, Nationalism, Expansionism etc and having these can individually unlock the "non-native" features of full civs (stop spontaneous Giftgiving, enforce formal borders, become able to explore and loot Goodyhuts, become able to attract Emigrants, will no longer become Vassals, etc).

The advanced or "Colonial" civs could start out with these, while the native or less advanced civs start out with Techs that give some yield bonus to production from local terrain. You would have a choice to trade with them for mutual benefit and grant them knowledge enabling them to advance, but have to consider the potential drawbacks that they are gradually becoming more mature civs and more able to present a challenge to you with each new feature that they unlock!

 

[Nightinggale]

I rebased civic_rewrite branch onto develop, solved the single conflict and ended up with a branch, which branches off from the head revision of develop. SF rejects when I want to push that because it rewrites already pushed revisions. Instead I made a new branch called CivEffects.

Hopefully nobody will add more conflicting code before I'm done with this feature

 

[Nightinggale]

I started building a player cache and it is working beautifully. I did run into a bit of a problem where existing code expects CivicInfo to contain arrays and the python exposed part might be a problem in the long run, but for now I fixed it by adding array like get in InfoArray.

Fun fact:
I just realized that the new cache generating code design is ideal for multithreading and through that multicore CPU support. All caches are calculated independently from each other meaning order doesn't matter. As a result, we can put all tasks into a threadpool and then wait until it says that it finished all of them.

However I designed this to be as fast as possible meaning it's unlikely that we can tell the difference in performance. Adding threading means adding the boost thread dll and it has to be next to the exe, which is quite bad.

 

[Nightinggale]

I just wrote this regarding goody huts and probability of getting each type. It's for the wiki, but I haven't uploaded it.

Notes: GoodyFactors XML setup behaves differently than vanilla!
Here is what happens whenever a unit enters a goody hut:
Imagine a bag with balls. There is one ball for each goody type. A random one is picked. Since there is one for each type, all events have an equal chance.
GoodyFactors adds or removes balls for selected goody types, which mean they increase or decrease in chance. In fact it is possible to have 0 balls for a type, which mean it can't be found.
Since default is 1 for each type, providing -1 will disable that type.
The XML setup have been changed to allow the effect of multiple CivEffects to stack.

However now that I think about it, I don't like the resolution of the probability. If there is just one ball for each, all we can do is either keep that or remove it and disable that type.

I'm thinking of using a higher number, such as 10 for each. That will allow CivEffects to add/remove chance of finding something at 10% steps rather than the off/normal/double steps. Maybe the default should be 4 or there will be too many to get a good random pick. Not sure.

What do you say? Is this worth looking into?

Also regarding prospecting levels. I'm considering rewriting the XML into being a simple bool rather than int. It makes little or no effect on the coding, but it can simplify the XML layout. The question is if we will ever need a prospecting score of more than 1 in a CivEffect. It is also a question if it makes sense to have negative scores, such as a civ starting with a CivEffect providing -1 and that civ will then dig one level higher than the other players...

I'm not sure what would be best.

 

[Kailric]

Actually, I'm not sure what you are asking here. It sounds like GoodyHuts have a new ability to grant CivEffects? Or perhaps, CivEffects can alter what gets found? What does vanilla do now, I haven't used this kid of effects for Goodyhuts that I know of?

For prospecting Levels, it can be a boolean, and if we need an int at some point we can convert it no problems.

 

[Lib.Spi't]

I think one civeffect might be the map reveal one, which is an FF power.

As to the specific question of is it worth looking into, my answer would probably be no. Goodyhuts are really just an early booster for a civ that prioritises exploration. After a point the rewards diminish in importance, unless you keep getting the absolute best results, which is rare.

I think there are much bigger, broader game mechanics that would give much greater gameplay effect than fiddling with goody huts would.

A more diverse goody system, might be good for a WHM mod, where you are discovering new untapped lands long into the game, but even then, I think it will still be minimal unless the whole thing receives some kind of mega effect expansion. Again though, much better mechanics would improve the game far more.

 

[Nightinggale]

Actually, I'm not sure what you are asking here. It sounds like GoodyHuts have a new ability to grant CivEffects? Or perhaps, CivEffects can alter what gets found? What does vanilla do now, I haven't used this kid of effects for Goodyhuts that I know of?

It's actually a trait feature I merged into CivEffects. It can be used to modify how likely it is to find each GoodyEffect, be it double chance or entirely disabled. The question is more about if the steps should be smaller than +-100%.

The trait TRAIT_FILIPPO_BRUNE gives double chance of getting GOODY_TREASURE. However no leaderhead is using that trait. Maybe I should make a script or DLL feature to check if traits/leaderheads/whatever is unreachable ingame. That will have to wait.

 

[Kailric]

It's actually a trait feature I merged into CivEffects. It can be used to modify how likely it is to find each GoodyEffect, be it double chance or entirely disabled. The question is more about if the steps should be smaller than +-100%.

The trait TRAIT_FILIPPO_BRUNE gives double chance of getting GOODY_TREASURE. However no leaderhead is using that trait. Maybe I should make a script or DLL feature to check if traits/leaderheads/whatever is unreachable ingame. That will have to wait.

I agree with Lib. I didn't even know what that trait was, but yeah I know the one that doubles the chance to find Treasure. Your proposed changes sound like they could be useful though but we can always go back and change it later.

 

[Nightinggale]

Your proposed changes sound like they could be useful though but we can always go back and change it later.

I think it's fairly easy to set up at any time. The init value (currently 1) is set in CvPlayer's constructor. This mean changing it later will just be changing that single number and then the values in XML (quickly done since it's just one trait).

I'm nearly done generating a cache of CivEffects in CvPlayer. I have run into an obstacle though

There is an array with buildingclass, yield and modifier (3 variables). I planned for this:

<aaa>
    <buildingclass>something<buildingclass>
     <yield>YIELD_FOOD</yield>
     <ivalue>5<iValue>
</aaa>

But the schema shows it is written like this:

<aaa>
    <buildingclass>something<buildingclass>
     <yieldModifier>
               <yield>YIELD_LUXURY_FOOD</yield>
               <ivalue>5<iValue>
     </yieldModifier>
      <yieldModifier>
               <yield>YIELD_FOOD</yield>
               <ivalue>5<iValue>
     </yieldModifier>
</aaa>

Both are arrays of aaa objects.

Both for DLL memory management+performance and for XML design reasons I like the latter design, but currently InfoArray can't handle it
I'm thinking of making some sort of recursive reader. I just need to figure out the details, but it mean so close to the end and yet... not really

I also need it to interact with the 2D JIT array, but I knew I had to do that. I wouldn't consider that an issue.

Also once I have the cache I also need to apply the reading functions. A cache, which is never read will not work even if the cache is correct.

 

[Nightinggale]

I just realized something. The approach of joining multiple XML files into a single array is fundamentally flawed. Sure it can be done, but it will likely cause an index offset hell. Also it will make it harder to make something, which only applies to some of the files, such as tech screen coordinates.

The real solution is actually a textbook C++ solution. We stick to one file-one vector and each vector has it's own info class (CivicInfo, TechInfo, TraitInfo...).

The current CvCivicInfo is renamed to CivEffectInfo. Next we make one info class for each file and each of those classes should have CivEffectInfo as parent. This will allow us to have one place to write file specific code while we also have CivEffectInfo, which is code, which applies to all the files.

Most of the data is only used for applying to cache in CvPlayer. Currently it takes a CivicTypes as argument and looks up the info class. The argument should be changed to a CivEffectInfo pointer, which mean the caller should look it up, not the function itself.

The trick here (which is the C++ schoolbook case) is that the CivEffectInfo pointer can be a pointer to a CivEffectInfo instance or an instance to any class, which has CivEffectInfo as parent. In other words, if we gain a civic, the change civic code can call the code with a civic info pointer. Gaining a tech will call the same function with a tech info pointer and so on. Multiple files, multiple classes, one function to handle the effect of all of them.

I think it would make sense to move the gift code to a class of it's own. Some of the classes should never use it. Imagine if a civic provides a free unit every time you switch to it. Some player could switch between civics multiple times to get that unit over and over. CivEffectInfo should contain the permanent effect data, such as yield bonus, unit movement bonus and so on.

The only reason why it would make a lot of sense to have all the CivEffects in a single vector is when you have to loop all of them. However now that my player cache is almost complete, I have yet to encounter any scenario where the new cache design needs to loop. This mean the only time where everything is looped is during game start/load where the cache is set for the first time. I think we can handle code to loop each array in that single location

 

[Kailric]

All I can say is do what you got to do I'm cheering for you

 

[Nightinggale]

I ran into a weird problem with CvXMLLoadUtility::GetXmlVal()

I was getting really weird numbers in pedia and through some debugging I ended learning a thing or two about GetXmlVal(). It provides the correct value when reading a string, which is why it managed to read the correct yields and stuff. However reading an int results in... well semi-random numbers.

I never found the cause of it (seems to be inside the exe), but I managed to work around the problem. Here is what the codes does now:

1. Reads a wide char string with GetXmlVal()
2. converts from wide char to an ASCII string
3. converts ASCII string to int

That works, but reading the int directly doesn't

All I can say is do what you got to do I'm cheering for you

So you have no idea what I'm talking about, but you just smile and hope that some awesome code will appear without your involvement. Here is a tip for you: whenever I do that, the magical great code never arrive