[We the People] Development "diary"

[raystuttgart]

Hi guys, "Happiness vs. Unhappiness" was improved and expanded a bit.
(I felt, it needed a bit more for high Happiness / high Unhappiness)

Improved:

• City Screen (because the display of Happiness and Unhappiness took away too much space)
• Messages and Balancing
• Herman of Alaska now gives Bonus on Happiness (Pedia adjusted as well)

New: "Festivities and Unrests"

• High Happiness (over some time) can trigger Festivities (which generate a small amount of Father Points from a randomly selected Father Point Category)
• High Unhappiness (over some time) can trigger Unrests (which will stop all production in the city for a short time)
• Festivities and Unrests have an XML setting that prevents them to be triggered too often - all other aspects of Festivities and Unrests are configurable in XML as well.

 

[raystuttgart]

Schmiddie has created new Gamefont Icons for Happiness and Unhappiness.
(The Smilieys really looked a bit too modern.)

Spoiler :

 

[raystuttgart]

Hi guys,

Happiness was enhanced a bit.
To be specific the factor "Unhappiness from Slaves" can now be modified by Traits.

• Spanisch Civilization Trait "Conquistador" now gives -25% Unhappiness from Slaves (Unhappiness got a new Icon, the "Raised Fist", see above)
• Founding Father Gregorio de Mattos e Guerra now also gives -25% Unhappiness fom Slaves (he was a little bit dull before because he only gave Units

Other things like fixing some small issues and improving balancing happen as well, but it would take too long to report all these minor things in detail.

 

[raystuttgart]

Hi guys,

I still consider our original plans (of accepted but not yet implemented features) to be valid:
(So generally I have not forgotten about them and still plan to work on them step by step.)

Still Todo:

It would however be nice if we could also finish and merge these improvement branches:

• scrollable Score List
• BTS style city Icons and effects
  
• K-mod pathfinder improvements
• New Savegame Format

--------------------------------------

So probably for 2.9, 2.10, and following:

• Upkeep for Military Units
• Water Improvement Fishing Nets
  
• Improving Feature Health
• Changing Cannons From Units to Professions --> Will need help with AI and graphics (Buttons in Atlasses)
  
• Captured City Art
• Pirate Nation and Pirate Raids

• Nation specific Unique Units and Unique Buildings
• Religions of the New World
• Large Rivers
• Techs?? (no game concept yet)
• Civics?? (no game concept yet)
• Rebuild of Military System (no game concept yet)

Of course it is possible that we change our minds and do not want to implement one of these anymore.
It is also very likely that we will have new ideas that we also want to implement.

For example, I would also like to implement these:

• Brave Lieutenants (and maybe Capable Captains)
• Royal Interventions

-------------

What we definitely need to figure out is:
What are we going to do and publish in Release 2.9?
(The list of already existing concepts and new ideas will definitely be too big for one single release.)

Currently my "most wanted list" and thus priorities for Release 2.9 are these features:
(But I would leave implementation to other team members if they are interested.)

• Improving Feature Health
  
• Upkeep for Military Units
• Pirate Nation and Pirate Raids
• Royal Interventions

Additionally I would consider implementing smaller features like these to be likely that I will work on them:
(But again, I really would not mind to leave their implementation to other team members.)

• Captured City Art <- if nobody else volunteers
  
• Brave Lieutenants (and maybe Capable Captains)

-------------

Comment:

I explicity currently do not start one of the "Giant Concepts" myself, because they would keep me busy for the complete release.
(Features like Techs or Large Rivers. I consider them more of a topic for later releases.)

But if other team members want to work on them and take responsibility for the implementation it would be cool.
I will of course also support, but simply not focus on them full time for now.

-------------

Summary:
Of course every team member can bring in new ideas and is free to choose the topics he wants to work on himself.
But it would be great to know who is planning to work on which features so we can coordinate and do not waste effort by double work.

 

[raystuttgart]

So ok, we are making progress.
It is slow - because the team currently is tiny - but relatively steady.

Achievements so far:

New Features:

• New Founding Cities Mechanic
  
• Failed Missionary (from Ramstormp)

Improvements and Fixes:

• Fixed Visibility Bug on Ocean
• Fixed Shift-Key Bug for Loading in Port Royal
• Fixed Python Exception in Port Royal when lacking Funds
• Fixed small Bug in King Unit Requests
• Fixed small But in XP Cap (for Wild Animals and Runaways)
• AI improvements for Missionaries and Native Traders
• Balancing improvements for Happiness (Rebel Bonus on Happiness from Buildings removed, max. Unhappiness from Tax Rate capped to City Population)
• Several small XML balancing corrections
• Several small text corrections
• Small code refactoring for performance and stabilities
• Fixed many small Asserts (uncritical Warnings in Code)
• Improvements considering Python Events
• New Python Event "Sell Native Slave" (from @Acheron81)
• New Python Event "Sell African Slave" (from @Acheron81)
• several AI improvements (from @devolution)
• Fixed small bug of "Changeable City Center Plot Yield"
• Integrated "Happiness Advisor" (in Domestic Advisor) (from @Raubwuerger )
• Improved Native Advisor

 

[raystuttgart]

@devolution

I was playing a short test game with your newest internal AI improvements.
It was only for about 30 turns but AI was doing really well.

Spoiler :

So well yes, I have only checked early game so far.
But in early game AI did everything fine.

In fact considering some game aspects several AIs did even better than me.
(I played on Game Difficulty Conquistador - so average difficulty.)

In these first 30 turns most AIs e.g. had a bigger population, one more City already and slightly more Military than me.
They also used their Starting Units really nicely - meaning they did their job as supposed to.
(Some AIs start with Expert Missionary, Expert Trader or Expert Scouts.)

However, in the beginning I usually focus on Scout Rush, Bell Rush and getting my first Galleon - usually not on population.
So considering Treasures and Bells I was still better than AIs - but as I said, I focussed on that.

So my game strategy explains, why AI did better in some areas and I did better in others.
In total I would have considered the player situation pretty much balanced.

I also explicitly took over AIs by cheat mode and checked their production.
They were also producing goods, constructing buildings and transporting Units from Europe well.

So it seems that AIs are not losing any ground at all - by e.g. bad AI decisions.
(I definitely did not play bad or make any real mistakes either.)

Summary:
I see absolutely no problem to publish your AI improvements with the next official release.

 

[raystuttgart]

Short information about current work and progress on WTP:

Currently @devolution is still working on improvements of AI.
Several of his improvements have recently been commited internally.

So even though it is currently a bit quiet considering WTP there is still some work happening.
(The rest of the team - including myself - seems to be currently taking a break though.)

 

[Nightinggale]

I have been finishing up features, which have been in development since WTP started, but never made it to any release. I will give an incomplete overview of what I have been working on. It's mainly technical, but even internal technical changes are intended to benefit the players.

For player:

• The two DLL files have been merged. You now select 1/2 plot at a game startup question just like map size. No more moving DLL files or issues with loading savegames "from the other DLL".
  
• New savegame format. The savegames will no longer break between releases if say a new unit is added. It's not impossible to break savegames, but they can survive a whole lot more now.
• Savegames starts out bigger (data to correct for xml changes), but should grow more slowly in size. Prepared for (but not using) savegame compression as in making a zip file inside the save file.
  
• The game is faster (though it's not trivial to measure to provide numbers)

For modders/ technically interested:

• XML indexes (and lengths) can optionally be hardcoded into the DLL at compile time. This makes debugging easier (example: unit type 89 is now reported as UNIT_NATIVE(89)).
• JustInTimeArray is replaced by EnumMap. Still a list of length (whatever xml file you pick), but it's rewritten from scratch to make it easier for the compiler to optimize. Benefits a lot from knowing the length of xml files at compile time.
• Various other changes to either move checks from runtime to compile time or otherwise help the compiler to speed up execution.
• Caching of indirect xml data at startup, like list of yields demanded by at least one unit or yields required by at least one profession.
• Added the CivEffect concept. Too big to explain briefly here (it's mentioned elsewhere on the forum). It adds xml freedom and multiple files (like leader, trait, civic etc) can share tags, hence functionality.
• Anything touched by CivEffect is cached to speed up performance even more than prior to adding it. It would be a slowdown if implemented using the vanilla approach.

A lot of what is mentioned under technical will forever be work in progress because it can always be expanded. For instance I did not rewrite all "loop all buildings/units/whatever" loops and only those with updated stop conditions will benefit from knowing the length at compile time. This means some of the gains from this will be spread out across multiple future releases, but the important part is that the foundation has been added to make those changes in the future.

 

[Nightinggale]

I ran into an issue regarding runtime selection of colony radius. Due to engine limitations, I can't ask the player when loading a scenario. To do that, I would have to mod code inside the exe. To get around that, I added a config file for user settings and now users can pick what they want in a txt file. If it's not present, the DLL will create the file at startup:

User configurable settings


# Default max catchment radius for colonies
# Can be changed at new game startup, but engine limitations prevents a GUI for this setting in scenarios
# Note: scenarios can ignore this setting and specify radius
Default Colony Catchment Radius=2

That's the default layout for now. If a user gets creative and decide to write say 3, the settings file parser will detect an invalid value and use the default (2).

While it's not possible to change values from this file ingame, it wouldn't be hard to add. Also the file I/O class is written to be prepared for more settings and it's obvious how if you read the source file. Right now I have no idea what else it could be used for, but we now have the ability to store data between games. Perhaps something graphical like unlocking the GameFont debug screen without the need to compile a non-release DLL. Time will tell if or how we use this.

Scenario files now have the setting City Catchment Radius under map. It can be used to specify which radius the scenario will use. Setting it to 0 will make it use whatever the user put in UserSettings.txt and this is the recommended approach unless there are cities in the scenario file. If the line is missing, it's assumed to be 0. The other two valid options are naturally 1 and 2. Anything else will be assumed to be 0.

 

[raystuttgart]

Currently it is "crunch time" for all team members - meaning we are quite busy with real life and work.
But all of us try to continue modding at least a bit at the weekends.

Progress is slow, but it is steady.
So do not worry that it might be a bit silent during the week.

Also, we had our first "Team Meeting" in Zoom yesterday.
It was really fun and we had a lot of interesting discussion.

-------

Summary:

Do not worry, we are still here and we still have big plans for this mod.
Yes, we are a buch of crazy fanatics ...

 

[raystuttgart]

@team:
Guys, this was one of the most productive weekends we ever had !!!

• Huge progress considering Large Rivers (which was really awesome)
• Many bugfixes and improvements for "New Save Game Format"
• Lots of progress for "multicore branch"
  
• Several great Graphic Improvements
• new Goodies Fire Camp and Shipwreck
  
• Enhanced Goody System (Unit Only Goody Huts)
• A new Bonus Ressource (Cactus)
  
• Integrated lots of new text improvements into the mod (from Kendon)
• A small new Event
  
• ...

If we continue like this, it will hardly be able to keep track.
(Most likely I have already lost track and forgot something on the list ...)

 

[Diegozubi]

All this progress sounds amazing. I really like the idea of simplifying the use of the different plot-radius and the large rivers. It became a completely new game long ago with your work. Do you have an estimated time for a new release? Thanks a lot!

 

[raystuttgart]

Do you have an estimated time for a new release?

Maybe we can release in December shortly before Christmas.

 

[raystuttgart]

Maybe we can release in December shortly before Christmas.

Sorry guys, this has become unrealistic.

At the moment I do not believe we will have a release this year.
(I will not speak about dates anymore because everytime I do, I am completely wrong ...)

Everybody in the team is currently quite busy and we still have issues to solve in both major branches of our development.

-----------

1. "Large Rivers" (the feature branch) is already nicely playable, very stable and pretty bug free, but it is not yet fully finished.
(Maps are not all adjusted yet, MapScripts do not yet generate the "Large Rivers".)

2. "Big Merge" (new Savegame Format, AI improvement, pathfinding, OOS analysis ...) is still in alpha testing and bugfixing.
(It is playable as well, but we still get reports of issues from time to time.)

Merging both of them will require additional effort and testing.

-----------

The team agreed to release once both branches are fully finished, tested and integrated.
But we will need more time for that ...

-----------

Sorry to community.
It is simply "real life" again ...

 

[Nightinggale]

At the moment I do not believe we will have a release this year.

Can't rule out a new beta testing release, but it will likely again be without large rivers. Time will tell. My biggest concern is that I have been unable to recreate the OOS issues so far. They are really annoying when they happen and yes I have tried them in the past, but without the ability to recreate them, I don't know what to do to fix them.

 

[XSamatan]

The team agreed to release once both branches are fully finished, tested and integrated.

That is actually a very good decision - take your time, the next version looks already to be much fun!

Regards
XSamatan

 

[Nightinggale]

The public testing is going well. Some nasty bugs were reported, which we then managed to find and fix. While we didn't manage to get a stable release out before Christmas, I say we are still progressing just fine.

I find one bug concerning and that is the hot seat crash. It's the type of bug, which can be almost impossible to find, meaning rather than fixing it, the goal should be to prevent it from showing up in the first place. In other words we should prevent the modders from touching the "exe interface" in the DLL file. Sadly this is easier said than done because any modder with access to the code can change it and there is no way we can lock specific lines. As useful as Dependency Walker is, it could not be used to catch this specific bug because DW doesn't look at const.

I came up with a plan though. I'm adding a new type of DLL to the project, one called test, which will not result in a dll file. Instead it will run various code to test for specific bugs. Next up is writing a script, which fails if the interface differs from vanilla. This way we will be informed of bugs even before anybody locates them ingame. With such an automated test we can also test for various other issues.

Status right now is that I have set up the test and it's using the existing test for 1/2 colony radius as in it will catch if the setting is altered at the wrong time. Nothing about the exe-dll interface yet, but it will come.

Yeah I know it might not be a super exciting feature because it adds nothing to the gameplay, but even the best features are worthless if the game is plagued with crashing or other game breaking bugs. It's not on top of my wishlist either, but since I'm not certain I can figure out a bug of this type again, I better do everything I can towards making sure it will never show up at all.

 

[Nightinggale]

Done. The test turns out to take less than 0.01 seconds and it's now always performed whenever anybody is compiling. It will make the compilation turn into an error (no DLL file) if anybody touches the EXE<->DLL interface at all. It's really stupid and will trigger on any change to the lines in question even accepted changes like adding whitespace. However I prefer that to the risk of not detecting a crashing change.

I also removed all unused DllExport keywords. It allows the exe to call a dll function and it makes no sense to add this to any function the exe doesn't call. We had 3671 such words in the source. Now we have 1322. Since this keyword usually means "crash if modded", getting rid of them for functions, which can be modded will make the code easier to mod with more degrees of freedom. It also makes the code cleaner and easier to read. Since the script knows which DllExports are needed, it will cause an error if it detects an unexpected one. Future modders aren't allowed to add unneeded cases of DllExport.

And most importantly: we should never again encounter a case where we have the exe crashing because somebody touched a function declaration with DllExport in it. Since that will cause a crash in the exe rather than the DLL code, figuring out the cause of such a crash can be near impossible.

 

[Nightinggale]

One more update on the exe interface. I added a .def file to redirect the calls from the exe. getDefaultProfession() in CvUnitInfo now has EXE_getDefaultProfession, which is called from exe and python. This function is a wrapper calling getDefaultProfession. The vanilla getDefaultProfession is then DLL only and free to be modded without nasty sideeffects. If it compiles, it works. I used this to change the return type from int to ProfessionTypes. Did the same with getDefaultProfession in CvCivilizationInfo. This removed 111 typecasts, which is good because typecasting is generally bad as it can hide bugs from the compiler. Now if somebody goofs with type in this specific case, the compiler will cause an error while the vanilla approach with typecasting would compile just fine and just not work correctly ingame. Using correct types also makes the output in the debugger more human readable, which helps during bughunting.

In short: having worked with the exe<->dll interface for a few days have given us code, which is more resistant to bugs, have a higher degree of modding freedom and is generally easier to mod.

Thanks to @f1rpo for mentioning the ability to use .def files at some point ages ago. Looking at his AdvancedCiv mod helped because the documentation for this specific use case isn't great. It looks like all guides wants to help with all sorts of other abilities the file provides, which we don't need.

 

[Maximus123]

Hello, I can translate to Spanish, if needeed.

Also, the new ideass of great rivers, military upkeep are great.