Yield issue

[Thalassicus]

I'm aware of some strange issues with yields in the latest beta release. Stored happiness, science, culture, and other yields will occasionally behave in strange ways. For example, I was playing along a game today, and my stored science suddenly jumped to a large negative number like -200,000 without a discernible cause. Others have reported issues with stored happiness (golden age progress) and culture (2mil turns to next policy).

I've been attempting to solve this beta bug for several weeks now. I thought I'd fixed it last Saturday, but apparently not. The pattern behind the bug has been very difficult to figure out from playing games. All the code I've looked at also appears correct. This is a very challenging problem to track down, but I'm working on it.

Please report here if you encounter the bug in the beta test - especially if you can figure out any pattern to it. Thank you.

 

[grrnonameleft]

For me, the bug manifested in 0 stored happiness for a while, and then a sudden GA out of nowhere (far too close to the last one for just the display to be broken). Unfortunately, I didn't have logging enabled back then, and I can not reproduce the unexpected GA.

However, here are things I noticed:
In the economic overview, resources & happiness tab, the happiness from difficulty level is right now about 800-900 and increases every turn by 20+. This does not change the total sum displayed in the summary.

Sometimes during turn processing, the happiness mouseover popup shows a large negative value that is approximately the true value minus the bugged difficulty happiness.

The bugged value and stuck stored happiness persist with saving/loading.

Edit: The earliest savegame with the bugged difficulty happiness is from turn 79, with a value of 50, and a stored happiness of 183. Next turn, those become 57 and 142, while I have a net happiness of 1. The plot thickens!

 

[grrnonameleft]

I played a savegame from turn 70 where the difficulty happiness is still a normal 8. Sadly, nothing usual caught my attention.
Things I did in these 10 turns:
Cleared a barb camp
Connected a new luxury
Built Stonehenge and The Great Library
Entered the classical era
Made friends with La Venta
Nothing obviously exciting.

 

[grrnonameleft]

Alright, I did a few test runs of ~5 turns starting from turn 79 and made note of the progression.

The difficulty happiness (DH) in the economic overview starts at 43, on every load.

If I enter the city screen of my capital, the DH will increase by 7 (repeatable).
If I change the production of my capital without opening the city screen, the DH will drop by one but only for the first time the overview is opened. Subsequently opening the overview will show the value without the drop.

Other cities don't seem to influence the DH.

If I advance a turn, the DH will increase by a multiple of 7, so far I have seen 7,14,21.

Sometimes, the stored happiness(SH) gets reduced (after ending a turn). It will unexpectedly decrease by an amount of exactly the DH of the previous turn minus 8 ( the true/supposed value, I think), plus the expected change due to the net happiness of this turn. (tested over 20+ reductions)

I can not see an obvious difference between normal turns and turns with reduced SH. When loading a game, as long as I restrict myself to the same few actions (fortify/sleep everything, and pick the same building for production queries), the reduction will occur in the same turn, so its probably dependent on the random seed in some way.

The between turn increases seems to be a multiple of the increase from viewing the city screen. A save game from turn 100 had a city screen increase of 2, which became 3 after a few turns. A save game from turn 120 had a capital city screen increase of 4, but this time, some non capital city screens also increased the DH by 4, 2, and 0.
No apparent system in all of this.

Enough poking around for now, I hope this proves useful.

Just some random rambling before I head to bed:
The nature of this bug and the odd numbers showing up suggest some sort of memory violation, possibly a released pointer that is still referenced somewhere, or some list or array that writes past its bounds. But I have no idea how that would look like in Lua or if it is even possible. The memory violation probably originates from the core game, perhaps some list of elements defined in the mod got too long to handle properly, but without triggering a warning? Anyway, idle speculation here.

 

[grrnonameleft]

Oh, and before I forget it:

The "Runtime Error" line shows up occasionally in my lua.log, though usually without the stack traceback.

[38874.137] Runtime Error: [string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:866: bad argument #1 to 'ConvertTextKey' (string expected, got nil)
stack traceback:
[C]: in function 'ConvertTextKey'
[string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:866: in function 'GetYieldTooltip'
[string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:1828: in function 'DoUpdateUpperLeftTooltips'
[string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:1510: in function 'OnCityViewUpdate'
[string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:302: in function <[string "C:\Users\Graniten\Documents\My Games\Sid Me..."]:300>

 

[Thalassicus]

I was thinking the same thing about an error with either memory out of bounds, or a cache error. Your discovery of a connection to the "difficulty happiness" might lead to something. I'm investigating that now.

Edit:

The difficulty happiness line is based on a combination of other functions, primarily pPlayer:GetHappiness(), a game-core function updated by the core DoUpdateHappiness. What baffles me is these are core functions which should always return vanilla values, so the mod shouldn't affect them in any way. Somehow the vanilla game thinks happiness is going up or down every turn. It seems like it's going up at the start of the game, then down later... almost as if some unknown factor is creating a happiness rate that changes constantly. It's really puzzling.

 

[Thalassicus]

I might have found the issue. The function city:SetNumRealBuilding does not have safeguards to prevent a negative building count. Cities with a negative number of a building type appear to corrupt game data, possibly because a value might be stored as an unsigned integer somewhere. The negative count does not appear in the city window, so there's no way to tell it happened. I believe the base game never uses stackable buildings, which would explain why Firaxis did not discover this problem.

I'm replacing the default function with a new one which prevents building count from going negative. This appeared to have halted the strange happiness behavior I experienced, which might have also been affecting the other yields. I posted an update including this change.

 

[grrnonameleft]

Nice work, Thalassicus!

Now that I think about it, shortly before the bug turned up, I may have lost friendship with a mercantile city state, possibly in the same round as reaching a new era. If so, the National Happiness building in the capital was still at the old bonus, but the new era bonus got deducted.
Edit: I was wrong, but I guess something like that could have happened to an AI

 

[lukeloh]

Hi Thal

The GA increment bug is mostly fixed. The only issue remaining is that it does not properly add the surplus during a GA. Otherwise, things look good.

 

[Thalassicus]

Thank you for bringing that up lukeloh. I disabled that feature when testing to find the problem. I will now re-enable the feature for beta 3.

 

[AnthonyG]

Ah, I had just noted that also in the Golden Age thread.

Another thing, then, that might have also been altered for testing purposes and is a yield of sorts is produce wealth and science are both available from the very beginning (beta 2).

 

[Tomice]

I think that's intended, Anthony.

 

[mystikx21]

It's in the patch notes actually, along with the shrine change.

 

[AnthonyG]

Oh, it is? My bad. Thanks. I guess I didn't really understand what that bit meant.

My City View UI is still messed up though.