The purpose of this article is to analyse the maths behind production in Civ4, the multipliers involved, overflow, and what's called "correction"; further to explain the original bug and the not so great work-around introduced by Firaxis; finally to outline the exploit introduced by the work-around, and how to minimize the impact of the wrong math on your civilization's economy. The answer is micromanagement of your cities, which we all thought to be gone for good; and I do hope that Firaxis will fix their formulae as soon as possible.

Many thanks to everyone who contributed to this thread (http://forums.civfanatics.com/showthread.php?t=140180)

Definitions:

BP:= Base production of a city (sum of hammers by worked tiles & specialists; also forest chops)
NP:= production needed to complete a project this turn (="costs-done")
P := total production
g := sum of generic bonus multipliers (e.g. 25% from forge)
s := sum of special bonus multipliers applicable to current project (e.g. 100% from stones for stonehenge)
TO:= total overflow (=P-NP)
C := overflow correction, hammers substracted from TO when an s-project finishes
O := actual overflow (=TO-C)

The actual production value shown in the city screen equals P-C. It's not really used or important for anything.

NP, TO and O can be calculated by substraction, all other values are displayed via pop-up menu in the city screen when you move the mouse over the hammer icon left of the production bar. Important: The value displayed there is essentially P-C. Keep in mind that ingame numbers are generally rounded, so the displayed values can be slightly different.

(A note to forests: Clearing forests will give you 30 BP for any tile within the 21-tile city radius, also 30 for the following ring, then 25 for the next, then 20, and so on, until 5. The production is added immediately, think of it as getting an additional turn with BP=30 (or 25...5).)

The variables are connected by the following formulae:

P = BP * (1+g+s)
TO = P-NP = C+O

(Note: more accurately, optional overflow from the previous project is also added to P; however for simplicity that's left out in further breakpoint etc. calculations. So in those cases where a new project will be comleted in one turn containing overflow, the formulae can be slightly off)

C is needed because any special bonus production should not carry over to a project that isn't allowed to use it.
The obvious, correct formula for calculating C is:

C = TO * s/(1+g+s)

Thus we get:

O = TO * (1+g)/(1+g+s) = TO-C
C/O = s/(1+g)

(In my opinion the overflow should be scaled up again in the next turn by the new s-bonus, but that isn't done at all. More on this issue at the end of this article.)

Unfortunately, Firaxis messed up procentual calculation, and uses the following formula:

C = TO * s/(1+s)
O = TO * 1/(1+s)
C/O = s/1

This seems fair at first glance, but it's plain wrong and introduces weird bugs when both s and g are not zero.

For example, it's possible to get less total production with an s-bonus than without. Using the above formulae, one can show that this happens exactly when NP < g*BP. In this case the loss due to (the bugged) overflow correction is larger than the added bonus production which normally should greatly increase the overflow. And we are not talking about a single incident here, in the late game g is often 100%, and the 25% from police state only marginally help to decrease the total loss during the turns when no overflow happens.

I believe that Firaxis experienced exactly this situation - but instead of looking for the bug and fixing the small formula, they created a bulky work-around, which (as we will see) kind of fixes the described problem, but introduces others.

Firaxis decided to split up overflow correction calculation, thus "manually" reducing the loss, but only for the described region. It really looks like they did that for the sole reason to never let the overflow with s-boni drop below the overflow the same city would generate without any s-bonus.

The full formula for overflow correction in the release version of civ4 is:

C = MIN[ TO * s/(1+s), BP * s^2/(s+1) ]

In other words, overflow correction is capped at f*BP, with f:= s^2/(s+1)

Let's analyse whether the overflow with capped correction could possibly be smaller than without s-bonus. This would be the case if:

BP * (1+g) - NP > BP * (1+g+s - s^2/(s+1)) - NP
<=> s < s^2/(s+1)
<=> s^2 + s < s^2

which is never the case for positive production boni.

Now we need to examine the location of the breakpoint:

TO*s/(s+1) > BP*s^2/(s+1)
<=> ((1+g+s) - NP/BP) * s/(s+1) > s^2/(s+1)
<=> 1+g > NP/BP

So the correction is capped if NP < (1+g) * BP, which is the exact condition for which the same city without special boni would generate overflow! Furthermore, the condition is weaker than NP<g*BP, so the mentioned weird situation will indeed never occur.

Of course, this is not how it *should* work, and we still get wrong results if both g>0 and s>0.

1. Below the breakpoint, there's always more production wasted than it should be, because of the wrong correction factor

s/(1+s) > s/(1+g+s).

Therefore you need to minimize overflow from an s-bonus project through micromanagement.

Waste equals BP*g*s/(s+1) - NP*g*s/(1+s)(1+g+s)
The maximum waste is for NP=1, with BP * g*s/(s+1) production wrongfully lost.

2. Above the breakpoint, the correction cap can result in insane amounts of hammers carried over to your next non-s-bonus project, which it should not be allowed to use. This is clearly an exploit, introduced by the weird work-around.
A quick rundown of the math follows:

The question is when a set amount of BP will produce more through overflow or through scaling with (1+g). Note that any overflow will not be added to BP next turn but to P. We need comparable BPs, so we introduce the "Overflow Base Production" OBP.

OBP := BP - NP/(1+g+s)

The amount of exploited hammers equals
O - OBP * (1+g)
= (BP*(1+g+s - s^2/(s+1)) - NP) - (BP*(1+g) - NP*(1+g)/(1+g+s))
= BP * s/(s+1) - NP * s/(1+g+s)

So, if you deliberately want to use this exploit, you need to maximize overflow through MM so that (at least)

NP/BP < (1+g+s)/(1+s)

The maximum exploit is for NP=1, with BP * s/(s+1) production won.

Note that this is a stronger condition than the breakpoint, since (1+g+s)/(1+s) < (1+g) is always true.

3. Overflow correction is always lost, even if you build the same project afterwards. Furthermore, even normal overflow from a non-s-bonus project will not be scaled by a possible s-bonus next turn.

This might be a deliberate decision, since it's questionable if excess -but in some sense "completed"- production should be granted a special production multiplier. However, I suspect that this was rather decided to minimize the already crucial exploit, which was introduced by the "correction cap" work-around.

Anyway, this strongly encourages MM to minimize overflow; either by reassigning population, or -better- by smart build orders.

/edit: total amount of wasted production for (1.) added

Great analysis, -- I guess you hold a Ph.D. of applied mathematics or sth. like that! :goodjob:
This should be an emergent problem for Firaxis to fix in a patch. I felt very glad when hearing MM is gone, then very sad when seeing it's back.

One other point that has been discovered is a possible max cap on total overflow based on current production (this is to avoid the wonder prebuild problem, which is back in a new and 'improved form')

Civ 1-3: Build a Palace and then switch production when Big building becomes available. (100% transfered..in civ2 less so)

Civ 4: Build any unit that can be completed with less than your current production (amount transferred = amount of overflow...so if you have a coastal city that makes 50 production, then it can churn out workboats and gain 25 production each turn in its overflow.) To increase this, use US gold to Rush the workboats each turn and get the full 50 as overflow each turn. (which can then be applied to a project.)
HOWEVER, there does appear to be a cap to how much you can get. The only test done so far was with explorers coming from a 100 base production city... the maximum carry over ended up at 302. Initial guess at a formula is ~3x base production, but we can't be certain. However that would mean that about 3 turns of production can be stored 'at a slight cost'.

Great analysis, -- I guess you hold a Ph.D. of applied mathematics or sth. like that! :goodjob:
Thanks -- and no (unless Physics count ;) )
Actually there's no complicated math involved at all, only simple calculation of percentage.

Wait until I finish the spectral analysis of the random number generator. I have the strong suspicion that the chances of getting a "lucky result" is not independent of previous battles...

(just kidding...about the spectral part)

One other point that has been discovered is a possible max cap on total overflow based on current production (this is to avoid the wonder prebuild problem, which is back in a new and 'improved form')
Yes, I noticed that too, but I don't think that the cap is a simple "3x base production". It could also depend on total production, or total overflow.

My guess is that it's a mixture of x*BP and "stored production is lost after a couple of turns" (which actually means the same as "3 turns can be stored at a slight cost").

On a similar topic, the manual mentions that production is stored on a partially finished project when you switch to something different - but that "you should not wait too long". I haven't tested this thouroghly, but got the impression that it doesn't depends on number of turns, but on number of projects.
E.g. when you start building a warrior but switch to worker in between (let's say 12 turns), nothing is lost. But, when chop-rushing a granary after the worker (less than 12 further turns) and only afterwards switching back to the warrior, you start at zero again.
So perhaps it's an all-or-nothing issue and you can only build one thing in between.

Switching research probably works the same.

lovely maths.... but i didn't really get the explanation.... what is the exploit exactly and how does it impact my game?

lovely maths.... but i didn't really get the explanation.... what is the exploit exactly and how does it impact my game?
The exploit is, that under certain circumstances too much production is carried over to the next project. When completing a project that receives a special bonus (that the next one doesn't), only the fraction of base production required to complete the project should be granted the special multiplier; It's easy to calculate the remaining amount of total production which should be carried over ((P-NP)*(1+g)/(1+g+s)), but in reality you sometimes get much less and sometimes much more (->exploit).

Example:
Imagine a city with 40 base production from tiles and specialists, forge+factory+energy (=>g=100%), and National (or was it Heroic?) Epic (=>military production +100%).

Building a unit which costs 120 will take exactly 1 turn (120=40*(1+1+1)), produce no overflow, so after the 2nd turn a building would have recieved a total of 80=40*(1+1) production. 3rd turn: 160.

On the other hand, by micromanaging your population you could reduce the base production to 39. In this case the unit would take 2 turns, but generate a total overflow of 117 - (120-117) = 114, and since the "overflow correction" is capped at (0.5*39)=20, your building will get 114-20 = 94 production. 3rd turn: 94+2*39 = 172.

So in this case, by reducing the base production you can increase the total production.

(The mathematically correct overflow in this example would be 117*(1+1)/(1+1+1) = 78, which is same as in the first case, minus the 2*1 lost hammers from MM)

(/edit: added 3rd turn numbers for clarification)

Hmm. So in this example - after 3 turns, without MM to reduce the production:
- You have your unit;
- The building has 160 hammers in it.

With the MM:
- You have your unit;
- The building has 94 hammers in it.

I know what I'd rather have - esp. if I'm timing a critical happy/health building with city growth, and I happen to desperately need that unit to fight.

The "pre-build" by saving up overflow hammers is interesting though. Shaving 300 hammers from the Eiffel Tower, say, might be handy.

Actually, it would be 160 vs. 174, if I'm understanding karmina correctly. In the first case, the 80 hammers is from your 2nd turn production. But the 94 in the second case is from the overflow - effectively you have it at the end of the 2nd turn, just like the 80.

So without MM, at the end of turn 3 you have 160 (80 turn 2 prod + 80 turn 3 prod).
With MM, at the end of turn 3 you have 174 (94 overflow + 80 turn 3 prod).

Okay, I get it now (but I still get my unit 1 turn late...).

Okay, so the calculation is snafued. Hopefully the patch will take care of it. Surely it'd be better if, when something is completed, the logic goes something like this:
0. Work out this turn's total production (the BP*(1+s+g) figure).
1. Are there enough hammers (including all bonuses) to complete production this turn? If yes, continue to 2. If no, just add everything and go no further.
2. Work out how many base hammers are required to complete production, after bonuses are applied. Would be...

(PC-GP) / (1+s+g)

Where:
PC is the total Project Cost
GP is the total Gathered Production for that Project

3. What is left is the next turn's overspill - to which appropriate bonuses can then be applied.

So...

O = BP - (PC-GP) / (1+s+g)

[Edit] Essentially - you're doing the logical maths. I.e. how much of your base production doesn't go towards the current project on the turn it finished. That's what the overspill ought to be.

[Edit2] This maybe something that is fixable with the SDK, of course!

So...
O = BP - (PC-GP) / (1+s+g)
[Edit] Essentially - you're doing the logical maths. I.e. how much of your base production doesn't go towards the current project on the turn it finished. That's what the overspill ought to be.
Exactly. That's the same as what I presented in the first post as "correct formula". With the exception of generic bonus already applied (your formula multiplied with 1+g), so that only special bonus needs to be taken care of in the next turn.
It doesn't matter if you count overflow in terms of base production or (1+g)*BP, as long as you treat it accordingly in the next turn. Mere BP might be easier to follow during the game, but you would always get overflow corrections as soon as you'd build a forge.

Rounding of course needs to be taken into account; specifically, rounding up of the (PC-GP) / (1+s+g) figure.

E.g.
Building a unit as before, on the last turn you need 20 hammers.
To get those 20 hammers you need 20 / 3 = 6.6666... base hammers - obviously this can't be the case so it'd actually be 7.

Leaving 40-7 = 33 overspill.

You have to calculate the overspill in base hammers because you may be building something that changes g, e.g. a factory, forge, power plant, or Iron Works.

Overflow production is lost during anarchy. This is another thing can be mm'ed (e.g. waiting for a turn in which nothing was completed to revolt or mm'ing so that nothing completes the next turn).

Overflow production is lost during anarchy. This is another thing can be mm'ed (e.g. waiting for a turn in which nothing was completed to revolt or mm'ing so that nothing completes the next turn).
And make sure not to pop-rush or finish chopping when (or the turn before) in Anarchy! Can be quite annoying once you realise that you just lost 20 shields.

Was this fixed in the patch? If not, has Firaxis been notified?

There's a way to exploit overflow in the early game with workers chopping down forests.

Say you are an aggressive civ, you get a 100% production bonus when building barracks. You have a city producing barracks and two workers timed to finish their chop on the same turn, giving the nearby city roughly 90 hammers. With the production bonus you get 90x2=180 hammers, 90 of which are used to complete the barracks and 90 free hammers (plus the base production of the city times 2, of course).

If you have more workers chopping forests, the overflow can be really ridiculously large. I hope they fix this in a patch?

If I understand that correct, the problem lies in making production boni produce more hammers instead of just making the building cheaper, correct?

Indeed, this is the problem.

I happened across this problem just today; thanks for the explanation!

Someone who has worked through all the details should file a bug report in the corresponding forum.

Thanks -- and no (unless Physics count ;) )
Wait until I finish the spectral analysis of the random number generator. I have the strong suspicion that the chances of getting a "lucky result" is not independent of previous battles...

(just kidding...about the spectral part)

Hehe. :D Actually did something similar for the civ3 RNG a long while back when it first came out. People were complaining so ferociously that it the results it produced were 'streaky' that I had to test it. :) So, I used the editor to remove all defense boni and sprinkled a map with warriors. After a rather meticulous counting of outcomes (the fact that civ3 saved the RNG seed in saves was helpful for experimental validity, IIRC) I found no reason to reject the hypothesis that they were anything but independent bernoulli trials. :)

I haven't read through all your work, but i think there is a mistake at the very beginning. I guess you already know it, in that case, please don't mind my message: I'll give two examples.

Example 1:

Let's imagine i am to build a 75 hammers building, with a s bonus of 50% (i don't care about the g bonus).
My base production is 100 hammers.

So:
BP = 100
s = 50%
NP = 75

TO = BP*(1+s) - NP = 150 - 75 = 75
O (expected) = 100 - 75 = 25 (since i am not supposed to benefit from the s bonus)

In that case, C = 50.

Example 2:
BP = 100
s = 50%
NP = 85 (the building costs now 85 hammers)

TO = 150 - 85 = 65
O (expected) = 100 - 85 = 15
C = 50

As you can notice, C and s are still the same, but TO as changed. That's enough to prove that C cannot be calculated as a percentage of TO.

Actually, C should be BP * s (which is 50), and
O = Max{TO - BP*s;0} (max because it shouldn't be below 0)...

But maybe i am wrong, since my formulae is quite simple and both firaxis and you are calculating it otherwise.

I haven't read through all your work, but i think there is a mistake at the very beginning. I guess you already know it, in that case, please don't mind my message: I'll give two examples.

Example 1:

Let's imagine i am to build a 75 hammers building, with a s bonus of 50% (i don't care about the g bonus).
My base production is 100 hammers.

So:
BP = 100
s = 50%
NP = 75

TO = BP*(1+s) - NP = 150 - 75 = 75
O (expected) = 100 - 75 = 25 (since i am not supposed to benefit from the s bonus)

In that case, C = 50.

Example 2:
BP = 100
s = 50%
NP = 85 (the building costs now 85 hammers)

TO = 150 - 85 = 65
O (expected) = 100 - 85 = 15
C = 50

As you can notice, C and s are still the same, but TO as changed. That's enough to prove that C cannot be calculated as a percentage of TO.

Actually, C should be BP * s (which is 50), and
O = Max{TO - BP*s;0} (max because it shouldn't be below 0)...

But maybe i am wrong, since my formulae is quite simple and both firaxis and you are calculating it otherwise.

Actually looking at your examples it depends on how you think of the bonuses working

Project=10
BP=10
Bonus=100%

now does that mean that
1. Base production is added to the project and Then Bonus production seperately (problem being this leads to MM being effective)

2. imagine each Point of production added one at a time through out the turn and each one getting its bonus.
Project: BP
0:10
+2(+1+100%): -1
2:9
4:8
6:7
8:6
10:5
so 5 BP left

This is the most effective way to model it because this minimizes the 'effect of the turn' which leads to errors of rounding/grouping production into clumps, etc. When production is grouped into clumps like that it leads to the painful unintuitive type of MM being effective.

Yeah you're right! I feel stupid not to have noticed that before. My solution was way too simple...

Well the update supposedly has changes in it, maybe they caught this bug (which also appears in research)

Well the update supposedly has changes in it, maybe they caught this bug (which also appears in research)

Where did you see that? I don't see anything about production overflow in the 1.52 change notes.

I didn't see anything but there were supposedly some undocumented changes. (that's why I Hope they fixed it)

The very first thing i noticed while playing under the new patch yesterday was that they fixed this. It's now exactly as was suggested in this thread.

The very first thing i noticed while playing under the new patch yesterday was that they fixed this. It's now exactly as was suggested in this thread.

Do you still get penalized on your carryover if you keep building the same thing with the same bonuses?

No.

Nor do you get a reward anymore for an overflow with a big bonus if the next thing you make doesn't have the bonus.

Like i said, it's been made exactly as it was suggested in this thread that it should have been in the first place.

No.

Nor do you get a reward anymore for an overflow with a big bonus if the next thing you make doesn't have the bonus.

Like i said, it's been made exactly as it was suggested in this thread that it should have been in the first place.

That sounds great! I'll have to look at it carefully once I finish GOTM 1 and can switch. :blush:

I checked out the claims about production overflow being fixed in v1.52, and there's one oddness. Overflow appears to be what it should be (in the cases that I checked), except that there's a cap on the possible overflow equal to the production cost of the last thing you built. So, e.g., if you are building a warrior (cost 15), and you're at 10/15, and you chop a forest for +30, and you add another 2 hammers from regular city production, you would expect overflow of 10+30+2-15 = 27, but instead the overflow is capped at 15.

This may be a "feature" rather than a "bug", but, in any case, it's important to be aware of.

I checked out the claims about production overflow being fixed in v1.52, and there's one oddness. Overflow appears to be what it should be (in the cases that I checked), except that there's a cap on the possible overflow equal to the production cost of the last thing you built. So, e.g., if you are building a warrior (cost 15), and you're at 10/15, and you chop a forest for +30, and you add another 2 hammers from regular city production, you would expect overflow of 10+30+2-15 = 27, but instead the overflow is capped at 15.

This may be a "feature" rather than a "bug", but, in any case, it's important to be aware of.
Wow, that's interesting. Thanks for finding this.


I have one question though. You said "...equal to the production cost of the last thing you built", but in your example, you only have the current production (a Warrior). Did you mean to say "...equal to the production cost of the thing you are currently building", is your example wrong, or do I not understand what you are trying to say?

I don't think the overflow cap is new -- I noticed under 1.09.

I have one question though. You said "...equal to the production cost of the last thing you built", but in your example, you only have the current production (a Warrior). Did you mean to say "...equal to the production cost of the thing you are currently building", is your example wrong, or do I not understand what you are trying to say?

If you just finished a Warrior (cost 15), and you had 42 hammers of production toward the Warrior, then you'll get only 15 hammers of carryover, not 27. This applies regardless of what you decide to build with the carryover.

When you're applying the carryover, then, by definition, you're not building the same thing any more. It's finished---that's why you are getting carryover.

Good to hear they really fixed this issue with the last patch! (haven't checked it myself yet)

About the overflow cap, there was something similar before, but it was way higher (about 2-3 times the city's base production), and thus allowed for limited wonder prebuilts.

Lowering the cap is definitely a good thing, although it might have made more sense to have a cap relative to the city's production rather than to the current product. As it is now, the same city can get you more overflow from e.g. a tank than from a much cheaper explorer.

Two cents: Overflow from one turn to the next should not even exist; it completely violates the spirit of the game model. Civ4 should force you to queue up enough projects to use up all the production on the current turn.

Correspondingly, there's no reason you shouldn't produce more than one thing a turn if you have lots of hammers and relatively cheap projects.

Project to project overflow is very simple to manage. All modifiers to production can be combined into one combined multiplier. If you have overflow on a project, just divide the overflow by the combined multiplier to get the number of raw hammers overflow to the next project. Repeat the same process until all the hammers are expended.

Was the loss of overflow due to anarchy affected by 1.52?

Carn

Was the loss of overflow due to anarchy affected by 1.52?

Carn

I don't know, but would like to know. Does anyone have the answer to this?