Production overflow explained and the return of MM

The purpose of this article is to analyse the maths behind production in Civ4, the multipliers involved, overflow, and what's called "correction"; further to explain the original bug and the not so great work-around introduced by Firaxis; finally to outline the exploit introduced by the work-around, and how to minimize the impact of the wrong math on your civilization's economy. The answer is micromanagement of your cities, which we all thought to be gone for good; and I do hope that Firaxis will fix their formulae as soon as possible.

Many thanks to everyone who contributed to this thread

Definitions:

BP:= Base production of a city (sum of hammers by worked tiles & specialists; also forest chops)
NP:= production needed to complete a project this turn (="costs-done")
P := total production
g := sum of generic bonus multipliers (e.g. 25% from forge)
s := sum of special bonus multipliers applicable to current project (e.g. 100% from stones for stonehenge)
TO:= total overflow (=P-NP)
C := overflow correction, hammers substracted from TO when an s-project finishes
O := actual overflow (=TO-C)

The actual production value shown in the city screen equals P-C. It's not really used or important for anything.

NP, TO and O can be calculated by substraction, all other values are displayed via pop-up menu in the city screen when you move the mouse over the hammer icon left of the production bar. Important: The value displayed there is essentially P-C. Keep in mind that ingame numbers are generally rounded, so the displayed values can be slightly different.

(A note to forests: Clearing forests will give you 30 BP for any tile within the 21-tile city radius, also 30 for the following ring, then 25 for the next, then 20, and so on, until 5. The production is added immediately, think of it as getting an additional turn with BP=30 (or 25...5).)

The variables are connected by the following formulae:

P = BP * (1+g+s)
TO = P-NP = C+O

(Note: more accurately, optional overflow from the previous project is also added to P; however for simplicity that's left out in further breakpoint etc. calculations. So in those cases where a new project will be comleted in one turn containing overflow, the formulae can be slightly off)

C is needed because any special bonus production should not carry over to a project that isn't allowed to use it.
The obvious, correct formula for calculating C is:

C = TO * s/(1+g+s)

Thus we get:

O = TO * (1+g)/(1+g+s) = TO-C
C/O = s/(1+g)

(In my opinion the overflow should be scaled up again in the next turn by the new s-bonus, but that isn't done at all. More on this issue at the end of this article.)

Unfortunately, Firaxis messed up procentual calculation, and uses the following formula:

C = TO * s/(1+s)
O = TO * 1/(1+s)
C/O = s/1

This seems fair at first glance, but it's plain wrong and introduces weird bugs when both s and g are not zero.

For example, it's possible to get less total production with an s-bonus than without. Using the above formulae, one can show that this happens exactly when NP < g*BP. In this case the loss due to (the bugged) overflow correction is larger than the added bonus production which normally should greatly increase the overflow. And we are not talking about a single incident here, in the late game g is often 100%, and the 25% from police state only marginally help to decrease the total loss during the turns when no overflow happens.

I believe that Firaxis experienced exactly this situation - but instead of looking for the bug and fixing the small formula, they created a bulky work-around, which (as we will see) kind of fixes the described problem, but introduces others.

Firaxis decided to split up overflow correction calculation, thus "manually" reducing the loss, but only for the described region. It really looks like they did that for the sole reason to never let the overflow with s-boni drop below the overflow the same city would generate without any s-bonus.

The full formula for overflow correction in the release version of civ4 is:

C = MIN[ TO * s/(1+s), BP * s^2/(s+1) ]

In other words, overflow correction is capped at f*BP, with f:= s^2/(s+1)

Let's analyse whether the overflow with capped correction could possibly be smaller than without s-bonus. This would be the case if:

BP * (1+g) - NP > BP * (1+g+s - s^2/(s+1)) - NP
<=> s < s^2/(s+1)
<=> s^2 + s < s^2

which is never the case for positive production boni.

Now we need to examine the location of the breakpoint:

TO*s/(s+1) > BP*s^2/(s+1)
<=> ((1+g+s) - NP/BP) * s/(s+1) > s^2/(s+1)
<=> 1+g > NP/BP

So the correction is capped if NP < (1+g) * BP, which is the exact condition for which the same city without special boni would generate overflow! Furthermore, the condition is weaker than NP<g*BP, so the mentioned weird situation will indeed never occur.

Of course, this is not how it *should* work, and we still get wrong results if both g>0 and s>0.

1. Below the breakpoint, there's always more production wasted than it should be, because of the wrong correction factor

s/(1+s) > s/(1+g+s).

Therefore you need to minimize overflow from an s-bonus project through micromanagement.

Waste equals BP*g*s/(s+1) - NP*g*s/(1+s)(1+g+s)
The maximum waste is for NP=1, with BP * g*s/(s+1) production wrongfully lost.

2. Above the breakpoint, the correction cap can result in insane amounts of hammers carried over to your next non-s-bonus project, which it should not be allowed to use. This is clearly an exploit, introduced by the weird work-around.
A quick rundown of the math follows:

The question is when a set amount of BP will produce more through overflow or through scaling with (1+g). Note that any overflow will not be added to BP next turn but to P. We need comparable BPs, so we introduce the "Overflow Base Production" OBP.

OBP := BP - NP/(1+g+s)

The amount of exploited hammers equals
O - OBP * (1+g)
= (BP*(1+g+s - s^2/(s+1)) - NP) - (BP*(1+g) - NP*(1+g)/(1+g+s))
= BP * s/(s+1) - NP * s/(1+g+s)

So, if you deliberately want to use this exploit, you need to maximize overflow through MM so that (at least)

NP/BP < (1+g+s)/(1+s)

The maximum exploit is for NP=1, with BP * s/(s+1) production won.

Note that this is a stronger condition than the breakpoint, since (1+g+s)/(1+s) < (1+g) is always true.

3. Overflow correction is always lost, even if you build the same project afterwards. Furthermore, even normal overflow from a non-s-bonus project will not be scaled by a possible s-bonus next turn.

This might be a deliberate decision, since it's questionable if excess -but in some sense "completed"- production should be granted a special production multiplier. However, I suspect that this was rather decided to minimize the already crucial exploit, which was introduced by the "correction cap" work-around.

Anyway, this strongly encourages MM to minimize overflow; either by reassigning population, or -better- by smart build orders.

/edit: total amount of wasted production for (1.) added

Great analysis, -- I guess you hold a Ph.D. of applied mathematics or sth. like that!
This should be an emergent problem for Firaxis to fix in a patch. I felt very glad when hearing MM is gone, then very sad when seeing it's back.

One other point that has been discovered is a possible max cap on total overflow based on current production (this is to avoid the wonder prebuild problem, which is back in a new and 'improved form')

Civ 1-3: Build a Palace and then switch production when Big building becomes available. (100% transfered..in civ2 less so)

Civ 4: Build any unit that can be completed with less than your current production (amount transferred = amount of overflow...so if you have a coastal city that makes 50 production, then it can churn out workboats and gain 25 production each turn in its overflow.) To increase this, use US gold to Rush the workboats each turn and get the full 50 as overflow each turn. (which can then be applied to a project.)
HOWEVER, there does appear to be a cap to how much you can get. The only test done so far was with explorers coming from a 100 base production city... the maximum carry over ended up at 302. Initial guess at a formula is ~3x base production, but we can't be certain. However that would mean that about 3 turns of production can be stored 'at a slight cost'.

Heroes said:

Great analysis, -- I guess you hold a Ph.D. of applied mathematics or sth. like that!

Click to expand...

Thanks -- and no (unless Physics count )
Actually there's no complicated math involved at all, only simple calculation of percentage.

Wait until I finish the spectral analysis of the random number generator. I have the strong suspicion that the chances of getting a "lucky result" is not independent of previous battles...

(just kidding...about the spectral part)

Krikkitone said:

One other point that has been discovered is a possible max cap on total overflow based on current production (this is to avoid the wonder prebuild problem, which is back in a new and 'improved form')

Click to expand...

Yes, I noticed that too, but I don't think that the cap is a simple "3x base production". It could also depend on total production, or total overflow.

My guess is that it's a mixture of x*BP and "stored production is lost after a couple of turns" (which actually means the same as "3 turns can be stored at a slight cost").

On a similar topic, the manual mentions that production is stored on a partially finished project when you switch to something different - but that "you should not wait too long". I haven't tested this thouroghly, but got the impression that it doesn't depends on number of turns, but on number of projects.
E.g. when you start building a warrior but switch to worker in between (let's say 12 turns), nothing is lost. But, when chop-rushing a granary after the worker (less than 12 further turns) and only afterwards switching back to the warrior, you start at zero again.
So perhaps it's an all-or-nothing issue and you can only build one thing in between.

Switching research probably works the same.

lovely maths.... but i didn't really get the explanation.... what is the exploit exactly and how does it impact my game?

Beeblbrox said:

lovely maths.... but i didn't really get the explanation.... what is the exploit exactly and how does it impact my game?

Click to expand...

The exploit is, that under certain circumstances too much production is carried over to the next project. When completing a project that receives a special bonus (that the next one doesn't), only the fraction of base production required to complete the project should be granted the special multiplier; It's easy to calculate the remaining amount of total production which should be carried over ((P-NP)*(1+g)/(1+g+s)), but in reality you sometimes get much less and sometimes much more (->exploit).

Example:
Imagine a city with 40 base production from tiles and specialists, forge+factory+energy (=>g=100%), and National (or was it Heroic?) Epic (=>military production +100%).

Building a unit which costs 120 will take exactly 1 turn (120=40*(1+1+1)), produce no overflow, so after the 2nd turn a building would have recieved a total of 80=40*(1+1) production. 3rd turn: 160.

On the other hand, by micromanaging your population you could reduce the base production to 39. In this case the unit would take 2 turns, but generate a total overflow of 117 - (120-117) = 114, and since the "overflow correction" is capped at (0.5*39)=20, your building will get 114-20 = 94 production. 3rd turn: 94+2*39 = 172.

So in this case, by reducing the base production you can increase the total production.

(The mathematically correct overflow in this example would be 117*(1+1)/(1+1+1) = 78, which is same as in the first case, minus the 2*1 lost hammers from MM)

(/edit: added 3rd turn numbers for clarification)

Hmm. So in this example - after 3 turns, without MM to reduce the production:
- You have your unit;
- The building has 160 hammers in it.

With the MM:
- You have your unit;
- The building has 94 hammers in it.

I know what I'd rather have - esp. if I'm timing a critical happy/health building with city growth, and I happen to desperately need that unit to fight.

The "pre-build" by saving up overflow hammers is interesting though. Shaving 300 hammers from the Eiffel Tower, say, might be handy.

Actually, it would be 160 vs. 174, if I'm understanding karmina correctly. In the first case, the 80 hammers is from your 2nd turn production. But the 94 in the second case is from the overflow - effectively you have it at the end of the 2nd turn, just like the 80.

So without MM, at the end of turn 3 you have 160 (80 turn 2 prod + 80 turn 3 prod).
With MM, at the end of turn 3 you have 174 (94 overflow + 80 turn 3 prod).

Okay, I get it now (but I still get my unit 1 turn late...).

Okay, so the calculation is snafued. Hopefully the patch will take care of it. Surely it'd be better if, when something is completed, the logic goes something like this:
0. Work out this turn's total production (the BP*(1+s+g) figure).
1. Are there enough hammers (including all bonuses) to complete production this turn? If yes, continue to 2. If no, just add everything and go no further.
2. Work out how many base hammers are required to complete production, after bonuses are applied. Would be...

(PC-GP) / (1+s+g)

Where:
PC is the total Project Cost
GP is the total Gathered Production for that Project

3. What is left is the next turn's overspill - to which appropriate bonuses can then be applied.

So...

O = BP - (PC-GP) / (1+s+g)

[Edit] Essentially - you're doing the logical maths. I.e. how much of your base production doesn't go towards the current project on the turn it finished. That's what the overspill ought to be.

[Edit2] This maybe something that is fixable with the SDK, of course!

eldar said:

So...
O = BP - (PC-GP) / (1+s+g)
[Edit] Essentially - you're doing the logical maths. I.e. how much of your base production doesn't go towards the current project on the turn it finished. That's what the overspill ought to be.

Click to expand...

Exactly. That's the same as what I presented in the first post as "correct formula". With the exception of generic bonus already applied (your formula multiplied with 1+g), so that only special bonus needs to be taken care of in the next turn.
It doesn't matter if you count overflow in terms of base production or (1+g)*BP, as long as you treat it accordingly in the next turn. Mere BP might be easier to follow during the game, but you would always get overflow corrections as soon as you'd build a forge.

Rounding of course needs to be taken into account; specifically, rounding up of the (PC-GP) / (1+s+g) figure.

E.g.
Building a unit as before, on the last turn you need 20 hammers.
To get those 20 hammers you need 20 / 3 = 6.6666... base hammers - obviously this can't be the case so it'd actually be 7.

Leaving 40-7 = 33 overspill.

You have to calculate the overspill in base hammers because you may be building something that changes g, e.g. a factory, forge, power plant, or Iron Works.

Overflow production is lost during anarchy. This is another thing can be mm'ed (e.g. waiting for a turn in which nothing was completed to revolt or mm'ing so that nothing completes the next turn).

eg577 said:

Overflow production is lost during anarchy. This is another thing can be mm'ed (e.g. waiting for a turn in which nothing was completed to revolt or mm'ing so that nothing completes the next turn).

Click to expand...

And make sure not to pop-rush or finish chopping when (or the turn before) in Anarchy! Can be quite annoying once you realise that you just lost 20 shields.

Was this fixed in the patch? If not, has Firaxis been notified?

There's a way to exploit overflow in the early game with workers chopping down forests.

Say you are an aggressive civ, you get a 100% production bonus when building barracks. You have a city producing barracks and two workers timed to finish their chop on the same turn, giving the nearby city roughly 90 hammers. With the production bonus you get 90x2=180 hammers, 90 of which are used to complete the barracks and 90 free hammers (plus the base production of the city times 2, of course).

If you have more workers chopping forests, the overflow can be really ridiculously large. I hope they fix this in a patch?

If I understand that correct, the problem lies in making production boni produce more hammers instead of just making the building cheaper, correct?

Indeed, this is the problem.