1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Food waste at hight surplus

Discussion in 'Bugs and Crashes' started by sorcdk, May 17, 2019.

  1. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    Ah... right. I remember that about this code but I didn't follow the session terminology in the way it was meant. Thanks for clarifying.
     
  2. Anq

    Anq Prince

    Joined:
    Apr 14, 2019
    Messages:
    351
    Gender:
    Male
    Location:
    Anser (geese) HQ
    I mean until you exit the game to desktop. The mechanic is just a lookup table, after the first run to calculate, much like how you cache what buildings enables other buildings for AI value calculation. It seems to change dynamically, but in fact it is a one-in, one-out function. Given the same total food production and the same population, this calculation turns out always the same. Lol I was too not quite specific about that "session" terminology.
    Ugh... that means when we plot the result of that function against its input on a graph, it will represent as an upwards curve, but this curve continues in a straight line after some point, and after this point we don't need to trace the curve anymore. After this point it is easier to calculate again and again than in the curved section.
     
    Last edited: May 18, 2019
  3. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    And it only ever calculates those amounts it needs to know an answer for and if its calculated that answer before in a given session, it becomes just a lookup instead. Right.

    Going back and rereading what you said to get here, I can see that you mean that the suggested performance improvement wouldn't improve anything much because the calculation is so infrequently run to begin with. Got it. I thought he was pointing at a potential miscalculation though?

    Admittedly this is where I was having trouble following a lot of the thread in general, and you wouldn't be the first person I frustrated with my graduated high school in 92' level of math terminology grasp. Sorry but there are limits for my ability to pick up the terms used in these discussions, even if the charts usually do make some sense as I do visualize numeric processes pretty well for an undereducated American. I have tons of respect for you guys who can use these phrases with proficiency.
     
  4. MattCA

    MattCA Warlord

    Joined:
    Jan 25, 2019
    Messages:
    246
    Gender:
    Male
    is this the type of thing that makes me wonder if it is necessary to calculate it in-game. Y not comment out the calculation and have just an array of the results in place of the calculation for any parts that don't have variables that will change from game to game.

    EDIT: so youd get it to calculate the surplass thing, then calculatedWaste[] would be an array of the correct numbers in CvGame or CvGlobals.
    would end up being somethin like return GC.getcalculatedWaste(surplass) instead of calc
     
    Last edited: May 18, 2019
  5. Anq

    Anq Prince

    Joined:
    Apr 14, 2019
    Messages:
    351
    Gender:
    Male
    Location:
    Anser (geese) HQ
    I struggled to put these phrases into context, too! In Chinese we don't have as much vocabulary for describing Mathematics as in English, and I haven't used Maths for a long time since graduated from high school in '12. I searched for the terms and examples how to use them, and used my high school understanding to describe the pattern, first in my native language, then in English. It feels so unnatural so speak in such way :lol:

    Why is it spelled SURPLASS, I feel it's cool to see different spellings, like how Tchaikovsky is spelled Tschaikowski somewhere!
     
    Last edited: May 18, 2019
  6. Toffer90

    Toffer90 C2C Modder

    Joined:
    Oct 16, 2011
    Messages:
    6,630
    Location:
    Norway
    You're doing great. :thumbsup:
    This is only calculated once, then cached for lookup...
     
  7. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    Well then you're just promising that the game starts up with the full amount of potential memory consumed. It is an attempt to only use the memory it's needed throughout the course of the session. If the calculation can be made to not take long, perhaps for the sake of memory, it might be better not to cache this at all... I doubt it's taking too significant a memory cost but it certainly would be a LOT heavier (storing the waste for every amount of potential food surpluss!!!) if we tried to go without the calculation at all. I do have to wonder if a simpler calculation without caching might be a better path to take with waste anyhow.
    Ok, well the struggle is real on both sides of the fence so that's good to know. Just know that at times we'll have to slow down and request some further clarifying. This math formula to start with was engineered like a true college grad would've. I think AIAndy might've been the one to invent this but Koshling was equally capable of it. Those guys were smarter than hell and sometimes they were very hard for me to follow. That you and @sorcdk were able to communicate like you were at the beginning of this thread makes me feel quite small with how little I could grasp of what you were saying sometimes.

    I don't think you were the one to start the Surplass misspelling... Some previous modders have been horrendous spellers. I'm cursed to see misspellings stand out like sore bleeding wounds sometimes.

    Well, each result is, whenever that particular food excess amount comes up. IIRC anyhow. Am I right?
     
  8. JosEPh_II

    JosEPh_II TBS WarLord

    Joined:
    Jun 20, 2007
    Messages:
    16,012
    Gender:
    Male
    Location:
    Western IL. cornfields
    I hope we just keep this whole food waste at a minimum of code. Being complicated = harder to find the Problems created "bugs" the further down the road we go.

    Just like koshling labeled the Revolutions coding years ago, it becomes spaghetti. And then no one will want to fix it when it breaks.
     
  9. sorcdk

    sorcdk Chieftain

    Joined:
    Jun 26, 2014
    Messages:
    75
    Location:
    Denmark
    Before I go on, I better explain a detail about the current wastage system, in the current system we split the food surplass into 3 parts:
    1. An amount that we do not caculate wastage for. This is currently equal to 50% how much your population eats
    2. A small fraction (about 4.8%) of the food that we calculate wastage for that is never wasted.
    3. The part of the food we calculate wastage for that is reduced. This relatively quickly reaches 100% of all extra food added. You can get about 18 non-wasted food from this category in total no matter how much extra food you through at it.
    I mainly see 2 problems (after the fixes to the actual bug part):
    1. The food wastage fairly quickly go to 100% wastage in category 3 above. Mostly the problem is how you seem to trigger a point around 40-60 surplass where you are barely going to get any more growth by adding food. I have included a figure showing this below. This means that you are not really getting any value from additional food bonuses, as opposed to just a reduced efficiency of those bonuses. This effect on the player is then that food wastage signifies a full stop sign for investment into food, and it fixes the growth rate of cities to be only based on food amount needed and storage after growth (the exception is the use of food merchants).
    2. The parameter, B, that we can use to tweak the current calculation of wastage has the unfortunate role of both decreasing and increasing wastage when changed. Specifically lowering it would increase the time until we hit the full stop sign, but it would also reduce the category 2 above, which the food you can get after you have exploited most of you could from catagory 3, so such an action would be kind of like pissing in the pants, unless we lower it by so much that fully exploiting catagory 3 would likely not happen, but that would also leave wastage mostly not affecting you early on, which is also a problem. If we instead increased B we would even more quickly reach the point where only catagory 2 would be in effect, but then we might as well just have a fixed reduction in food, since that is all it would do outside of a small region. In total there arent very many good options for B, and if we need to do anything other than minor tweaks, then we are kind of screwed by the math.
    Spoiler Close look at the food after wastage :
    foodWastage3Small.png


    The W(s) and W_2(s) referer to formulars in the earlier full on math discussion. W(s) is indeed the current code and W_2(s) a suggested alternative. I use the _2 notation as naming convention for model #2, with the use of w,f,W,F describing whether it is wastage or food lefter after wastage (w vs f) and whether it is the effect on the last added food or the total food (lower vs capital letters). Model #2 have an additional model parameter called A, which is used to describe the width of catagory 3, and it has the feature that catagory 3 is not limited to a constant possible amount that you can get out of it.
    I would also like to propose a third model, which is simpler to calculate, as it would just apply a power function to the total food, with the model parameter P. Note that this would in the current from not have a category 2.
    Here is a recap of the math each of the 3 models are based on:
    • f(s) = (B + (1-B)^(s))/(1+B)
    • f_2(s) = B + (1-B)/(1+A*s)
    • F_3(S) = S^P
    Remember that (for all the models) w(s) = 1-f(s), F(S) = S - W(S), and that W(S) = sum(w(s)) where s goes from 1 to S.
    Below I have done a model parameter study, where I have plotted 4 different possible values for A and P, so we can see which ones we might want to use.
    Spoiler Parameter study of model #2 and #3, normal plots :
    foodWastage0.png foodWastage1.png foodWastage2.png foodWastage3.png

    Spoiler Parameter study of model #2 and #3, logarithmic plots :
    foodWastage0log.png foodWastage1log.png foodWastage2log.png foodWastage3log.png


    I have also included the modified python code (as a .txt file), which creates these graphs. Note that it will save the files by default, unless you set the manual variable to True. Note that it requires the numpy and matplotlib modules.
     

    Attached Files:

    Anq and MattCA like this.
  10. Toffer90

    Toffer90 C2C Modder

    Joined:
    Oct 16, 2011
    Messages:
    6,630
    Location:
    Norway
    @sorcdk: Thanks.

    Model number 2 with A set to 0.1 seems to produce the most reasonable result imo.

    The "model 3" calculation may actually take longer to process. In most programming languages one can do a lot of multiplications/additions/subtractions/division before the processing time of them becomes comparable with one exponentiation call.
    e.g. math.pow(x, 30) probably take longer to process than x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x... One should benchmark exponentiation vs. simpler math if performance is of any interest.

    The math for "model 2" should be so fast that one could consider not to cache anything, and rather calculate the wastage on demand.
    The RAM saved would be small, but the processing added would not be noticeable unless there is perhaps as many as 5000 cities on the map.
    The mixed system is good though, cache up to a point and let a linear function calculate wastage for food surplus past that point.

    I think we should let the discussion continue for about a week, or until it's decided that there's nothing left to discuss.
    After that either you or Anq join the team, or one of you upload a modified source file for one of the current team members to compile for the SVN.
     
    MattCA likes this.
  11. sorcdk

    sorcdk Chieftain

    Joined:
    Jun 26, 2014
    Messages:
    75
    Location:
    Denmark
    The problem with model 2's performance is that we do not (yet?) have a nice closed form solution for it, so we would need to calculate all the previous values each time by recursion, which is why we might want to have it cached (it is at most a few kb of ram). Looking up the type of series it is based on, their results are known as harmonic numbers H_n, and they are reasoanably approximated with the natural logarithm times a constant. Calculating the natural logarithm would in general be more expensive that a relatively simple power function, since there are a few algorithmic tricks that make power functions faster than strait up multiplication. It does use fractional powers though, so we would have to measure it to see whether it is a problem.

    Personally I am testing out model 2 with A = 0.05 in my game, and for now it seems to feel problematic, but I am still very early in the game (pre-tribalism). Note that model B with A=B follows the curve of model 1 fairly closely early on, and then just tapers off to less efficiency slower than model 1. With A = 0.1, we get more wastage when below 20 surplass compared to model 1, and less afterwards, so it really depends on whether you want that effect.

    I also noticed that both model 1 and 2 have S shaped graphs for efficiency in the logarithmic graphs, this means that there is a specific order of magnitude of surplass where most of the loss in efficiency happens, this will tend to be somewhat of tell-tale sign that you shouldnt be pushing as hard to food anymore, though how significant such a change is depends on the model, #1 clearly quickly drops all pretense of there being anything left for you, while model 2 is a bit more soft, though it depends on A.

    Looking at the graphs, I one part of me is slightly partial to model 3 with P around 0.75, it is much more punishing early on, which will show the player quickly that they are hitting the wastage region, and what it means to hit the wastage region, but it will only slowly increase after that, which let the player make educated guesses at how much wastage he would be hit by when adding more food. Then again that shape might not be optimal if we want to affect it with tags, though the technical challenge of doing so looks smaller (it is easier to apply changes if we do not have to keep a cache of the function, meaning we would be allowed to change its shape), though if we start with a low P, we could have P constructed as 1-c*v, where c would be the base and r would by some modifier from tags (or 1-c/v if we want to not be as restricted in where we add the tag). If we want to do something similar to model 2, then we could simply do the scale S down by a factor v, and then multiply the result with v, this would have an effect similar to dividing A with v. If we go with that, we might want to choose an A parameter that is more agressive than we would otherwise have chosen.

    I might have rambled along a bit too much there though. Do we need some more graphs with parameters choosen closer to the values we have considered, in case we want more fine-tuning? We could also wait with this till later.22

    Anyway, here is the c++ code as I am testing it currently, in case we choose model 2 (I havent bothered creating a tag to read in A, so it is just hard coded for now):
    Spoiler c++ source: CvCity.cpp: CvCity::foodWastage(...) :

    Code:
    float CvCity::foodWastage(int surplass) const
    {
    #define    MAX_SURPLASS    2000
        static    float    calculatedWaste[MAX_SURPLASS];
        static    int calculatedTo = -1;
        int        startWasteAtConsumptionPercent = GC.getDefineINT("WASTAGE_START_CONSUMPTION_PERCENT");
        float    wastageGrowthFactor = GC.getDefineFLOAT("WASTAGE_GROWTH_FACTOR");
    
        if ( wastageGrowthFactor == 0 )
        {
            wastageGrowthFactor = (float)0.05;    //    default
        }
    
        if ( startWasteAtConsumptionPercent >= 0 )
        {
            if ( surplass == -1 )
            {
                int iPopulationExponent = getPopulation() - 1; // Each pop past the first increases consumption per population by .1, rounded down.  Each point of population means more actual people the higher the amount goes.
                int iConsumptionPerPopulationBase = iPopulationExponent + (GC.getFOOD_CONSUMPTION_PER_POPULATION() * 10);
                int iConsumptionbyPopulation = (getPopulation() * iConsumptionPerPopulationBase) / 10;
                surplass = foodDifference(true, false) - (iConsumptionbyPopulation * startWasteAtConsumptionPercent)/100;
            }
        }
        else
        {
            surplass = -1;
        }
    
        //    Nothing wasted if there is no surplass
        if ( surplass <= 0 )
        {
            return 0;
        }
        //    Cache what we can as it's not a trivially cheap computation
        else if ( surplass <= calculatedTo )
        {
            return calculatedWaste[surplass];
        }
        else
        {
            if ( surplass >= MAX_SURPLASS )
            {
                //    After the max we shift to from assymtotic behavior toward the limit to the limit of efficiency
                return foodWastage(MAX_SURPLASS - 1) + (foodWastage(MAX_SURPLASS - 1) - foodWastage(MAX_SURPLASS - 2)) * (surplass - MAX_SURPLASS + 1);
                //return foodWastage(MAX_SURPLASS - 1) + ((float)1.0 - wastageGrowthFactor/((float)1.0+wastageGrowthFactor)) * (surplass - MAX_SURPLASS + 1);
                //    After the max we bother calculating it all gets wasted
                //return calculatedWaste[MAX_SURPLASS-1] + ((float)1.0 - wastageGrowthFactor) * (surplass - MAX_SURPLASS + 1);
            }
            else
            {
                calculatedWaste[surplass] = foodWastage(surplass - 1) + (float)1.0 - (wastageGrowthFactor + ((float)1.0 - wastageGrowthFactor)/((float)1.0 + (float)0.05 * (float) surplass ));
                //calculatedWaste[surplass] = foodWastage(surplass-1) + ((float)1 - (wastageGrowthFactor + pow((float)1.0 - wastageGrowthFactor, surplass))/((float)1.0+wastageGrowthFactor));
                calculatedTo = surplass;
    
                return calculatedWaste[surplass];
            }
        }
    }
     
  12. Toffer90

    Toffer90 C2C Modder

    Joined:
    Oct 16, 2011
    Messages:
    6,630
    Location:
    Norway
    Right, I forgot about the recursive part of the calculation as the formula f_2(s) = B + (1-B)/(1+A*s) that you wrote looked so simple without any recursive steps being obvious.
    I can agree that model 3 does do the job fairly well.

    I'm not sure there's really a great need for new building tags related to the food wastage system... But I'm sure we would find ways to implement them for a power function too.
    Perhaps the easiest and most sensible tags to develop for the wastage system would be a tag that adjust how much of the surplus food that is immune to wastage, currently it is 50% of the food consumed by the city, perhaps that should be reduced to simply 1:food: +25% of food consumed as a base and then we introduce tags that modify that amount.
    The help text for e.g. Storage Pit could be "+25% surplus :food: safe from wastage." Tag A=25
    The help text for e.g. Modern Granary could be "+100% surplus :food: safe from wastage." Tag A=100
    The help text for e.g. Dried Meat Maker could be "+1 surplus :food: safe from wastage." Tag B=1
    So a city of pop 1 that consumes 4 food per turn that have a Modern Granary and a Dried Meat Maker would be able to have 6 surplus food completely protected from wastage (150% of consumed food). (1 + 1 + 1)*2
    A city of pop 10 that consumes 40 food per turn that have a Modern Granary and a Dried Meat Maker would be able to have 24 surplus food completely protected from wastage (60% of consumed food). (1 + 10 + 1)*2

    Anecdote: The wastage system should really have a pedia entry in the concept category at some point...

    We should imo not overdevelop the wastage system with lots of tags prematurely.
    We also have to consider if it's possible to radically change how cities grow, many modders have played with that thought, perhaps population growth shouldn't be 100% dependent on food availability, perhaps we want to change how cities starve too... etc.
    How would such changes affect how we regard the wastage system, in the big picture there's a lot to consider.

    Spoiler An example of a different approach to city growth :
    An example of a different approach to city growth could be to have a status bar from 0-100% for growth that slowly increases as long as there is a surplus storage in the city, and decreases the moment all surplus storage runs out.
    So if you are at 80% progress towards the next population point and the storage runs out of food, then it may take multiple turns before the progress bar reaches 0% and the city loses a pop point. One could add in a lot of factors that decide how fast the progress bar rises or falls, it increases faster with poor education levels but also falls slower when the food storage are empty, lacking food could impose a happiness and unhealthiness penalty to make a slower fall problematic. The Ideas concept that TB is planning could be a major factor in how fast population grows when there is available food in storage.
    The amount of negative food surplus per turn would impact how fast the progress bar falls.

    The maximum possible amount of food that can be stored could for example have a base of 100 multiplied with the city population size.
    Building a storage pit may add 10% more storage capacity.
    When a city reach 100% in the growth progress bar the city would grow by one size but the food storage would not change other than the maximum storage capability increasing.
    Events could destroy harvests reducing food yield in the city for x turns and other events could ruin most of the stored food. Perhaps we could add a small randomness to how much food yield a city have per turn, let's say that 15% of the food yield is uncertain for each turn where a random amount within the range defined by those 15% is what you get on top of the certain 85%.

    It's supposed to be a small example, so I'll stop here.
    This might deserve its own thread if there is enough interest in brainstorming something like this further.
     
    Last edited: May 19, 2019
    Anq likes this.
  13. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    That's about all I was hoping it would be. That would be perfect. I'm surprised to learn that it's currently what is over 50% of the food consumed by the city - I had thought I understood that wastage only applied to food NOT consumed by the city already.

    Anyhow, this set of simple tags would work nicely I believe.

    Completely agree.

    For the rest of your post - interesting further thoughts. Will require more consideration.
     
  14. Toffer90

    Toffer90 C2C Modder

    Joined:
    Oct 16, 2011
    Messages:
    6,630
    Location:
    Norway
    That's not entirely accurate, the food wastage only applies to the food amount that is not consumed, the surplus food that is.
    However, the amount of surplus food that is safe/immune from wastage is equal to 50% of the food currently consumed by the city each turn.
     
  15. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    Oh... I see. Thank you for clarifying!

    So are we implementing what @sorcdk has come up with then?
     
  16. Thunderbrd

    Thunderbrd C2C War Dog

    Joined:
    Jan 2, 2010
    Messages:
    27,264
    Gender:
    Male
    Location:
    Las Vegas
    @sorcdk I'm including your code since I'm hearing nothing further on this and would like to have it in play for testing during the freeze. I was hoping I'd get a report from your testing but I'll assume no news is good news? We'll see I suppose.
     
    MattCA likes this.
  17. sorcdk

    sorcdk Chieftain

    Joined:
    Jun 26, 2014
    Messages:
    75
    Location:
    Denmark
    I am only at early ancient in my game (I had to restart a couple of times for various reasons, such as forgetting to turn on the right options, and I havent had much time to play in 1-2 weeks).
    At the moment it still feels like food is worth investing in, and cities are not signifcantly larger than what they usually tend to be at this stage. Capital is size 9 (I did settle a lot of great hunters in it and build the +1 all cities wonder), which is maybe 1-2 sizes larger than I would normally reach, the rest of the cities are still in the "build all the hammer buildings" phase and havent really grown yet.
    Are there any specific points you want reports on?
     

Share This Page