Gedemon's Civilization, development thread

[DemonEmperor]

Question: Where can I see employment figures? and how they affect manufacturing output?

 

[DemonEmperor]

Got a Divide by 0 Error. attached Lua log below

Edit: It appears to be caused by someone running out of food, causing the migration value calculation to bug out

Edit 2: Appears to be usage of "Div" utility function in PlotScripts, lines 2681 - 2686
Recommend replacing with standard division. Divide by 0 is good, since then you get infinite push in case you have no food left, and nan in case no food, and no food demand.

In case of nan, the following if statement won't trigger, which is good, since no food need means food is no motivation to migrate.

In case of inf, assuming it's always inf, since negative stock values shouldn't happen
Case foodNeeded = 0 (shouldn't be possible, but here for completeness):
migration.Push.Food remains 0, no change necessary (again no migration trigger)

Case foodstock = 0 (empirically possible):
migration pull goes to 0, migration push goes to inf
thus, starving popnum = pop - (pop / inf) = pop - 0 = pop
which is good, because no food ==> literally everyone is starving

 

[Gedemon]

small update

- prevent a few dead end on the tech tree by tweaking prerequisites
- tweak Food status icons on the City Banner in relation to rationing
- fix the "Monitor" function that was not using pcall with the correct syntax
- bug fix: sometime a newly created unit failed to register
- bug fix: unsynchronized UI/Gameplay could cause a call to a nil city for the banner
- log calls to the orderedNext function in case a script fails midway
- bug fix: unlock Government at the prereqTech now that it exists
- bug fix: don't try to collect resources for an unit on a nil plot (ie removed from map)

if you get a unfinished City turn error in the log, you can change (in GCO_CityScript.lua)

function DoCitiesTurn( playerID )
    --GCO.Monitor(CitiesTurn, {playerID}, "Cities Turn Player#".. tostring(playerID))
    CitiesTurn( playerID )
end

to

function DoCitiesTurn( playerID )
    GCO.Monitor(CitiesTurn, {playerID}, "Cities Turn Player#".. tostring(playerID))
    --CitiesTurn( playerID )
end

it will run slower but should output the precise error message in the log after replaying that turn

Same for units, changing (in GCO_UnitScript.lua)

function DoUnitsTurn( playerID )
    --GCO.Monitor(UnitsTurn, {playerID}, "Units Turn Player#".. tostring(playerID))
    UnitsTurn( playerID )
end

to

function DoUnitsTurn( playerID )
    GCO.Monitor(UnitsTurn, {playerID}, "Units Turn Player#".. tostring(playerID))
    --UnitsTurn( playerID )
end

 

[DemonEmperor]

@Gedemon so apparently the serializer/deserializer *really* doesn't like nan values. So, if a nan value is going to be in a saved table, you have to work around it and prevent the nan from existing. "inf", on the other hand, *should* be fine, since tables are able to use it as an index.
Currently testing if the serpent serializer/deserializer knows how to handle it

 

[Gedemon]

View attachment 554558

Got a Divide by 0 Error. attached Lua log below

Edit: It appears to be caused by someone running out of food, causing the migration value calculation to bug out

Edit 2: Appears to be usage of "Div" utility function in PlotScripts, lines 2681 - 2686
Recommend replacing with standard division. Divide by 0 is good, since then you get infinite push in case you have no food left, and nan in case no food, and no food demand.

In case of nan, the following if statement won't trigger, which is good, since no food need means food is no motivation to migrate.

In case of inf, assuming it's always inf, since negative stock values shouldn't happen
Case foodNeeded = 0 (shouldn't be possible, but here for completeness):
migration.Push.Food remains 0, no change necessary (again no migration trigger)

Case foodstock = 0 (empirically possible):
migration pull goes to 0, migration push goes to inf
thus, starving popnum = pop - (pop / inf) = pop - 0 = pop
which is good, because no food ==> literally everyone is starving

thanks, have you tested it, a divide by 0 was causing this issue too, not just the save corruption.

I suppose we'll have to convert the string by doing a loop on the table after deserializing too.

Also Lua references say that (0/0 == 0/0) should return false, but civ6 Lua does return true. Confusing.

 

[DemonEmperor]

@Gedemon wait what? that violates nan principles. Best way to test for nan should be to check:

if (not x >= 0) and (not x < 0) then -- x is nan

Haven't been able to see if allowing divide by 0 in that file causes fleeing the city yet, currently getting the new serializer to work

 

[Gedemon]

Thanks, it would be good to have it fail proof (as long as it stay fast)

But I'd like to remove the cases where I was dividing by 0, that was never intended, and that's why I added the divide function.

 

[DemonEmperor]

Personally I would be fine with leaving select / 0 things in, as illustrated in my earlier population migration case, but yeah divide by 0s are generally useful for highlighting logic errors

In regards to the serializer, got a version working - no idea if it works on such cases or not. Going to start a new game with that foodstock/foodNeeds div check disabled to see if it handles it properly

The serializer has been tested to work on next turn (comparing initial to final tables after serialize/deserialize), and reloading an earlier turn. Have not tested past turn 10, so any exotic effects like NaN/nil/inf, as stated above, hasn't been tested yet.

File has been attached for reference.

 

[DemonEmperor]

Got another set of Divide by 0s detected. Is not from that specific migration location because I disabled it.
But, it is from migration due to starvation in GCO_CityScript.lua, lines 7001/7002

-- Starvation
            -- Todo : get values per population class from NeedsEffects instead of global values here
            local consumptionRatio                        = 1
            local foodNeeded                            = self:GetFoodConsumption(consumptionRatio)
            local foodstock                                = self:GetFoodStock()
            cityMigration.Pull.Food[populationID]        = Div(foodstock, foodNeeded)
            cityMigration.Push.Food[populationID]        = Div(foodNeeded, foodstock)
            cityMigration.Migrants.Food[populationID]    = 0

if cityMigration.Push.Food[populationID] > 1 then 
                local motivationWeight = cityMigration.Push.Food[populationID] * migrationClassMotivation.Food[populationID]
                if motivationWeight >= bestMotivationWeight then
                    cityMigration.Motivation[populationID]      = "Food"
                    bestMotivationWeight                        = motivationWeight
                end
                local starving                              = population - Div(population, cityMigration.Push.Food[populationID])
                cityMigration.Migrants.Food[populationID]   = math.floor(starving * classesRatio[populationID] * math.min(1, migrationClassMotivation.Food[populationID]))
            end

Have attached lua log for completeness.
Going to replace Div and see if allowing for inf/nan messes up save/load

 

[CalebR_300]

Question: Where can I see employment figures? and how they affect manufacturing output?

Hover over the population number in the city panel.

Thanks for the rundown for my various questions Gedemon. So here's another:
I just realised that employment size is based on the same exponential as population, so after 10 units of pop are employed (630957 people), the next employer that provides 1 unit of pop employment will require 192990 workers (if my math is right). Whereas the first employer will only require 1000 workers.
From memory, pop units convert using (pop^2.8*1000), but why the exponential system? Why not a linear (say, 50,000 per pop)? Is it a just a game limitation?
No rush to answer, I know you tech heads are busy solving problems out of my depth.

 

[Gedemon]

I see "buildings" in cities as real world "district", and the employment in one should scale with a city size (larger districts in larger cities)

With a linear value I couldn't match the population number, or a late building would have required too many population in a small city (or wouldn't provide enough employment in a large city)

But anyway the direct exponential calculation for cities employment was the case before the balance pass, now the employment size is directly relative to the city size, much easier to balance, and I'll surely adapt it to other mechanism related to population size (done for luxuries requirement, to do for employment in map tiles, ...)

Now it's calculated like this:
MaxPopulationEmployment = CityPopulation * (SumBuildingsEmploymentSize / CitySize)

This way it scale with cities but shouldn't reach absurd values.

 

[CalebR_300]

I understand it's all in proportion, but it's a bit difficult for me to conceptualize an exponential, since '10000' people may be equal to 2 and bit pops, or just a fraction of one. I suppose if everything was measured in pop units (like how employment is) it would probably remove any confusion.
Moving on, time for some general feedback from my last play-through (prior to the most recent patch), up to turn 204:

• My cities seemed perpetually hungry, despite having a focus on improved bonus food resources; there just doesn't seem to be enough rural workforce, and thus not enough food. I'm guessing what you were talking about making map tile employment scale will probably fix this.
  
• Government stability seems highly dependent on your clay/leather/wood output, as with 4 cities, my stability dropped from 80% to around 60% before I got a decent enough supply of clay/leather to produce more admin tablets/scrolls despite each city having an ancestor's hall and forum. I imagine this is tied to above.
• Ironically, loss of productivity in the city reduces materials available for admin tablets/scrolls, which can then lead to more stability loss as the city grows, thus further reducing productivity. Thankfully I've managed to avoid a full feedback loop.
  
• A lot of my empire stability drop was tied to the amount of land I had. I'm not sure what the rules are regarding claiming tiles, but I think land claims beyond the workable city radius should be suppressed whilst lacking stability. Furthermore, cities should probably not even bother attempting to claim tiles beyond it's workable radius until the radius itself has been claimed. It's rather annoying to have production dip because the city is arbitrarily overextended.
  
• Luxury systems seem to be working, the upper class have more reasonable demands nowadays.
• Urban employment systems seem to working fine, but maybe to the detriment of rural employment, as only roughly 20-33% of population is rural at the moment. I'm don't think the population is meant to be that urbanized in the classical-medieval period.

That said, it sounds like you're working on a fix for most of the above issues anyway in regards to the immigration and map tile employment systems.

Some more questions, as I can't seem to find anything concrete in your other threads or concept notes (and I'm not savvy enough to make sense of the code directly):

• What are the rules regarding land claims with this mod?
• What are the driving factors for migration? (It seems like food, employment and housing are part of it).
  
• Besides placing cities and working tiles, are there other ways as the player that I can encourage migration? (Do improvements help?)
• What determines the size limit of a map tile, or is that also dynamic?

Once again, if you have time. I'm trying to get a better understanding of these systems, so I can have a better idea of how the mod works, and thus a better idea of what isn't working as intended.

 

[Gedemon]

Thanks again, this helps me to organize the UI feedback.

Yes, currently working on the migration values, I've removed the population class distance factor when migrating from a city center to a plot owned to that city (there is still a global distance factor), which should help a lot with rural employment. There were also rural/urban factor related by eras that have been removed on the city side (they were linked to the exponential calculation) but will be restored at some point.

Factors are indeed Food/Employment and Housing, still tweaking those values. Security/Threat will be added at some point.

For plots "Food" is more a general sustainability value (MaxSize in the code), including plot's appeal.

For city owned plots that factor is also affected by the city rationing, as the city is requisitioning "Food" from plots.

Land claim is related to migration and ethnicity, it's something that I need to make place for in the plot's tooltips.

Basically, IIRC my own code, you claim a plot when you have 100 population of your ethnicity on it and no other major ethnicity is above that. The behavior will also change with eras and diplomacy if possible. There will be a front line feature (ie capturing plots with units, temporary or not) at some point.

Plot to plot migration follow the terrain, faster along rivers/road, slower through mountains, rivers, desert, jungle, ...

For a fast answer, I don't remember the exact effect of a Civilization culture output, but it will be applied to convert ethnicity on owned plots, at the moment the conversion only occurs in cities, without culture weight I think.

Improvements raise the output of a plot and so its migration's pull values: Employment factor for all improved resources and Sustainability factor for improvements adding to the Food yield. In my current dev version, "Housing" is the best value between max employment and max sustainability, it was only max sustainability before.

Working a plot has the most effect on migration, so it's also an indirect way to select in which general direction you want to expand your territory.

 

[DemonEmperor]

Some general feedback:

• Lack of administrative capacity has no maluses. I've been sitting at around 16 - 36%, and I don't think there's been any penalties for that
• Having a major debt doesn't seem to have any consequences. I've racked up multi-thousand, and tens of thousands of gold debt with no major consequences
• 
  
  
• I don't know if that behavior is also in vanilla, but my island city, that I expected to be able to work the wheat farms, can't work the wheat farms
• Outside of looking pretty, I'm not sure aqueducts actually do anything atm.

The below are balance concerns, but those can probably be safely put aside, as they're solveable by introducing civ-specific bonii tailored to their spawn locations ... which is still far away

• Rivers are *massive* growth multipliers - the extra food really helps. Desert is ironically great (not just because the city graphics match), but because floodplains are an insane source of bricks/clay, meaning you don't have to split population between material (input) production, and food production
• On a similar vein, If you spawn in pure jungle, you're kinda screwed. It'll take ages before you're able to get rid of it, and repurpose it into actual farms to grow, and the extra forests are also useless, because it takes *way* too long before you're able to build lumber mills
• If you spawn on tundra, you're also screwed, since it takes *literal ages* before you're able to build any farms

Future suggestions for consideration:

• Adding something that explains what the Roman Numerals above units mean. I think they have to do with total number of men in that unit? Unsure. I just know my infantry went I ==> II ==> III ==> XX
• Adding the ability to culture flip cities
• 
  
  
• Most of their population follows my culture - there should be an ability or something that flips their city over to me, especially since I've basically taken all of their territory at this point

 

[Gedemon]

Personally I would be fine with leaving select / 0 things in, as illustrated in my earlier population migration case, but yeah divide by 0s are generally useful for highlighting logic errors

In regards to the serializer, got a version working - no idea if it works on such cases or not. Going to start a new game with that foodstock/foodNeeds div check disabled to see if it handles it properly

The serializer has been tested to work on next turn (comparing initial to final tables after serialize/deserialize), and reloading an earlier turn. Have not tested past turn 10, so any exotic effects like NaN/nil/inf, as stated above, hasn't been tested yet.

File has been attached for reference.

thanks, complete new one, tested a bit, may handle more cases, but seems slower:

new one

GCO_CityScript: WARNING : Serialize and Save CityData timer = 0.6489999294281 seconds at line 799
GCO_CityScript: WARNING : Saving And Checking CityData timer = 0.99399995803833 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 1.2349998950958 seconds at line 799
GCO_UnitScript: WARNING : Saving And Checking UnitData timer = 0.5460000038147 seconds at line 799
GCO_CityScript: WARNING : Serialize and Save CityData timer = 0.64199995994568 seconds at line 799
GCO_CityScript: WARNING : Saving And Checking CityData timer = 1.0309998989105 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 1.1829998493195 seconds at line 799
GCO_UnitScript: WARNING : Saving And Checking UnitData timer = 0.53799986839294 seconds at line 799
GCO_CityScript: WARNING : Serialize and Save CityData timer = 0.61199998855591 seconds at line 799
GCO_CityScript: WARNING : Saving And Checking CityData timer = 0.98100018501282 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 1.1059999465942 seconds at line 799
GCO_UnitScript: WARNING : Saving And Checking UnitData timer = 0.52100014686584 seconds at line 799

old one (all "Serialize and Save" are under 0.5 seconds, which is the trigger for warnings, and all actions on UnitData are also under 0.5)

GCO_CityScript: WARNING : Saving And Checking CityData timer = 0.63800001144409 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 0.87400007247925 seconds at line 799
GCO_CityScript: WARNING : Saving And Checking CityData timer = 0.68099999427795 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 0.84600019454956 seconds at line 799
GCO_CityScript: WARNING : Saving And Checking CityData timer = 0.53600001335144 seconds at line 799
GCO_PlotScript: WARNING : Saving And Checking Map timer = 0.80100011825562 seconds at line 799

 

[Gedemon]

And thanks for the feedback !

Administrative Efficiency only affects city production of resources ATM, it will affect stability when that's introduced. The debt will also be a factor for stability, or maybe will generate more administrative cost.

As for Health it's only displayed for balance testing, while I introduce way to control it. Stability is also linked to Ethnicity and Separatist, and I'm not yet ready to code that over the current balance.

Aqueducts should affect the Health value, providing fresh water (removing the associated penalty when there is none)

Start Position will need an overhaul, that's clear. While I never restarted in vanilla civ6, I'm doing it a lot with the mod.

Not sure where to show the information about Military Organisation. Maybe adding the full effect in the related techs description ? but as that would be "hard coded text" (I prefer "generated text" while values are changing a lot from balance pass) that may have to wait.

The whole mechanism is discussed here:
https://forums.civfanatics.com/thre...ing-and-equipment.623838/page-5#post-14907928

Culture flipping is planned, it will be a part of revolution/rebellion once stability is coded.

 

[Gedemon]

If you need wood fast, you can use a builder charge to cut a forest near a city with a transfer route to your city requiring timber (with a carpenter somewhere in the path)

Dense Forest (give lot of wood) cut down to -> Forest (less wood) cut down to -> sparse forest (even less wood)

Forest don't grow back, keep some for when you'll get lumber mill.

 

[DemonEmperor]

Anyone have the generated harbor graphics bug out for them? One of mine is in the wrong position.

 

[DemonEmperor]

@Gedemon found a case of foodstock reaching 0 in a city. Full log is included for completeness, but relevant areas are around lines 159109 (only 3-4 mentions)

The exact changes I made are in this code snippet: (original is around line 7000, GCO_CityScript.lua)

-- Starvation
-- Todo : get values per population class from NeedsEffects instead of global values here
local consumptionRatio                        = 1
local foodNeeded                            = self:GetFoodConsumption(consumptionRatio)
local foodstock                                = self:GetFoodStock()

if foodstock == 0 then
    if foodNeeded == 0 then
        print( "DETECTED NAN" )
    else
        print( "DETECTED INFINITE PUSH" )
    end
elseif foodNeeded == 0 then
    print("DETECTED INFINITE PULL")
end

cityMigration.Pull.Food[populationID]        = foodstock / foodNeeded
cityMigration.Push.Food[populationID]        = foodNeeded / foodstock
cityMigration.Migrants.Food[populationID]    = 0

if cityMigration.Push.Food[populationID] > 1 then
    local motivationWeight = cityMigration.Push.Food[populationID] * migrationClassMotivation.Food[populationID]
    if motivationWeight >= bestMotivationWeight then
        cityMigration.Motivation[populationID]        = "Food"
        bestMotivationWeight                        = motivationWeight
    end
    local starving                                = population - (population / cityMigration.Push.Food[populationID])
    cityMigration.Migrants.Food[populationID]    = math.floor(starving * classesRatio[populationID] * math.min(1, migrationClassMotivation.Food[populationID]))
end

So, if that div by 0 resulted in saved table instability, can confirm new serializer is more robust, apparently at the cost of performance

 

[CalebR_300]

I don't know if that behavior is also in vanilla, but my island city, that I expected to be able to work the wheat farms, can't work the wheat farms

I think that's a vanilla behavior. I'm assuming another city claimed the continent first? If so, that city will still have possession of those tiles until they're manually swapped by the second city (even if those tiles are outside the first city's radius).
Swapping has to happen from the city outwards, but because sea tiles are not claimable in this mod, you can't swap your way to the wheat farms.

Anyone have the generated harbor graphics bug out for them? One of mine is in the wrong position.

Can't say I've had that happen, but I found out you can build coastal wonders on top of your harbours...

Speaking of funky tiles, why can't I place this aqueduct? Two adjacent river tiles and two mountains... (Might have something to do with glitchy mountains).

Some random thoughts:

• Taxation (civics) should be based on number of pops, not the percentage in the city. Doesn't make sense the upper/middle classes should pay less tax just because there are a lot of lower class folks about.
• A 'stockpile' project (literally build nothing) would be handy pre Pictogram tech, just so you can accumulate some material in the early game to help reduce the price (having low stock is why I was so broke a few games ago).
• I'm not sure if it's still a problem, but I remember way back, a city would cop a health penalty for having bad tiles (rainforest/marsh) within it's borders, but outside the workable radius. (Having a hard time replicating this, happen to be on really lush tiles atm).