Roadmap to Preview Version .2

[Gedemon]

As discussed in the development thread, the current priority is to provide a second preview version (the first version is incompatible with the Fall patch)

While waiting for the first expansion for Civ6 before pondering to update the mod for it or stick to vanilla, here are the points to work on:


- Mid-game slowdown:

The main issue here seems to be the mod's trade networks that are becoming massive once every civs on the map know each other and have a visibility on a lot of cities.

ATM a city try to connect to every other cities in its trading range, I'm going to try to limit the number of possible connections (closest cities coming first), and lower the ranges of the different trade routes. I'll keep the trader units as they are in the first tests.


- Naval upgrades:

The difficulty here is about units scale, while it's fine to have multiple cities building and stocking "equipment" like swords, war horses or tanks before sending them to units requiring them (or when training an unit), building and stocking battleships and aircraft carriers would be an important waste of resources, and an "unit" represent a single ship.

On the other hand, upgrading using gold like the vanilla system would shortcut the whole supply chain mechanism the mod is based on.

An alternative would be to allow the action of "upgrading using gold", but then the unit would be automatically send to the nearest harbor (or coastal city), with 0 health and averything in "frontline" moved to "reserve", and "locked" there until the required resources are transferred using the healing mechanism of the mod to bring it back to 100% health. That way you don't lose the unit's current resources (materiel & personnel), XP and promotions (whatever mechanisms we decide to use for that) but don't magically gain resources you don't have access to (like "Steel" or other "equipment" we may use to define a "Battleship" or an Aircraft Carrier)


- Promotions:

The problem here is technical, I'd keep the current mechanism if we could (with different promotions type, but we're talking on how to assign promotions to units)

For the organization level mechanism, I need to attribute temporary promotions to units, but to remove a temporary promotion, the only way I've found is to delete the unit and recreate it without the promotion(s) to remove. A literal overkill, but not so difficult to code.

The real issue is that while we can reattribute the XP "points" of an unit, we can't set the XP "level", which means that the unit start again at level 1 and will gain a new promotion when it reaches 15 XP points.

I don't have any easy-to-code solution here for short term development, I may either keep it as a "known issue" or disable promotions entirely.


- Balance on Cost/Income:

That's the most complex issue. I may change a bit the values to lower the speed of prices variations, but I don't think there is a point to balance the cost of "steel pikes", "Iron Swords" and "War Horses" while all those equipment are kind of placeholders until we have some complete unit lines

It's also something that will require a lot of work on the UI to give a good feedback (we'll get graphs at some points BTW, I want them), but what we can do now is fixing some limits to expense per turn using the policy mechanism (and a minimal UI feedback to show if an unit is not healing because of that kind of limit, or if a city can't produce something because the resource cost is too high relatively to the fixed limit)

And of course the "debt" being also a placeholder, it's not game breaking to lose a lot of money


- Supply chain for Air Units

That one is not hard, the code exist, it's just a matter of providing data (what building producing which resource/equipment for this or that air unit)

I may use that one to provide a (relatively) more complex chain for later air units (like separating the buildings producing electronic equipment / engines / fuselage and the one assembling the aircraft)


- Expand a bit the simplified units tree currently used

I'd like to have a look at "Moar Units" (which would then be a requirement at first) to include at least some upgrades to the "skirmisher" line (actually starting at the slinger, stopping at the ranger), add a rifleman unit to the "infantry" line (and maybe an upgrade that is not mechanized to the "infantry" unit), and add an unit between the Knights and the Cavalry.

For later development, Wolfdog's "Warfare Expanded" will be a great source too.


- Fuel Consumption /Rationing

That would be a bonus if I could include it.


- Bug fixes

That means testing all the above when available

 

[JonnyH13]

I think it would be best that you create 2 types of trade routes. Local automatic trade routes that form between cities less than a certain distance apart and can transfer all resources between them. And Actual trade routes that are created by traders and only trade finished products and luxury resources.

An easy solution for your naval problem is to use a system of "parts" to create a ship ie iron -> Steel --> Naval Part --> Ship(A unit, not a resource) (it can be more complicated than that with more types of resources like electronics and fuel). In fact when it comes to equipment larger than personal weapons (ie chariots, tanks, siege engines.) the more advanced the equipment is, the more parts and types of parts it takes to create them.

Also, you have already taken some steps into doing this, but how hard is it to make something like the "size matters" part of Civ 4 Caveman2Cosmos? All you need is to apply this to all units, have the appropriate strength values calculated, and the ability to change unit size manually and you basically have it.

When it comes to promotions, Unit synergy promotions would do nicely. Personally I think you can do away with unit per tile limit if possible and implement a logistics system and promotion synergy like Civ 4 Realism Invictus. I am of the opinion that the less hard limits imposed the more opportunities there are.

When it comes to the cost of equipment and resources, you need a standard unit. What does 1 unit of Iron equal to? A pound? A Kilogram? A ton? That is the first thing you need for all resources in order to determine what ratio of resources you need to produce this resource.

The next thing is supply and demand and how goods are traded, which is complicated to say the least. With those units established, what does each person need? How will they get it? This then goes into the problem of what defines a person's job, what does he produce? How many produce that resource? How will they trade it for another needed resource? What is the demand for their own goods and what it is worth in relation to other resources. Bartering is complicated (Thank God for Currency!) which then means once you reach a standard unit of currency, what will that currency be based off? Gold? What if you have not discovered gold? Cocoa Seeds? Grain? A complicated system of weights and measures (needs mathematics of course!) though you can possibly abstract this part though but then you run into the problem of a lack of inflation/deflation and an accurate market economy if you have a set form of currency. There are many design choices you must make here.

When it comes to a unit tree, it might be simpler to have a rock paper scissors sort of system between units with some units sitting in the middle of it all that are either support roles or require specific promotions to be specialized.

There are many choices you have to make and this path is long but keep on going! I think you can take inspiration from other complex mods for past versions of civ. If they can do that for the older games and make it run, I am certain you can make your mod run here. Keep up the good work!

 

[Knasp]

- Naval upgrades:

The difficulty here is about units scale, while it's fine to have multiple cities building and stocking "equipment" like swords, war horses or tanks before sending them to units requiring them (or when training an unit), building and stocking battleships and aircraft carriers would be an important waste of resources, and an "unit" represent a single ship.

On the other hand, upgrading using gold like the vanilla system would shortcut the whole supply chain mechanism the mod is based on.

An alternative would be to allow the action of "upgrading using gold", but then the unit would be automatically send to the nearest harbor (or coastal city), with 0 health and averything in "frontline" moved to "reserve", and "locked" there until the required resources are transferred using the healing mechanism of the mod to bring it back to 100% health. That way you don't lose the unit's current resources (materiel & personnel), XP and promotions (whatever mechanisms we decide to use for that) but don't magically gain resources you don't have access to (like "Steel" or other "equipment" we may use to define a "Battleship" or an Aircraft Carrier)

Having every single ship as a separate naval unit will also bring problems due to the scale. It seems to me that having naval units representing fleets, is the best solution to avoid having say 300 separate ships as individual units. But maybe the option of having separate units could be added when choosing City-production?
When you have a naval city you could choose either:

• Build a new naval unit (single ship?)

OR

• Build new ship(s) for an existing naval unit

A Civ in the classical era might have 2 naval units of Galleys and when choosing production for the naval city, you could choose between:

• Build Galley (new unit)
• Build Trireme (new unit)
• Build Galley (fleet 1)
• Build Trireme (fleet 1)
• Build Galley (fleet 2)
• Build Trireme (fleet 2)

Considering that most players/AI doesn't build that many naval units and that there are few warships for each era, there shouldn't be a problem with having too many options available. Although to dissuade spamming of separate units, you could award a tiny combat strength bonus to every additional ship in a fleet (compared to having the same unit as a single ship). Or possibly reduce the building time/upkeep cost of reinforcing an existing fleet to give the player incentives.You could also add the option of merging naval units, into a single fleet, to reduce the number of naval units.

Upgrading of naval units is tricky though. Transfering personnel and materiel to a new vessel should definitely happen. But you shouldn't be able to simply upgrade an existing bireme into a trireme, by simply paying some gold. The solution of moving the ship to harbour sounds better than gold upgrades.

On the other hand, if ships are treated as resources then each vessel could have its own "life cycle", and fleets would need to be replaced continuously (by naval cities). Ships that aren't replaced (due to lack of resources/supply) would instead mean that the sailors (personnel) of the old ship would stay in the naval city, and wait for a new ship to be built (while working with fishing/naval trade). Maybe you could have Sailors as a specific resource and then you could have several types of sailors (representing different levels of training and experience). Ships/fleets that see a lot of battle could thus level their sailors, meaning that the crews would become more experienced and they could be "stocked" in a naval city even if you aren't building new ships.

The wiki page on triremes illustrates the costs and labour involved, and I think it shows treating ships as resources to be a good idea:

Triremes required a great deal of upkeep in order to stay afloat, as references to the replacement of ropes, sails, rudders, oars and masts in the middle of campaigns suggest.[17][18] They also would become waterlogged if left in the sea for too long. In order to prevent this from happening, ships would have to be pulled from the water during the night. The use of lightwoods meant that the ship could be carried ashore by as few as 140 men.[19] Beaching the ships at night however, would leave the troops vulnerable to surprise attacks. While well-maintained triremes would last up to 25 years, during the Peloponnesian War, Athens had to build nearly 20 triremes a year to maintain their fleet of 300.[17]
[...]
The construction of a trireme was expensive and required around 6000 man-days of labor to complete.[27]

And also some interesting notes on naval warfare (at the time of the trireme):

In the ancient world, naval combat relied on two methods: boarding and ramming. Artillery in the form of ballistas and catapults was widespread, especially in later centuries, but its inherent technical limitations meant that it could not play a decisive role in combat.
[...]
Unlike the naval warfare of other eras, boarding an enemy ship was not the primary offensive action of triremes. Triremes' small size allowed for a limited number of marines to be carried aboard. During the 5th and 4th centuries, the trireme's strength was in its maneuverability and speed, not its armor or boarding force. That said, fleets less confident in their ability to ram were prone to load more marines onto their ships.
[...]
Whereas the average percentage of fatalities from a land battle were between 10-15%, in a sea battle, the forces engaged ran the risk of losing their entire fleet. The number of ships and men in battles was sometimes very high. At the Battle of Arginusae for example, 263 ships were involved, making for a total of 55,000 men, and at the Battle of Aegospotami more than 300 ships and 60,000 seamen were involved.[65] In Battle of Aegospotami, the city-state of Athens lost what was left of its navy: the once 'invincible' thalassocracy lost 170 ships (costing some 400 talents), and the majority of the crews were either killed, captured or lost.[65]

To sum up: Naval units should work the opposite way compared to land units. Constructing a naval unit should yield a ship (of a quality determined by shipbuilding tradition, tech, naval trade, presence of experienced sailors). In order to crew the ship you'd use regular personnel or possibly a new category of personnel called sailors. Sailors would represent personnel of varying experienced levels, which could be treated like weapon resources. So a ship crewed by personnel could be replaced by sailors. These experienced personnel can only be obtained by using your naval units (navigating, fighting etc) or you could obtain them by buildings in your naval city (or be gained from naval traders, fishing boats etc).

Later eras could reduce the number of ships (or even get rid of fleets), simply due to ships being bigger, made of steel, requiring fuel, and being more costly to build and maintain. Removing fleets in a later era could leave room to allow a greater variation of individual naval vessels, like battleships, aircraft carriers etc.

 

[Gedemon]

Thanks for the suggestions.

I was referring to the "battleship" or "aircraft carrier" for "single ship" units, earlier units would represent multiple ships. Still unsure if I should represent the number of ship like for land units equipment, but I think I'll go for a triggered by gold upgrade followed by an automated transfer to an harbor to be refited.

Specialized personnel may come later, not a priority, but it's a possibility.

I can't do merging/splitting with the current modding tool (cf the old discussion about new actions and the problem of broadcasting them for other players in MP in the main thread, time taken by implementing a new UI for that, and finally coding the AI to use it)

 

[gaborpesti]

I think even late game "Aircraft carrier" and "Battleship" can be a material and 2 of them compose an on map unit: Usually they operated in divisions of 2 and did not have 100% readiness anyhow.
Probably aircraft carrier as unit should also need planes at least for CAP duty. Bombers probably better modelled as a moving airfield - though I never reached this stage of the game, therefore do not know how planes are modelled in vanilla, despite 300+ hour playtime logged on civ VI in steam....

It would be even better if an on map ship unit (SAG - surface action group) would not only require 2 capital ship but a few escort as well.

I dislike the idea to use "Steel" as equipment for it. A ship as it is not upgradable, its unit is.
Though there is a merit in it as well. In RED WWII using material for repair kind of worked. But in that case it should absolutely not be upgradable....

 

[Gedemon]

Another possibility (as we're going to have units with limited "lifespan" when implementing non-standing army units) is to effectively completely remove upgrades for naval units and give them a limited lifespan (while lowering production cost, a least for earlier units)

That or obsolescence at some techs, combined with a decay rate.

No AI still using galley alongside submarines, no circumnavigation by a galley after a 4000 years exploration trip.

A quick draft for a decay/obsolescence mechanism:

- naval units require equipment or resources only during the construction phase (ie wood/steel/...)
- active naval units lose a small bit of materiel (and so health) each turn
- obsolete units does not receive materiel anymore from supply line (and will slowly die, personnel will be send back to nearest harbor/city on unit's death)

Now that call another mechanism to handle experience at the personnel level, not the unit level as Knasp has started discussing.

 

[Gedemon]

Latest release should have fixed some early game breaking bugs, but my latest auto play got stuck at turn 426/750 (1830 AD)

Seems to be an AI that can't end its turn, the game is not frozen (when it happens outside of autoplay you can still look at the UI and enter city screens or other reports) it's just that a turn never ends.

No idea what's causing it yet, but it seems to be always triggered around this era, maybe something I've removed from the base game but that the AI is still expecting to use... If that's the case, it's a bug that would illustrate the difficulty of neutralizing vanilla features that are not needed by the mod (and explain why I want to take the time to ponder the potential transition to R&F, if we don't get the source code)

 

[heinous_hat]

Seems to be an AI that can't end its turn, the game is not frozen (when it happens outside of autoplay you can still look at the UI and enter city screens or other reports) it's just that a turn never ends.

I've seen this same behavior at least once. I'm not doing autoplays, but in the last game where I made it to the modern era, I did have an unending AI turn cycle. UI was still accessible, etc.

Step through debugging would be nice, wouldn't it?

 

[Lacero]

I had a look at the route finding code in github. It looks like it could be a bit faster, but you'd still need to limit the amount it runs.

It doesn't seem to have an early out? Like if you only care if a route is shorter than one you know you can stop looking at a route once it grows bigger. Maybe I'm looking at the wrong bit.

This looks like a one man project, but if that's something useful for you I could give it a go? Playing with the github release was pretty fun once I realised I could ignore debt

 

[Gedemon]

I had a look at the route finding code in github. It looks like it could be a bit faster, but you'd still need to limit the amount it runs.

It doesn't seem to have an early out? Like if you only care if a route is shorter than one you know you can stop looking at a route once it grows bigger. Maybe I'm looking at the wrong bit.

This looks like a one man project, but if that's something useful for you I could give it a go? Playing with the github release was pretty fun once I realised I could ignore debt

I can use help on coding, especially on optimization, but from the look of things, I don't think that the other parts I'd like to work on for the preview v.2 will be ready before the R&F expansion is out, and the first civ5 expansion was the time of the DLL source code release IIRC.

So it may be safer to change my initial plan and wait to see if it's the same for civ6 before trying to change parts of the Lua code that would surely be converted to C++ ASAP

On the other hand, this is just optimistic speculation, for all we know (IE nothing), the DLL could be released in two years or never, so far Firaxis has made no real promises about it.

 

[Lacero]

OK. February isn't far away, and this should all be c++ you're right.
I'll see how things look after release.

 

[Gedemon]

About the Balance on Cost/Income, I have a few suggestions that may not requires a lot of coding.

The easiest would be to test with cost going down faster than going up, and give material a lower global cost.

There is also the possibility to create separate budget, in a way similar to the debt: allow unlocking a military and a urban budget.

For example a policy would reserve a percentage of your income each turn for the military, and all military expense would be taken from that budget first and from your treasury second, with a discount.

Then there is the question on how to limit production/reinforcement (assuming we're setting a variable limit to the debt).

With the current code, I may be able to determine the available gold for production and reinforcement (from budget + treasury + available debt), and share the part for production between cities then limit production in a city when it has reached its allowed portion of gold (that portion could be relative to the cities sizes), but that would be more difficult for units the way resources are reserved for them during cities "do turn".

I could either do a first pass on units to get an estimation of the cost, then sort the units in a list and iterate it unit by unit and stop when the portion allowed for reinforcement has been reached (maybe with a turn timer on units to reinforce those who where not reinforced the previous turns first) or make a global estimation and reinforce units only if we have enough gold for all of them. But partial reinforcement for each units would require too many changes to the code.

 

[Gedemon]

In the latest update, I've reduced equipment and materiel cost, maybe too much, waiting for feedback.

Still for the economy, I've also added a consumption from upper class population for luxuries (create demand -> increase prices -> higher income on export/import taxes)

I've tweaked parts of the diplomacy system to (try to) give more sense into it and artificially create allied/opposed groups.

When allied defensive pact are now automatics, but declaration of wars are also shared, meaning you accept that risk when entering an alliance with someone.

Defensive pact are still available, but earlier, at friendly relation. Research Agreements are buffed but require Alliance.

All those diplomatics changes need testing. A lot.

Mid-game slowdown is still an issue, and we didn't get any word on the source code, so I guess we'll have to do without it for at least another year. I'm afraid that I'm going to hit a development wall way before that because of the increasing compexity in debugging and that slowdown... I'm afraid of adding the proposed population system to the mix...

I can use help with the code, I plan since some time to switch to A* for the pathfinding (allowing supply to units to take into account terrain movement points), but for the trade routes following roads or on sea tiles (ie same movement cost on each tile, but limited in range) I don't know it it would be faster than @whoward69 (hi !) code.

I'm already using an implementation of A* for the river routes (as I had no idea on how to adapt the actual code to follow hexagons edges)

There may be better way (and more efficient !) to handle the trade network that what I've done, and I'm sometime lost in my own code because of the requirements (for example sharing the available resources between all cities demanding it, not giving it sequentially meaning some cities will get more than others)

So yes, that's still a problem, help welcome.

I may also add some kind of upgrade system for buildings (ie a later building replacing multiple early building) to the todo list for the preview version, as the production screen is now overcharged starting mid-game.

 

[whoward69]

Waves!

(I wander back on these forums occasionally, but my serious modding days are done)

 

[heinous_hat]

Mid-game slowdown is still an issue, and we didn't get any word on the source code, so I guess we'll have to do without it for at least another year. I'm afraid that I'm going to hit a development wall way before that because of the increasing compexity in debugging and that slowdown... I'm afraid of adding the proposed population system to the mix...

Not that it's any direct indication, but first source release for Civ5 came alongside the seasonal patch following G&K. So maybe we'll be gifted in a few months or so, if there's anything left of that routine.

Anyway, lots of additions here to look at

 

[heinous_hat]

(I wander back on these forums occasionally, but my serious modding days are done)

What?!? No UI pick'n'mix?

Best of luck with new endeavors

 

[Gedemon]

Waves!

(I wander back on these forums occasionally, but my serious modding days are done)

Well, I had to try !

Not that it's any direct indication, but first source release for Civ5 came alongside the seasonal patch following G&K. So maybe we'll be gifted in a few months or so, if there's anything left of that routine.

Anyway, lots of additions here to look at

Maybe... maybe not... let's plan without...

 

[Knasp]

There may be better way (and more efficient !) to handle the trade network that what I've done, and I'm sometime lost in my own code because of the requirements (for example sharing the available resources between all cities demanding it, not giving it sequentially meaning some cities will get more than others)

Hmmm, so it seems that one way to increase performance would be to simplify the trade/commerce/supply system? I can offer some suggestions of design, that might limit the need for processing.

The list below presents suggestions in no particular order. Some of these could possibly be combined.

• Limit the number of trade routes to each foreign Civ to only 1 city of theirs. The one picked being the closest/wealthiest/biggest foreign city of that Civ. The benefit I imagine would be that the code wouldn't need to check connections to each neighboring foreign city every turn. Hopefully this system could result in some cities becoming trade hubs, since they would have the highest demand of goods to trade with, meaning that your other cities would export to these hubs.
• Arbitrarily limit the number of trade routes / connections, to other cities. Maybe 2 for standard cities and 3-4 for Capital cities? This would reduce the need to check every route each turn right? Maybe combine it with suggestion below?
• Set a distance limiter for checking city connections, for e.g. 6 tiles for land-routes, 18+ for naval routes? (numbers are just for example). Then you could have a "static" list of cities and their distance to other cities noted, and thus you wouldn't need to check all possible routes to all cities. Of course, the limits to city connections could be influenced by certain buildings/policies/development etc.
• Check city connections and resolve trade internally (between your own Civ's cities) before resolving foreign trade routes?
• Cities only buy resources that they don't produce themselves, even if their production is low. (Though food resources and such might be an exception).
  
• Cities only sell resources that exceed their stock.
• Only check and award supply lines to your military units within your own, or allies/vassals borders. Add a supply unit (carriage train) that you need if you want your military units to resupply outside friendly borders.
• Instead of each city having a "stock" of resources, only the production (rate) of a city is noted. Each turn, every Civilization's production are summarised into an abstract national inventory. From which redistribution and/or prices are determined with respect to internal demand/supply. Then national inventories can be compared with each other, and only resources which are demanded and lacking are imported from another Civ. The following turn, each city could then display (through mouse-over UI) the resources that were supplied or that were lacking, the previous turn.
• Combine cities stocks of resources if they're close to each other? (Forming entities)
• Similar to the previous two suggestions, you could create economic blocks, meaning that cities that are close to each other and connected, regardless of nationality, are grouped together. These kinds of groupings could be based on the cities' unique resources and stocks, how well they complement each other. Thus each economic block would be semi-static, and trade be limited to within. Thereby reducing need for checking connections to other cities.
• Smaller cities could be limited to only exports, and bigger cities would never export to smaller cities. Basically this would represent economic purchasing power of big, important cities. In effect you could have an economic hierarchy, where goods are concentrated. It kinda makes sense since merchants want to sell their goods at the highest price and other small cities wouldn't be as wealthy, being able to afford the same luxuries. There would be competition between smaller cities of getting to supply the capital, or bigger population centres.

 

[Gedemon]

Yes some good suggestions here, in fact some of them are already implemented

I think it would help (starting by me !) if I describe the current code, to see what could be improved/modified

Here is how a turn is currently handled for a city at the beginning of a player's turn:

1. first pass, for each of the player's city:
   • set the value of food rationing if required: based on previous turn supply, current turn demand and current stock
   • get the list of units linked to the city and their requirement: no path checking here, it's done at the end of an unit's turn (and would require improvement BTW as it's only looking at the closest city)
   • collect resources on the city tiles and update stock: this one could be optimized when checking the path for sea resources outside the civilization territory (by marking unreachable plots), it's one of the function that get reported in the logs
   • recruit personnel : convert a % of population to personnel
   • get the "food" from the city's yield (that's the vanilla yield from buildings/tiles calculated by the game) and add it to the "food" resource stock
   • get and fulfill population needs:
     • check for available food for each class (based on rationnement), update death rate, consume food and update stock
     • check for housing for each class and update birth rate
     • check for available luxuries, update social stratification min/max % for upper class, consume luxuries and update stock
   • create and consume resources in industries (including creation of the "food" resource from wheat, rice, etc...)
   • check construction progress on the city current item (buildings, units, ...)
     • calculate production efficiency based on available resources
     • calculate resources needed based on efficiency
     • remove resources from stock, add them to the current production item temporary stock
     • neutralize this turn production (by removing it from the current item), it will be added later (calculated from efficiency) in the code after the production has been updated (to prevent a production item to be completed even when there is not enough stock left)
     • update player's treasury based on the cost of the resources moved to the current production item stock
   • reinforce units
     • calculate the global number of required and available resources for all linked units
     • calculate and share each resources between units requesting it
     • update city and units stock
     • calculate global cost and update treasury
     • send all surplus from units to city, update city and units stock
     • calculate income from selling unit's surplus to the city and update treasury
2. second pass, for each of the player's city:
   • update internal trade routes (another function that my debug code is reporting in the log for the time it takes):
     • get the list of the other cities, sorted by distance with the current city
     • get the number of available land/sea/rivers internal trade routes for the current city (actually it's one of each)
     • for each of the city in the list, starting by the closest city, until there is no available route left
       • if the other city is in the "out of reach" table (this table is saved with the game, it was updated on reloading a game in the initial implementation, but that was taking more than 5 minutes)
         • if the number of turn since the last update is > distance / 2 then mark the city to get an update next turn (this means that the path to an unreachable city at 10 tiles is tested every 5 turns), a possible optimization is to make two list: one for cities that are out of reach because of geography, not updated, and one for cities out of reach because of temporary events (like blocade), to update
       • if the other city is not in the "out of reach" list:
         • if the other city is in the trade route table of the current city (another saved table, including the path between city):
           • check which city has created the route (if it's the other city, then the route doesn't count for the current city limit, else instead of a "network" we'd get "pairs")
           • check if we need to update the path
             • if the number of turn since the last update of the route is > distance / 2 then mark the path to be updated
             • if there is an obstacle on the current path (like an enemy unit), then mark the path to be updated
           • if the path is marked for update, or if the other city is not in the trade route table of the current city:
             • for each route type, if the current city has not reached its limit for that type and distance < max route length for that type then try to get a path
           • if there is at least one type of route available:
             • keep the most efficient in the route table (efficiency, path, ...)
             • update the number of route left for that type
             • get the resources required by the other city and available in the current city:
               • for each resource that can be exchanged:
                 • get the number requested by the other city and the stock available in the current city, abort if the other city has more than the current city (need to be changed to percentage of full stock as this check doesn't allow small city at full stock to transfer to big city at low stock)
                 • get the transport cost, add it to the current city cost and check if total cost > the other city cost
                 • if an unit linked to the other city is requiring that resource for reinforcement, mark it as a priority and ignore the cost check
                 • if the other city is producing an item (units, buildings, ...) that requires that resource, mark it as a priority and ignore the cost check
                 • if the other city has not that resource in stock at all, ignore the cost check (but do not mark it as a priority)
                 • add an entry for the other city requirement in the current city transfer table, to be used in the third pass.
           • if there is no type of route available, add an entry for the other city in the "out of reach" table of the current city
3. third pass, for each of the player's city:
   • transfer resources to all other internal cities, now that all of them have made their request after servicing industries and units
     • get the current city transfer table, sorted by trade route efficiency
     • for each requested resource
       • get the total stock available for transfer (this is a percentage of the current city stock)
       • if there is a priority request for that resource, make a higher percentage of the current city stock available for transfer, but ponder it if the resource is also a priority for the current city (to do: change the priority tag from a Boolean to a number)
       • calculate and share each resources between cities requesting it, share resource with priority requests only between cities that have marked that resource
         • update each city stock and the other city local cost (based on the number transferred, transport cost and each city local cost, the cost in the current city will be updated in another function, based on all supply/demand)
   • update the export routes to other Civilizations cities in a similar function than the one to update internal trade routes, but using diplomatic relations to filter available resources, and without priority (and this is also a function reported in the log for the time it takes)
   • export resources to other Civilizations cities in a similar function than the one to transfer resources to all other internal cities, but calculating export/export income taxes
4. fourth pass, for each of the player's city:
   • city growth:
     • calculate death rate and birth rate for all population class, and update population values
   • social class stratification:
     • calculate min/max values for each class, based on buildings (Palace, Housing) and needs (currently only Luxuries)
     • move parts of the population from one class to another, depending of the calculated min/max
   • set city wealth value, based on the current social stratification
   • update city size based on total population
   • heal city defense
   • handle surplus:
     • convert personnel to population if required
     • destroy half of the obsolete equipment stock (a recycling building may be added at some point)
     • destroy all other surplus
     • place and remove "unlocker" buildings that are dummy buildings used to control what a city can or can't produce

One of the problem with the current implementation of the trade route networks with a limited number of routes is that it can generate separate networks in a same civilization, with resources not transferred between network A and B.

And Lacero is right, an early exit is surely required in the pathfinding function.

 

[Knasp]

Thanks for the summary! I've been trying to wrap my head around it all, and I don't have experience with the game's code, so please excuse my ignorance. I have some questions and suggestions below:

second pass, for each of the player's city:

• update internal trade routes (another function that my debug code is reporting in the log for the time it takes):
  • get the list of the other cities, sorted by distance with the current city
  • get the number of available land/sea/rivers internal trade routes for the current city (actually it's one of each)
    
  • for each of the city in the list, starting by the closest city, until there is no available route left

Is it limited to 1 trade route of each type, per other city? Or just 1 route per type, regardless of destination?

Trade route hierarchy?
Historically, transportation by ships would almost always be a lot faster and cheaper than going by land, until maybe railroads / motor vehicles in the late 19th-20th century. A trade route going by rivers or coastal waters, should always be prefered over land routes, and therefore it doesn't make sense to restrict trade routes to one of each type.

As an example, in the Roman Empire (c. 30 BC - 476 AD), you could transport 100 tonnes of grain over 5000 km by sea with just a 20 man crew, cheaper than you could move the same weight only 110 km by land (using 150 wagons drawn by oxen). The average speed according to the same estimate was 15 km / day for wagons, and 100 km / day for ships.

Always shortest route?
If we implement such an hierarchy, then you'd only need to calculate paths once (when founding a city), and always go by the shortest route (for each type?). A city with a coastal trade route that gets blocked, will not risk sending their ships around the enemy (looking for a new longer path of the same type), rather they would look for alternative destinations or try to trade by a less effective and more safe alternative, such as using a land route. In which case, you could attempt to use the other route types, only if the shortest trade route is blocked.

Keeping track of trade route tiles and units?
Would it be draining performance to keep a list of current trade route tiles, and if any units occupies these tiles? For e.g. is there an enemy unit on tiles x-y?

Or how about keeping and updating a list of the current location and owner for every unit, and then you'd run a check only on units that are standing on trade route tiles, or only check units whose owner is at war (with anyone). Barbarian units would potentially have to be checked a lot, if they wander around.

If trade route paths are determined only once (upon founding of a city), then you could record those tiles in a table. And checking just those tiles for enemy units should be simple right?

Start with the biggest cities?
The problem remains on how to limit trade routes and checks, and that's a tougher nut to crack. Maybe start each run with the biggest cities (presumably having the most wealth) and let them establish trade routes first. If you can list cities by their unique resources/goods and production rate, then you could prioritise trade with cities that can supply the most of demanded goods and unique luxuries.

Such a system might mean a lot of trade routes going to the capital cities, but you'd need to stop the code from running through the minor cities without any luxury/prioritised goods. If a big city trades with a smaller city, and the resulting trade leaves the smaller city without goods to trade, then you'd want to skip the trade route check of the smaller city, since it no longer has any surplus to trade. Maybe there should be a cut off point where the current city's, or the other city's, stock and/or production rate is too low, to merit trade.

Hopefully, such a top down system could match cities together, and then you could skip checking trade routes for smaller cities (that have nothing unique/prioritised to trade). Of course, all these examples are more or less based on bartering. If you want trade where currency could be used in trading, then you'd need to establish a cut off point where the coins available and/or the stock is too low to merit trade.

Slower trade? Less interactions per turn?
Regarding resource transfers, supply etc. It seems that having instantaneous resource transfers every turn is requiring a lot of processing, and I'm not sure it's necessary? I mean, if instead transportation was slower, maybe taking a few turns to accomplish its delivery, then you wouldn't need to redistribute resources every turn, and therefore you wouldn't need to check/update trade routes every turn.

Trader units - performance drain?
If taking the slow road above, you could have automatically spawned trader units pop up and then let those units carry the resources (in their stock) to their destination. I suppose that sadly this would require even more processing, though it would be nice to have units to plunder. Another benefit of a slow transportation/trade system would be that you could determine the price upon delivery or before dispatch. Though it could be difficult to determine when and how often trade missions should be conducted, but maybe it could be tied to total population, demand and stock (%) of the city.

In this system, you could have both the sending and receiving city to commit to the transaction beforehand, for e.g:

Spoiler Example :

City A needs 200 tonnes of grain (estimated from previous turns lacking supplies, or predicted future needs).
City B has 100 tonnes of grain stored/surplus (maybe produced in the current or previous turn).
City A commits to importing 100 tonnes of grain from City B in return for other resources or valuables that City B needs.
City B dispatches their trader unit.
City A will now look to import the lacking food from other cities, but it will only buy/request 100 tonnes, counting on their trading partner.
Upon delivery, City B receives the gold/resources in return, and carries them home to the city of origin.

Slow vanilla trading units?
Of course, you could bring back vanilla trader units, and let the player choose which city to trade with, showing the resources that are available and demanded by both cities (requiring more UI programming...) including estimates of how much could be traded and gained from such a mission. Unlike vanilla however, these trader units would actually carry their stock, deliver and return.

End of turn script?
Is there any difference between start and end of turn? Can you run code when clicking to end the turn, before starting the next turn, or is there no difference?

Manual supply (to blocked cities)?
With an automatic supply/trade system, you'd probably want to have the possibility of trying to bring supplies around an enemy blockade. The simplest solution in my mind, would be to add the ability to build supply units, for whcih the city will transfer food to the temporary production stock (just like weapons for a unit).
Upon completion, the unit can be freely maneuvered and escorted to any city (where goods can be delivered).

I guess that the A.I. couldn't handle the freedom well. So instead, the A.I. Civs could get a special trader unit, that only has blocked/out of reach cities as it's possible destinations.
Unlike vanilla traders, this unit would try to move around enemy military units or attack them, though its movement would be completely automated.