Roadmap to Preview Version .2

Is it limited to 1 trade route of each type, per other city? Or just 1 route per type, regardless of destination?
1 route per type, regardless of destination

Trade route hierarchy?
Historically, transportation by ships would almost always be a lot faster and cheaper than going by land, until maybe railroads / motor vehicles in the late 19th-20th century. A trade route going by rivers or coastal waters, should always be prefered over land routes, and therefore it doesn't make sense to restrict trade routes to one of each type.

As an example, in the Roman Empire (c. 30 BC - 476 AD), you could transport 100 tonnes of grain over 5000 km by sea with just a 20 man crew, cheaper than you could move the same weight only 110 km by land (using 150 wagons drawn by oxen). The average speed according to the same estimate was 15 km / day for wagons, and 100 km / day for ships.
The routes efficiency is dependent of the type, and the code set the most efficient route available, but the problem is that road and river routes are much faster to get than sea routes, if we allow 3 undefined routes per city, all coastal cities will use 3 sea routes as they are far more efficient.

Always shortest route?
If we implement such an hierarchy, then you'd only need to calculate paths once (when founding a city), and always go by the shortest route (for each type?). A city with a coastal trade route that gets blocked, will not risk sending their ships around the enemy (looking for a new longer path of the same type), rather they would look for alternative destinations or try to trade by a less effective and more safe alternative, such as using a land route. In which case, you could attempt to use the other route types, only if the shortest trade route is blocked.

Keeping track of trade route tiles and units?
Would it be draining performance to keep a list of current trade route tiles, and if any units occupies these tiles? For e.g. is there an enemy unit on tiles x-y?

Or how about keeping and updating a list of the current location and owner for every unit, and then you'd run a check only on units that are standing on trade route tiles, or only check units whose owner is at war (with anyone). Barbarian units would potentially have to be checked a lot, if they wander around.

If trade route paths are determined only once (upon founding of a city), then you could record those tiles in a table. And checking just those tiles for enemy units should be simple right?
I'm already keeping track of the current routes, so it's only the length of the route that need to be tested for enemy units before recalculating, but keeping track of the most efficient route between two cities is a good idea, and try to update a current route to that one first (after being changed because of enemy units) will make things faster. Still have to be updated from time to time for sea routes (because of canal cities that can be added and, obviously, roads)


Start with the biggest cities?
The problem remains on how to limit trade routes and checks, and that's a tougher nut to crack. Maybe start each run with the biggest cities (presumably having the most wealth) and let them establish trade routes first. If you can list cities by their unique resources/goods and production rate, then you could prioritise trade with cities that can supply the most of demanded goods and unique luxuries.

Such a system might mean a lot of trade routes going to the capital cities, but you'd need to stop the code from running through the minor cities without any luxury/prioritised goods. If a big city trades with a smaller city, and the resulting trade leaves the smaller city without goods to trade, then you'd want to skip the trade route check of the smaller city, since it no longer has any surplus to trade. Maybe there should be a cut off point where the current city's, or the other city's, stock and/or production rate is too low, to merit trade.

Hopefully, such a top down system could match cities together, and then you could skip checking trade routes for smaller cities (that have nothing unique/prioritised to trade). Of course, all these examples are more or less based on bartering. If you want trade where currency could be used in trading, then you'd need to establish a cut off point where the coins available and/or the stock is too low to merit trade.
We'll need the routes to/from capitals for stability at one point or another. A centralized network was my first idea, we could go back to that. I'd need a step by step description for multiple center in the same civilization.

Slower trade? Less interactions per turn?
Regarding resource transfers, supply etc. It seems that having instantaneous resource transfers every turn is requiring a lot of processing, and I'm not sure it's necessary? I mean, if instead transportation was slower, maybe taking a few turns to accomplish its delivery, then you wouldn't need to redistribute resources every turn, and therefore you wouldn't need to check/update trade routes every turn.
Not sure that the calculation are causing the slowdown, but I'm approaching turn 250 on my current Enormous map, I'll try to put more timers in sub-functions to determine where it is exactly.

And I would have to re-code/re-balance the whole resource system, so I hope it's not that.

Trader units - performance drain?
If taking the slow road above, you could have automatically spawned trader units pop up and then let those units carry the resources (in their stock) to their destination. I suppose that sadly this would require even more processing, though it would be nice to have units to plunder. Another benefit of a slow transportation/trade system would be that you could determine the price upon delivery or before dispatch. Though it could be difficult to determine when and how often trade missions should be conducted, but maybe it could be tied to total population, demand and stock (%) of the city.

In this system, you could have both the sending and receiving city to commit to the transaction beforehand, for e.g:

Spoiler Example :

City A needs 200 tonnes of grain (estimated from previous turns lacking supplies, or predicted future needs).
City B has 100 tonnes of grain stored/surplus (maybe produced in the current or previous turn).
City A commits to importing 100 tonnes of grain from City B in return for other resources or valuables that City B needs.
City B dispatches their trader unit.
City A will now look to import the lacking food from other cities, but it will only buy/request 100 tonnes, counting on their trading partner.
Upon delivery, City B receives the gold/resources in return, and carries them home to the city of origin.


Slow vanilla trading units?
Of course, you could bring back vanilla trader units, and let the player choose which city to trade with, showing the resources that are available and demanded by both cities (requiring more UI programming...) including estimates of how much could be traded and gained from such a mission. Unlike vanilla however, these trader units would actually carry their stock, deliver and return.
AFAIK automated units are impossible with the current modding access.

End of turn script?
Is there any difference between start and end of turn? Can you run code when clicking to end the turn, before starting the next turn, or is there no difference?
Some computers won't like to process every players at once at then end of the game's turn, better to process each player during their turn, start is easier from a coding perspective with the available events.

Manual supply (to blocked cities)?
With an automatic supply/trade system, you'd probably want to have the possibility of trying to bring supplies around an enemy blockade. The simplest solution in my mind, would be to add the ability to build supply units, for whcih the city will transfer food to the temporary production stock (just like weapons for a unit).
Upon completion, the unit can be freely maneuvered and escorted to any city (where goods can be delivered).

I guess that the A.I. couldn't handle the freedom well. So instead, the A.I. Civs could get a special trader unit, that only has blocked/out of reach cities as it's possible destinations.
Unlike vanilla traders, this unit would try to move around enemy military units or attack them, though its movement would be completely automated.
no automated units, sadly, we could do that with civ5 without the DLL (my convoy units in the WWII mod were exactly that, but also automated for the player, you had to protect/escort them with your other units)
 
1 route per type, regardless of destination
The routes efficiency is dependent of the type, and the code set the most efficient route available, but the problem is that road and river routes are much faster to get than sea routes, if we allow 3 undefined routes per city, all coastal cities will use 3 sea routes as they are far more efficient.
Excellent, that would make coastal cities far more important, which makes sense since they're having access to more cities and probably access to more unique/exclusive goods*. You could choose to prioritise river routes over coastal/oceanic (I don't know which of these are most efficient really). If you want to, you could let each city start with 1 land trade route, connected to the capital city, or the closest friendly city.

Instead of a limit of 1 trade route per type, or 3 per city, why not just add "trade buildings" and award 1 trade route of a specific type, per building?

For e.g:
  • Riverine traders (requires city to be adjacent to river) = Gives 1 river trade route
  • Maritime traders (requires coastal city/harbour) = Gives 1 coastal/ocean trade route
  • Caravansery (requires adjacent land tile or city?) = Gives 1 land trade route
Just like housing, you'd be able to construct a higher tier trade building to get an additional trade route of the same type. However, it shouldn't be more expensive to get an additional trade route of the same type, instead of a different type. Land routes are inherently less effective, so you wouldn't want to build a caravansery unless it gave access to a unique/special goods that are unobtainable from naval routes. Of course, you could always limit the number of trade routes (and trade route buildings) by the city size.

On the other hand, a possible benefit of land routes would be to get roads, which would make it easier to establish effective systems of couriers (messengers on horseback), since such couriers could be faster than ships.

If possible, it would be good if river trade routes could be combined with naval routes, meaning that you could trade between cities on different rivers, connected by a common sea. But since you say that sea routes take longer time to get, this suggestion would require even more processing. Also, this depends on the ships used, since only flat-bottomed ships with a low draft can (safely) navigate shallow rivers, while they would be disadvantaged on the sea where you'd want more freeboard. One solution to this would be to only let the city that is furthest downriver trace coastal routes from the mouth of the river to coastal cities/harbours.
I'm already keeping track of the current routes, so it's only the length of the route that need to be tested for enemy units before recalculating, but keeping track of the most efficient route between two cities is a good idea, and try to update a current route to that one first (after being changed because of enemy units) will make things faster. Still have to be updated from time to time for sea routes (because of canal cities that can be added and, obviously, roads)
Yes, I suppose that new sea trade routes would need to be updated every time a new coastal city / harbour is founded. When it comes to choosing which city to trade with, for coastal/riverine trade routes at least, proximity isn't as big of a deal as it is for land routes. Rather, it is the resources that your city demands, and the resources that the other city wants, that should determine the destination.
We'll need the routes to/from capitals for stability at one point or another. A centralized network was my first idea, we could go back to that. I'd need a step by step description for multiple center in the same civilization.
This would depend on whether you choose to introduce trade buildings (suggested above), and if so you could simply start with the city that has the most trade routes (being a commercial center), and if tied, you can go by total stock, then by population size, and so on. I'm not sure I can suggest something that would be easy to code or more efficient to execute than the current code, but I'll share my thoughts anyway:

You could either restrict trading to internal trade first, and then do a second run with external trade (like the current system), or you could resolve all trade simultaneously. For each turn you'd need to check which cities that are connected to the trade network, and exclude those cities whose access are dependent on a blocked route. The excluded cities would form a separate trade network, until the blockade ends in a following turn. Within each network (whether internal cities first, or all cities combined) you could use one of the two following ways of matchmaking:
Either go by:
a) available goods (ability to supply) or,
b) purchasing power (ability to consume).

a) Ability to supply:
1. Start by summing up every city's total stock, all goods combined. At the same time, sum up the total stock of goods that the cities demand*, i.e. the amount of goods that each city has capacity to store and use.
2. Start with the city that has the highest total stock, all goods combined, resolve ties by starting with the city that have the least goods demanded*, otherwise pick the one with largest total population.
3. Match the current city with the other city that have the greatest demand* of its goods. And mark those types of goods to be transferred as "locked" for further trade in both cities. (E.g. Wheat is locked in city A and B)
4. Continue matching with other cities until all goods of the city are "locked" or can't be traded.
5. Then go on to the next city, that has the second highest total stock (minus locked goods), and continue matching cities.
6. Once all cities and their goods are matched, or no trade routes are available, then:
7. Transfer goods and update stock.

b) Ability to consume:
1. Start by summing up every city's total stock, all goods combined. At the same time, sum up the total stock of goods that the cities demand*, i.e. the amount of goods that each city has capacity to store and use.
2. Start with the city that has the highest total demand, all goods combined. Resolve ties by starting with the city that have the highest population or goods available for trade.
3. Match the current city with the other city that have the greatest supply of goods demanded. And mark those types of goods to be transferred as "locked" for further trade. (E.g. Wheat is locked in city A and B)
4. Continue matching with other cities until the city's type of goods are all "locked", unavailable for trade OR if the total demand of goods is now lower than the next city in line (OR lower than the city that started out with the lowest demand).
5. Then go on to the next city, that has the second highest total demand (minus locked goods), and continue matching cities.
6. Once all cities and their goods are matched, or no trade routes are available, then:
7. Transfer resources and update stock.

AFAIK automated units are impossible with the current modding access.
So trader units can't be modified? That sucks, but then we can disregard my suggestions regarding automatic units. Less possibilities make for easier choices.

Some computers won't like to process every players at once at then end of the game's turn, better to process each player during their turn, start is easier from a coding perspective with the available events.
Good to know! I'm so used to playing with simultaneous turns in multiplayer, and I thought that maybe all CPUs did their turns simultaneously or something.

*Goods. I use the word goods for the items that a city have in stock, or which they can stock should they acquire them through trade. I avoid using the word resources to avoid confusion with resources gained from tiles.
*Demand. I use the word demand to mean items that a city have capacity to store and use, minus the current stock. E.g. if the stock of Item A = 20/100. Then demand of item A = 100-20 = 80.
 
Last edited:
Thanks, I'll give some thoughts at all that.

Some time ago, I was pondering a new improvement outside civs territory: a Trading Post, build with two restrictions: on coast and on river, that would be a waypoint for trade routes.

I've started logging the timer, here are the total time used by the pathfinding functions for all civs on one turn (Enormous Terra Map, 30-40 civs):
Code:
---------------------------------------------------------------------------
Turn = 283
IsPlotConnectedCoastal timer = 0.85699892044067 seconds
GetRiverPath timer = 0.25099873542786 seconds
IsPlotConnectedRoad timer = 0.88199973106384 seconds
IsPlotConnectedOcean timer = 19.434997081757 seconds
IsPlotConnectedLand timer = 10.66600227356 seconds

---------------------------------------------------------------------------
Turn = 284
IsPlotConnectedCoastal timer = 1.17999958992 seconds
GetRiverPath timer = 0.28900003433228 seconds
IsPlotConnectedRoad timer = 1.2270045280457 seconds
IsPlotConnectedOcean timer = 20.577000379562 seconds
IsPlotConnectedLand timer = 11.589004039764 seconds

---------------------------------------------------------------------------
Turn = 285
IsPlotConnectedCoastal timer = 0.65200018882751 seconds
GetRiverPath timer = 0.44500041007996 seconds
IsPlotConnectedRoad timer = 1.129002571106 seconds
IsPlotConnectedOcean timer = 25.346997737885 seconds
IsPlotConnectedLand timer = 11.008998632431 seconds

---------------------------------------------------------------------------
Turn = 286
IsPlotConnectedCoastal timer = 0.8240008354187 seconds
GetRiverPath timer = 0.29300117492676 seconds
IsPlotConnectedRoad timer = 0.95999574661255 seconds
IsPlotConnectedOcean timer = 25.783000707626 seconds
IsPlotConnectedLand timer = 11.871002674103 seconds

---------------------------------------------------------------------------
Turn = 287
IsPlotConnectedCoastal timer = 0.44700145721436 seconds
GetRiverPath timer = 0.24899959564209 seconds
IsPlotConnectedRoad timer = 0.87500047683716 seconds
IsPlotConnectedOcean timer = 29.584996700287 seconds
IsPlotConnectedLand timer = 11.215998411179 seconds

Land is the supply path for units, which is not saved/cached
Coastal is the sea trade routes for civs that haven't researched Cartography yet

For a start, I'll try to limit the pathfinders to the tiles that are in range of a trade routes, instead of the whole map (I mean the code don't try to find a trade route between two plots that are outside that range, but for plots that are in range without a direct path, the pathfinders may actually try to go through plots that are outside the maximum range of the trade route)
 
Well, that escalated quickly !

But now I have a base saved game to test optimizations on.

Code:
---------------------------------------------------------------------------
Turn = 292
IsPlotConnectedCoastal timer = 0.091000080108643 seconds
GetRiverPath timer = 0.52599859237671 seconds
IsPlotConnectedRoad timer = 1.6469964981079 seconds
IsPlotConnectedOcean timer = 112.94199919701 seconds
IsPlotConnectedLand timer = 15.778000354767 seconds
 
Thanks, I'll give some thoughts at all that.

Some time ago, I was pondering a new improvement outside civs territory: a Trading Post, build with two restrictions: on coast and on river, that would be a waypoint for trade routes.
Yes, letting the player construct trading posts would be accurate and would let you decrease the initial length of naval routes (having to rely on the posts for extending your trade enetwork, enabling longer routes). It could also give some use to those tiny 1 tile islands that you find. Though trade posts should be useable by all friendly nations (giving you a share of the profits for those trade routes).
I've started logging the timer, here are the total time used by the pathfinding functions for all civs on one turn (Enormous Terra Map, 30-40 civs):
Code:
---------------------------------------------------------------------------
Turn = 283
IsPlotConnectedCoastal timer = 0.85699892044067 seconds
GetRiverPath timer = 0.25099873542786 seconds
IsPlotConnectedRoad timer = 0.88199973106384 seconds
IsPlotConnectedOcean timer = 19.434997081757 seconds
IsPlotConnectedLand timer = 10.66600227356 seconds

---------------------------------------------------------------------------
Turn = 284
IsPlotConnectedCoastal timer = 1.17999958992 seconds
GetRiverPath timer = 0.28900003433228 seconds
IsPlotConnectedRoad timer = 1.2270045280457 seconds
IsPlotConnectedOcean timer = 20.577000379562 seconds
IsPlotConnectedLand timer = 11.589004039764 seconds

---------------------------------------------------------------------------
Turn = 285
IsPlotConnectedCoastal timer = 0.65200018882751 seconds
GetRiverPath timer = 0.44500041007996 seconds
IsPlotConnectedRoad timer = 1.129002571106 seconds
IsPlotConnectedOcean timer = 25.346997737885 seconds
IsPlotConnectedLand timer = 11.008998632431 seconds

---------------------------------------------------------------------------
Turn = 286
IsPlotConnectedCoastal timer = 0.8240008354187 seconds
GetRiverPath timer = 0.29300117492676 seconds
IsPlotConnectedRoad timer = 0.95999574661255 seconds
IsPlotConnectedOcean timer = 25.783000707626 seconds
IsPlotConnectedLand timer = 11.871002674103 seconds

---------------------------------------------------------------------------
Turn = 287
IsPlotConnectedCoastal timer = 0.44700145721436 seconds
GetRiverPath timer = 0.24899959564209 seconds
IsPlotConnectedRoad timer = 0.87500047683716 seconds
IsPlotConnectedOcean timer = 29.584996700287 seconds
IsPlotConnectedLand timer = 11.215998411179 seconds

Land is the supply path for units, which is not saved/cached
Coastal is the sea trade routes for civs that haven't researched Cartography yet

For a start, I'll try to limit the pathfinders to the tiles that are in range of a trade routes, instead of the whole map (I mean the code don't try to find a trade route between two plots that are outside that range, but for plots that are in range without a direct path, the pathfinders may actually try to go through plots that are outside the maximum range of the trade route)

Escalating updates due to on and off blockades and sensitivity to other cities routes?
Oceanic routes really seem to be the big problem, and I would guess that having so many interconnected cities, with so many routes that can be blocked, will lead to having to check new paths every turn. A city whose route is blocked could choose a different destination. While the city who was previously recieving that trade now is forced/enabled to check the route to the city that stopped its route. If the blocking unit moved, then the new route may be established, but then it could be blocked by the same unit in the following turn which would force a new update. And so I can imagine constant updating and path finding being forced upon many naval cities every turn.

That's why I believe in doing path checking only when a city is founded (always tracing the shortest/most efficient path regardless of temporary obstacles). Then keep track of those paths and only modify/check for new paths when a new city/new tech/new policy or other event requires it.

Naval supply lines?
Do the timers above (isplotconnectedocean) include supply lines to naval units?

Oceanic routes blocked on coastal tiles?
Oceanic routes will still go through coastal tiles if it leads to the shortest path, right?
Any I would assume that any route following the shoreline would easily be blocked.
Can only enemy units (like barbarians) block paths?
Whenever a path is blocked, does the code recheck paths to all other cities or only to the city whose path was blocked?

Avoiding coastal tiles?
Can the oceanic path finder be instructed to always prefer oceanic tiles as far as possible?
Let the path finding start by checking the path from the coastal city to nearest ocean tile and likewise for the destination city. Then try to find the shortest path between those 2 oceanic plots, preferring oceanic tiles. That should avoid coastal tile blockades more often.

2nd pass, global tables or a table for each city?
Reading the "second pass" bit I'd like to to ask for some clarifications (if you have time):
Does every city keep its own table with distances, paths and efficiency? Or is there a shared table of all cities?
Is the out of reach table a global table, being updated whenever a city rechecks it's paths? Or is there an OOR table for every city?
 
Last edited:
Each city has it's own table (in a global table) same for the out of reach table.

I've done two pass, first implement A* (GetPathToPlot functions in the table below) and compare with the old pathfinder:
Code:
---------------------------------------------------------------------------
Turn = 295

IsPlotConnectedLand timer = 12.657998323441 seconds
GetPathToPlotLand timer = 0.55500364303589 seconds
IsPlotConnectedLand tested plots count = 1624020
GetPathToPlotLand tested plots count = 95202

IsPlotConnectedRoad timer = 1.5800004005432 seconds
GetPathToPlotRoad timer = 0.55299925804138 seconds
IsPlotConnectedRoad tested plots count = 265704
GetPathToPlotRoad tested plots count = 115458

IsPlotConnectedCoastal timer = 0.12600040435791 seconds
GetPathToPlotCoastal timer = 0.002000093460083 seconds
IsPlotConnectedCoastal tested plots count = 22308
GetPathToPlotCoastal tested plots count = 192

IsPlotConnectedOcean timer = 67.616000652313 seconds
GetPathToPlotOcean timer = 15.197998762131 seconds
IsPlotConnectedOcean tested plots count = 6336216
GetPathToPlotOcean tested plots count = 1973436
---------------------------------------------------------------------------

then added a check to not add plots to the possible path if they are at a greater distance of any end of the path than the MaxRange of the route
Code:
---------------------------------------------------------------------------
Turn = 295

IsPlotConnectedLand timer = 11.91999745369 seconds
GetPathToPlotLand timer = 0.26800131797791 seconds
IsPlotConnectedLand tested plots count = 1500774
GetPathToPlotLand tested plots count = 31830

IsPlotConnectedOcean timer = 62.275001049042 seconds
GetPathToPlotOcean timer = 7.7089986801147 seconds
IsPlotConnectedOcean tested plots count = 5942496
GetPathToPlotOcean tested plots count = 883314

IsPlotConnectedCoastal timer = 0.10700106620789 seconds
GetPathToPlotCoastal timer = 0.00099992752075195 seconds
IsPlotConnectedCoastal tested plots count = 19518
GetPathToPlotCoastal tested plots count = 174

IsPlotConnectedRoad timer = 1.5199999809265 seconds
GetPathToPlotRoad timer = 0.53000140190125 seconds
IsPlotConnectedRoad tested plots count = 250170
GetPathToPlotRoad tested plots count = 84636
---------------------------------------------------------------------------

Now I need to check if I still get the correct routes...
 
Last edited:
- Naval upgrades:

The difficulty here is about units scale, while it's fine to have multiple cities building and stocking "equipment" like swords, war horses or tanks before sending them to units requiring them (or when training an unit), building and stocking battleships and aircraft carriers would be an important waste of resources, and an "unit" represent a single ship.

On the other hand, upgrading using gold like the vanilla system would shortcut the whole supply chain mechanism the mod is based on.

An alternative would be to allow the action of "upgrading using gold", but then the unit would be automatically send to the nearest harbor (or coastal city), with 0 health and averything in "frontline" moved to "reserve", and "locked" there until the required resources are transferred using the healing mechanism of the mod to bring it back to 100% health. That way you don't lose the unit's current resources (materiel & personnel), XP and promotions (whatever mechanisms we decide to use for that) but don't magically gain resources you don't have access to (like "Steel" or other "equipment" we may use to define a "Battleship" or an Aircraft Carrier)
I think I'm going to do the following:

- Set specific buildings (Shipyards) to be able to stock and build naval equipment (stock a lot of "woods", makes/stock some "wooden hull parts" then makes/stock a few Galley/Trireme,...)
- Mark naval equipment to be only stocked in those buildings and not available for trade, that will prevent over-production and waste, with maybe units-specific shipyards for the late game, only able to stock one unit of final equipment.
- Use the "final naval equipment" (ie completed ships) as the other equipment for new units, reinforcements, upgrade...
 
Small update on my progress, the code for the naval upgrade seems to be working, but it has highlighted an issue with the global upgrade system, that is also visible when moving from chariot to riders for example.

When an unit has an upgrade that use an equipment class with a different personnel/equipment ratio (for example, there is one chariot for 4 personnel in a chariot unit and 1 personnel for one war horse in a rider unit), that unit requires reinforcement in frontlines using only the ratio of its specific equipment.

Which means that when upgrading to riders, there is not enough war horses to maintain the unit's current health and you end up with a low health unit.

I need to change that, but it will require some important changes in the current code, and as that part of the code is quite complex, I will review it first and make a full description of an unit turn, as I've done for cities, that will allow me to have a clear understanding of my own code and what to change exactly... And how to change it because there is still the concern of keeping things fast, and using a different/variable equipment ratio for reinforcement may introduce a lot more checks each turn.
 
Does the situation you describe account for the occasional event where all units are awoken from fortify/sentry status (with lowered health), requiring that you go around and reset all of them to a passive state?

At least I think it's all units... definitely a lot of them.
 
Top Bottom