Trade automation

vetiarvind

Prince
Joined
Oct 20, 2013
Messages
329
Location
Chennai, South India
Hi Gents,

This thread is meant to discuss ideas regarding improving automation.
I have seen some recurring themes being played out over a few releases, so it looks like there is a unanimous need for improving how supply chains are automated in the game.
Here is an idea by agnat86 which I believe has been discussed a while back between Nightinggale and myself.

One solution for this problem (overflows occur while automating trade routes) might be to use the boxes where you currently can insert a number for the minimum value, and use the number inserted there for both a minimum and a maximum storage quantity, meaning that export cannot reduce the stored goods below the given value, while import cannot increase the stored goods above that value.

This seems like a promising addition for point to point networks. Perhaps import feeder could also use this eventually. For now, some issues arise from a max. value in the trade: what will the trade units do with the excess goods they are carrying? Will we have to do another "cycle" with the excess items where no futher loading will take place for the yields we are carrying over? Will the decision to export be made in the exporting city? i.e we will only load an item if there is space to unload in an importing city? I am by no means an expert on how the trade is implemented, but I believe these questions must be thought off before we work on an upgrade.
 
The problem of the combined min/max value would be that you can't stockpile in a given city and distribute from there at later turns. If you deliver up to say 50 items, none of them can be picked up again. They can only be consumed locally.

Imagine having a supply chain like this:
(City/demand per turn/travel time to next city in turns)
Producer (P)/+100/2 -> B/-10/2 -> C/-15/3 -> D/-25/2 -> Harbor to Europe (H)
Total travel time from P to H will be 2+2+3+2=9 turns. Back and forth this will be 18 turns.
Total demand for production is 50 per turn (just to keep it simple).
18 turns * 50 total demand means 900 items to be delivered (for consumption) within this timeframe, but 1800 to be transported in total (otherwise you would have an overflow in P leading to a stop of production there).
It is quite obvious that you will need more than one transport to fulfill all the local demands in time.
But even with more transports you're running into problems.
4 turns are needed for P->B (back and forth), meaning that you'll have to load 40 items to B.
8 turns are needed for P->C (back and forth), meaning that you'll have to load 120 items.
14 turns are needed for P->D (back and forth), meaning that you'll have to load 350 items.
It is quite obvious that you would have to stop the transports in P until the needed quantity is available, and then you must only load the exactly needed quantity, as otherwise the next transport may be short of supply (not to mention that sooner or later 2 or even 3 transports will be waiting in P at the same time).
Of course you could send a transport to B and C, and one to D and one to H, instead.
Nevertheless, the problem of timing the transports would remain.

And this is just for one yield and only 5 cities and they are all in one line.

In total, it would result in a complete nightmare to handle all this for like 10 or 15 yields in the later game.
So, I think that min and max values have to stay seperated.

An obvious idea would be to have the feeder system changed in such way that transports can be designated to only deliver "feeder-managed" yields.
Then you would have three different kinds of transports:
- fully automated,
- designated to planned trade routes,
- and "feeder-system" transports,
which at first glance would make the fully automated ones obsolete, sooner or later.

Nevertheless, timing problems and competing demands would still happen.

At the current moment I think getting rid of the feeder system and replacing it with a simple max value would be the best option (under the assumption that a second value field can be added). You may then actually need some more transports than required under perfect conditions, but it would be the easiest to manage and could be explained in simple words.

If no second value field can be added, the feeder system should be rewritten so that it will work in combination with designated transports.
This would require the following:
a) delivery stops if the feeder value is met (already implemented, as far as fully automated transports are concerned)
b) delivery starts again if the quantity drops below the 75% (already implemented, as far as fully automated transports are concerned - but this value should be revised)
c) withdrawal up to a minimum value defined by the new x% value has to be made possible

Still, one problem remains: what to do with the AI (which actually means total revision of the fully automated transports, including a proper demand/in time calculation based on the available transporting quantity)
 
Nice writeup!
4 turns are needed for P->B (back and forth), meaning that you'll have to load 40 items to B.
8 turns are needed for P->C (back and forth), meaning that you'll have to load 120 items.
14 turns are needed for P->D (back and forth), meaning that you'll have to load 350 items.

Won't P->C require 200 items? 8 turns while we consume 25 units per turn? (10 for B, 15 consumed by C)

I thought these sort of networks worked as a ring? Where you'd import\export from intermediate nodes, like [B,C,D] or were you just giving these as a hypothetical example?

Anyway, I also believe we need a seperate min\max value. :thumbsup:

An obvious idea would be to have the feeder system changed in such way that transports can be designated to only deliver "feeder-managed" yields.
Then you would have three different kinds of transports:
1- fully automated,
2- designated to planned trade routes,
3- and "feeder-system" transports,

which at first glance would make the fully automated ones obsolete, sooner or later.
I think "fully automated" ones will remain, atleast for the AI. Any upgrade we do will only be for the human player as of now, and will cover 2) and possibly 3)
I doubt if the AI uses the feeder system, maybe Nightinggale or Ray can confirm for sure. The AI uses the "automate full" feature 1).

At the current moment I think getting rid of the feeder system and replacing it with a simple max value would be the best option (under the assumption that a second value field can be added). You may then actually need some more transports than required under perfect conditions, but it would be the easiest to manage and could be explained in simple words.

This was also what I had in mind and importantly is doable without an equivalent to a PHd project. I wasn't planning on removing the feeder system though since it only complements any changes we make to 2).

Still, one problem remains: what to do with the AI (which actually means total revision of the fully automated transports, including a proper demand/in time calculation based on the available transporting quantity)

I thought we could simply leave the AI system as it is without changes. Going by "If it ain't broke, don't fix it." So, doing a demand/intime calculation is something we can avoid.
Of course, any changes we may make reg. this may never be part of the R&R mod, but that is fine. Our goal for now is to remove as much tedium from the human player's experience.
 
One of my goals is to have overflows occur only at source, and not at the intermediaries. As a player, I want to keep things simple, so if there are too many yields getting stuck at source, it's my job to add more transports\routes and distribute it. The last thing I want is for some trade-junction to overflow because some of my transports just got captured or are busy doing something else. Also, I'm a bit of a micro-manager and like to assign my own transport routes instead of depending on the automation or even import feeder.

So, keeping this in mind, my initial plan is to have a min\max values for each city. It is up to the player to configure these values and set a route cycle such as you described.. P->(A->B->)...-->Harbour
The logic I am proposing is simple and greedy. A player if he wishes to carry forth a yield from P->B will have to configure an export in P, import and export in A and and import in B. He will have to give min\max to accomodate this temporary storage of the yield in A.

So, while trading from city i I will try to fill in as many allowable yields that can be carried to the next city i+1. If i am unable to find a yield for my capacity, i will try to fill for city i+2 and so on. To see what is "allowable", we will use the simple calculation. :think: (just read the comments in italics if you're not a coder)

  1. projectedAmount[yield][nextcity] = current[yield][nextcity]+ (delta[yield][nextcity]*turns-to-reach-nextcity) //how much will the next city have when i arrive there?
  2. amountToTake[yield] = maxCapactiy[yield][nextcity] - projectedAmount[yield][nextcity] //how much do i take to the next town?
  3. if(current[yield][currentcity] - amountToTake[yield] < minCapacity[yield][currentcity])
    then amountToTake[yield] = current[yield][currentCity-minCapacity[yield][currentCity] //we cannot breach the min capacity value while exporting
  4. if(amountToTake[yield] < 0) then amountToTake[yield] = 0

We will sort by yields based on amountToTake, and use value of yield as a tie-breaker. If there is still room, we will try to fill yields for subsequent cities of nextcity in our cycle, running the above logic. Of course we will terminate the loop with the same city, since we are not going to export to the current city itself.

I will try to implement this in my modmod when i get some time, test in my game and if everything looks good, publish it. Maybe I should create a seperate modmod? :hmm: The only unsure bit like you mentioned is adding the additional max column in the UI since there is very little space as it is, but let's see, might just be able to squeeze it in or rearrange the columns to push the import feeder button to the right and add a scroll bar.
 
I haven't given much thought to colonial transport and European trade yet, but I have wrote up a brief idea concerning fishing vessels. It is my hope that the idea is sound, but I expect that it would require significant coding. Blegh.


Fishing Boats and Whaling vessels can be assigned to a patch of resources and set up. When this is done, a dialogue box comes up which asks the player to choose a Home Port for the operation. Every turn thereafter, the boat will passively send resources to that port, the amount dependent on the naval building of that city. The capacity for how many fishing expeditions a city can support is tied to the building's level.

Pier: +4 resources, 2 operations.
Dock: +5 resources, 4 operations.
Drydock: +6 resources, 6 operations.
Shipyard: +8 resources, 8 operations.
Naval Port: +10 resources, 10 operations. The Naval Port is a national wonder.

Currently, fishing vessels sometimes travel to Europe with empty hulls or just sit around when there are unused fishing spots. Reducing the need for an AI should make fishing a reliable enterprise. This also eliminates the "wait until I have 300 of X" behavior from fishing, and adds a stable resource increase to a given colony for each turn. That is good, considering that a colony relies on predictable income to operate effectively - be it in real life or a virtual scenario.
 
I haven't given much thought to colonial transport and European trade yet, but I have wrote up a brief idea concerning fishing vessels. It is my hope that the idea is sound, but I expect that it would require significant coding. Blegh.

I actually like the existing fishing\whaling system! Having a passive income would simply reduce the chance of losing your boat by pirates, enemies, etc. As for regular food income from fishing, you can already do that using fisherman, tada!
 
I actually like the existing fishing\whaling system! Having a passive income would simply reduce the chance of losing your boat by pirates, enemies, etc. As for regular food income from fishing, you can already do that using fisherman, tada!

As far as I have understood Sabin's concept you still have to steer the fishing/whaling boat to the respective ressource, where it will stay. So, there would be still the risk of losing it.
 
As far as I have understood Sabin's concept you still have to steer the fishing/whaling boat to the respective ressource, where it will stay. So, there would be still the risk of losing it.

That is correct. The big change is that fishing boats will not be wandering off to Europe or be paralyzed, because there is less movement and thinking to be done by the AI.

Concerning the fisherman part, not all fish tiles are readily accessible by a colony - and colonies can't directly mine whales under the current system. Fishermen have superior production abilities when compared to fishing boats, but they lack the reach and vulnerability that fishing vessels have. These are important tradeoffs and choices to consider.
 
I have been thinking a bit about this and how it would be possible to implement. First of all I think we should ignore any GUI issues until we have an idea of what needs to be accessible in the GUI.

A bit of brainstorming:

I'm thinking maybe the best approach is to completely ignore the current code and make a brand new system custom designed to what we want. Right now the system is based on vanilla, which is likely a limitation when designing a new system. At the same time the current system will continue to work and the player will then select which one a transport unit is assigned to.

Say we introduce a new concept, which we call traderoute groups. One group then has a list of colonies, which the units should visit and they are visited in that order. Units can then point to the traderoute group it is assigned to and any change to the group will affect all units using that one.

For practical reasons, a traderoute group should be a new class and it should have a list of colony orders. The colony orders should then be another class.

A colony order group should then contain one or more arrays of yields.

Behaviour of an automated unit:
It looks at what it can pick up from the current colony
Loops through the list of destinations to figure out what to pick up and how much
Moves to the next colony on the list
Repeat

If we store thresholds in colony orders, then we can have different threshold in the same colony for each traderoute group, which can be quite useful at times. Example:
We have route A, which is set to maintain 400 lumber in colony C1
Route B is then set to pick up lumber from C1 and is allowed to let it drop to 200. B will then drop off lumber in a few nearby colonies. That way long distance lumber movements will always go to C1 instead of any of the nearby smaller colonies. Likewise produced yields will gather up in C1 ready to be moved back by the lumber carrying units.

If we use a signed variable for threshold, we can do stuff like:
-1: use colony settings
-2: never (un)load this yield
-3: ignore threshold
etc

Default values for all yields should then be to use colony settings.

I think this can be made into something which the player can use to make an efficient automated trade network. It should be fairly simple to set up and at the same time it has advanced features which allows a whole lot of micromanagement, which players can use to customize transports to precisely their own strategy.

The question is how much work it would be to code. I don't think it will be unreasonably much and the concept of adding new classes should make it reasonably simple to add this to savegames.

I would like a system like this to be so much an addon, that it ends up with most/all the code in a new cpp file. That way it is very modder friendly and can be copied to other mods with minimal effort.

I don't have any idea for a GUI though.
 
Any system that is put into place should include the feature that when you assign a transport vehicle to whatever route that it renames the unit to note what you've assigned so if it gets unassigned due to whatever, you can easily reset it to what it was. This might be difficult for it to fit in the name length restriction but some sort of abbr should be able to be devised. Maybe simply adding a number to a trade route.
 
Any system that is put into place should include the feature that when you assign a transport vehicle to whatever route that it renames the unit to note what you've assigned so if it gets unassigned due to whatever, you can easily reset it to what it was. This might be difficult for it to fit in the name length restriction but some sort of abbr should be able to be devised. Maybe simply adding a number to a trade route.
That's a very good point, but we should be able to do better than that.

We can store whatever we want in a unit. We should store which route it is assigned to, next stop and an on/off switch. If it is unassigned for some reason, it just switched off the automation, but remembers the route and next stop. That way the player can just click a button and the unit resumes the route where it left off.
 
Say we introduce a new concept, which we call traderoute groups. One group then has a list of colonies, which the units should visit and they are visited in that order. Units can then point to the traderoute group it is assigned to and any change to the group will affect all units using that one.
Seems like an interesting idea for sure. :) I believe if it is added, it will be only for the human player?
However, this feature can be simulated using the current trade route features. You could have a group of transports that are assigned to a series of routes. Just name the head of the group to whatever route name you want.
Of course, to add new units to the route-group you will have to manually set this up.

Also, implementation wise I guess you could create trade route groups and then hook it up to use the existing trade route system. Your idea seems like a nice wrapper around the existing trade system. (a route group could be implemented as the routes a SelectionGroup is assigned to)
The actual trade-runs can still be the same core implementation. Of course, it will be hard to incorporate your idea of colony orders to the existing implementation directly.

Currently, I am working on having a max import limit for a yield in each town and I am modifying the existing implementation in a modmod. My goal is to keep the existing logic in place for the AI and human players who don't have a "max capacity" set, and only extend the logic where the user has specified it.

If we store thresholds in colony orders, then we can have different threshold in the same colony for each traderoute group, which can be quite useful at times. Example:
We have route A, which is set to maintain 400 lumber in colony C1
Route B is then set to pick up lumber from C1 and is allowed to let it drop to 200. B will then drop off lumber in a few nearby colonies. That way long distance lumber movements will always go to C1 instead of any of the nearby smaller colonies. Likewise produced yields will gather up in C1 ready to be moved back by the lumber carrying units.

This can be done once I finish the max threshold mod-mod using the import-export system. You will have to set a large max threshold for the C1 colony (since it's a distribution point) and smaller max thresholds for the small colonies in route B so they don't overconsume from C1.

If you are keen on creating a totally new trade system, you are welcome to take ideas from the modmod i am working on, since you may encounter the same issues due to max overflows. (no idea when it will be published cuz I'm busy but hopefully before 2.2 gets released)
 
That's a very good point, but we should be able to do better than that.

We can store whatever we want in a unit. We should store which route it is assigned to, next stop and an on/off switch. If it is unassigned for some reason, it just switched off the automation, but remembers the route and next stop. That way the player can just click a button and the unit resumes the route where it left off.

Yes, that would be fantastic.
The only issue is recognizing that it has become unassigned. That's where the name is helpful. When I find myself being asked to give a command to a unit that has a trade name, I know it's become unassigned.

Maybe the best solution would be using a switch like you said but maybe also just mod the name to include like a capital A so you know it was assigned at one point.

But just the switch would be REALLY helpful.
 
May I propose creating a thread for TAXES AND NAVIGATION and link to it from here.
I think Taxes and navigation 2 falls safely within the "trade automation" topic, so this should be a good place to continue the disccussion.

I'm sure I will have something to say once I have taken a closer look. For now I do have one comment. Since you moved the spinbutton to the second row, the first how gained more space and there is now room for adding text to the feeder button. I know it's a minor detail, but it was the first thing I spotted when looking at the screenshot. Also I would like to say that at first I thought why use two rows for each yield. However thinking a bit I would say that it is actually a good idea because it mean gaining the space you need while still using well tested (possibly vanilla) code.

I will review the code in details later. Partly because I'm interesting in how you solved certain issues, but also as quality check. One suboptimal piece of code I noticed so far is that you send two network packages with import/export is modified even though only one is needed. That will increase bandwidth usage and lag. It does work and it will not cause desyncs (at least I don't think so) meaning it is a detail, which is the difference between good and great. I checked right away because it would be a candidate for introducing desyncs.
Sounds good! About the network packet thing, could you locate the portion of code for me? I don't recall touching that portion at all. All my mod code will be commented by \\R&R mod, vetiarvind, max yield import limit

As far as the UI, I was really forced to put a two row layout simply because the popup cannot be resized. Items that exceed the width disappear. The creation of the new "condensed" exports screen will offset the inconvenience with having a popup with too many rows though.
Yeah I got credited for coding even though I didn't actually touch the code in your mod. Looks like my code has taken a life of it's own to give me credit :king:

Your existing code saved me atleast an hour or two of getting acquainted with the trade features and the UI popup. And yes, otherwise the modmod is entirely my work.
 
About the network packet thing, could you locate the portion of code for me? I don't recall touching that portion at all. All my mod code will be commented by \\R&R mod, vetiarvind, max yield import limit[*quote]
CvDLLButtonPopup::OnOkClicked()
[code]int iMaxLevel = pPopupReturn->getSpinnerWidgetValue(NUM_YIELD_TYPES + iYield);
if(iMaxLevel != pCity->getImportsLimit(eYield))
{
gDLL->sendDoTask(info.getData1(), TASK_YIELD_MAXLEVEL, iYield, iMaxLevel, false, false, false, false);
}
// R&R mod, vetiarvind, max yield import limit - end
if (bImport != pCity->isImport(eYield) || bExport != pCity->isExport(eYield) || bMaintainImport != pCity->getImportsMaintain(eYield) || iLevel != pCity->getMaintainLevel(eYield))
{
gDLL->sendDoTask(info.getData1(), TASK_YIELD_TRADEROUTE, iYield, iLevel, bImport, bExport, bMaintainImport, false);
}[/code]
The network code is completely undocumented (at least I haven't found any documentation). However it follows a pattern I have seen before, which tells me that the exe must do for this to work. gDLL->sendDoTask() sends a package on the network every time it is called. The same goes for all the other send functions in CvDLLUtilityIFaceBase.h.

The way to reduce bandwidth usage and lag is simply to use those functions as little as possible. At the same time not using them when they should have been used will cause desyncs.

What I would do here would be to add iMaxLevel to the existing data being send. Because of vanilla modder-unfriendliness, there is no free variables to use. (the network arguments are bare minimum for vanilla to work with no expansions in mind). In other words we are going back to the old school art of counting bits to ensure that we have enough. The number we transmit is in the range of 0 to 999. 10 bit unsigned goes from 0 to 1023 meaning the 32 bit int can hold 3 such numbers with the right code.
Code:
int iBuffer = iMaxLevel << 16;
iBuffer |= iLevel & 0xFF;
Send iBuffer instead of iLevel and both ints are send in the space used by a single int.

Splitting in CvCity::doTask()
Code:
int iLevel = iData2 & 0xFF;
int iMaxLevel = iData2 >> 16;

Alternatively you can use the bit operation macros from M:C (line 160) http://sourceforge.net/p/colonizati...LL/ci/master/tree/DLL_Sources/CvGameCoreDLL.h
Those macros will likely make more readable code.

As far as the UI, I was really forced to put a two row layout simply because the popup cannot be resized.
I experienced that too, hence the reason there is no text on the feeder button.
 
Hi,
I'm fairly certain the trade system has bugs. Now that I have conquered all of New France (so quite an hefty territory) I see some of my regular transports, for example well established lane like Plymouth-Boston, go to some of the former French cities.

This is totally outside their road! There is no danger around their normal road (plus they are on ignore danger because I deal with anything with my army) and the automation is still correctly set to my 2 English cities, and absolutely not with a French city at something like 50 tiles away from their normal route.

I'm very confused by that, if the trade system falls apart, the game will be a nightmare.
 
The trade system figures out to send a transport from colony A to colony B. It's not concerned with how that should be done and it will use the regular pathfinder to do so. I somewhat trust the pathfinder, which mean the issue is the colony it picks as destination.

It might be the issue where it picks up cargo and then drops it off in the next colony. However after it is maxed out, it figures out the best colony to drop the rest off, which could be an old french colony. The "drop off at semi random colony" code from TAC has already been the cause of one bugreport. It might work well most of the time, but not always.

A savegame could be good when you report a bug. Right now I have to take your word for this bug.

Also a fix is likely to appear in RaRE, not RaR. Sadly RaR development is dead :(
 
Oh ok, so if it has 'nothing to do', you mean there can be a kind of override and it uses another algorithm, like perhaps the one used when you set transports in full automation (I never do that).

I would have no problem switching to RaRE really, but you don't have a comprehensive changelog and I have modded RaR on my side too (nothing changing architecture or adding units, just editing values of units, yields, etc. so not that complicated). For example I would like to get your fix on 'forts can grow outside of city exploitation range but inside culture' but I'm not even sure how you did that and if I can just take this part for myself.
 
Oh ok, so if it has 'nothing to do', you mean there can be a kind of override and it uses another algorithm, like perhaps the one used when you set transports in full automation (I never do that).
The problem is mainly with non-fully automated transports. If it carries cargo, it will go through all the traderoutes assigned to that transport to find a destination. If it fails, then it will go through all colonies and find one. It seems that it will ignore traderoutes completely when doing this and can even pick a colony which doesn't import the cargo in question. As I said it appears to be TAC code kicking in here and I don't know why it was made like this. I know I had problems in vanilla where cargo could accumulate in transports, making them useless, which mean the TAC code is likely an attempt to "reset" the transport in order to regain lost cargo capacity.

For example I would like to get your fix on 'forts can grow outside of city exploitation range but inside culture' but I'm not even sure how you did that and if I can just take this part for myself.
I don't recall making such a fix and it's not in the changelog. I do remember this issue being mentioned by somebody else and I think it is included in the last RaR release, but I'm not sure.

RaRE changelog: https://sourceforge.net/p/religionandrevolutionextended/code/ci/master/tree/RaRE%20changelog.txt

RaRE changed the following xml files:
  • Bonus (added turkey on savannah, though I thought it was a RaR fix :confused:)
  • GlobalDefineALT (two settings regarding 1/2 plot city radius)
  • Yield (added bIsExportYield)
  • GameOptions
  • Command
  • Added CIV4GameTextInfos_RaRE.xml (no other change in Text dir)
Other than that, you can just overwrite the files. RaRE haven't changed anything related to graphics or audio either. It's mostly a pure code modmod.
 
Top Bottom