Import feeder?

[ConjurerDragon]

In my current game I tried to use the import/export automated functions for the first time (before I always did everything manually).

The list offers to import all / export all or to choose for all goods what settings I want to choose.

After setting the first 3 cities to import the goods needed for the domestic market in the import feeder I find that that grows tiresome. Is there a way to open a city and set the import feeder to "import everything needed for the domestic market" (e.g. 1 beer, 1 salt, 1 coffee) that may grow a bit later with more population to 4 coffee etc?

Especially when founding more than a dozen cities settings those values several times, every time the demand changes due to population seems to me to be more work than loading a ship manually to send 100 coffee to the city with the demand.

 

[Nightinggale]

I had great plans for import feeder, but it was viewed as too much "the game plays itself" or something when I negotiated to even get it into RaR. I have done a bit since (the all buttons), but so far it has not turned into the feature I envisioned

Simply put it has a level it wants to maintain in a colony. It's set by the player in a rather obvious way. However building queue requirements can also alter the threshold, like the player sets tools to 0, but the construction queue requires 20, then the colony will import to maintain 20. This means import and export to maintain 0 tools in all colonies will automate transports to move tools around to fulfill tool requirements. Automated transports will even be more eager to fulfill requirements by feeder service/construction queue than it will move wares to ports for export. It incorporates hysteresis to avoid jamming a wagon train to move 3 units from A to B each turn. I did add more logic behind the scene than first meets the eye for this feature.

One thing, which has never been added is any connection with the domestic market. It's planned, but.... The same goes for the AI. It can be used by the AI to maintain yield levels in colonies, but it never actually activates those settings because nobody has taught it to do so.

It's still far from the goal. In fact we aren't completely happy with the trade route system at all. Ideas for improvements are most welcome, both in terms of mechanics and GUI for how the player should control the various controls needed for the player to have full control of the automation.

In short: if we are to make a new traderoute system, how should it work and how should the player control it?

 

[ConjurerDragon]

As long as it is optional I do not see a problem. A player can always ignore that system and manually control every single wagon and ship, or automate some or all of them.

So depending on player choice the player would either manually load and send the transport units as before,
or create trade routes like in the original colonization city A --> city B wood
or use the import/export feeder and automate a part of his fleet.

In my current game an automated wagon chose to load goods in the area of New England (random gigantic america map) to go to a colony in Yucatan for over 25 turns - despite a Merchantman being automated too. Does the automated system only take city -city connections into account (as in one inland city has to use a wagon and that wagon will distribute the goods) or that city A is inland but only 4 tiles away from city B which has a port and where the ship could bring the goods to Yucatan in half the time?

My wish would be for the import feeder to
a) not only have buttons for "import everything" "export everything" "clear everything" but something like "import all locally needed goods" or "ensure minimum storage of all locally needed goods"
b) show the locally needed goods in the import feeder (currently I have to change between the local market demand page and the import feeder back and forth until I have set all needed goods).

 

[Nightinggale]

a) not only have buttons for "import everything" "export everything" "clear everything" but something like "import all locally needed goods" or "ensure minimum storage of all locally needed goods"
b) show the locally needed goods in the import feeder (currently I have to change between the local market demand page and the import feeder back and forth until I have set all needed goods).

I feel like this follows the current approach of "write an addon to the current system, even if it's hackish in nature". I like those ideas though.

I'm not happy with the current popup because it tries to do more than the popup system can handle. The popup engine can't be modded, meaning if it is to display anything better than the current display, then we would have to rewrite it from scratch. This is less farfetched than it sounds.

If you should design a GUI from scratch, how should it be and what should it contain? We can do pretty much whatever we want using all the GUI elements from whatever screen we have in the game. In fact (very much overlooked) the python screens supports drag-n-drop, something which isn't supported in BTS.

Let your imagination run wild. Don't hold back because "it's hard to code" or something like that. Even impossible to implement ideas can be useful because it can inspire ideas for doable interfaces.

In my current game an automated wagon chose to load goods in the area of New England (random gigantic america map) to go to a colony in Yucatan for over 25 turns - despite a Merchantman being automated too. Does the automated system only take city -city connections into account (as in one inland city has to use a wagon and that wagon will distribute the goods) or that city A is inland but only 4 tiles away from city B which has a port and where the ship could bring the goods to Yucatan in half the time?

This is a really annoying problem and part of why the automated transport system is broken by design. It works very well if you connect 2-5 close colonies by road and then mass producing automated wagon trains. It works horrible if not all transports are equal, like some land and some ship transports. It has no real concept of distance like it will pick up tools and spend 5 turns when another transport could deliver in 2 turns.

Since this is a design issue, it's not easily fixed. Instead we would likely need a replacement system. Ideas are welcome.

In essence how should the transports behave and how should a player be able to give restrictions or guidelines to transports to make them behave in a more sane manner? Ideally same behavior of multiple clusters of colonies should be supported. One large group of colonies should also behave better like it does matter which colony sends off the cargo even if it's owned plots all the way.

One thing to consider is how to use multiple cargo slots. If say your unit has 5 cargo slots, but it keeps competing against wagon trains for moving one slots transports each turn, your unit will only deliver 20% of what it can do. How can those big transports fill up without too much micromanagement?

 

[Thorik1492]

Maybe abstract it?
Instead of giving slots to transports units, give them to cities through buildings ("waystation" and such). Each slot would then manage transport of a single stack of goods from this origin city.
Number of city transport slots can be interpreted as:
- maximum number of transport units exporting from this city (which will still leave some problems with optimization). After delivery these units would return with goods needed in their origin city (if available in destination).
- respawn rate of single-use transport units (essentialy how many stacks are exported per turn, this is most optmization friendly I think - just sort goods by priority and leave the rest for the next respawn). No return of units to origin, they disappear with delivery, so export only.
- as above but also import, that is: city would use its tranport slot to create a single-use unit in different city, that would deliver needed goods. Most abstracted and easy to optimize.

Land and sea slots would be separated, given by different buildings (waystations vs ports).

This way, player still has to manage transport capabilities of colonies and connect them with roads. The more they produce and farer they are -> the better transport building they'll have to use.

If you'd make these transport units smaller than normal, they wouldn't clutter the map too much. To the contrary - they would visualize important transport routes as continuous river of goods.

If you've played Endless Space 2 compare it to their Space Port and migration system. Also private sector transports in Distant Worlds.

With this system old transport units would be left for manual control, especially in the early game, before transport buildings are constructed.

 

[CptBadger]

I personally would much prefer and use more of a system like the original Col, manually planning trade routes with multiple load/unload points and goods. My experience with AIs decisions leaves me frustrated more than not.

 

[Nightinggale]

I personally would much prefer and use more of a system like the original Col, manually planning trade routes with multiple load/unload points and goods. My experience with AIs decisions leaves me frustrated more than not.

I have been thinking something along those lines as well. However while Colonization classic has a more manual automation, it is rather primitive. We should be able to do better than that and I'm not just talking about the limit of 4 colonies in a list.

Let's say we make a trade route container. We add a list of colonies to it and then we add transports to follow that list. Sounds simple, but it doesn't actually solve the problem because it tells nothing about how to handle cargo or how to make a GUI to set this up.

First of all, imagine we have a player with 3 colonies, all producing cotton. There is a traderoute going through all 3 and it stockpiles cotton at the port ready for export. Sounds simple to set up, but what if the natives gives you fur? Surely you want to move those to the port as well, but manually adding fur and whatever else the natives might give you is time consuming, which is part of the reason why the import/export buttons exist in the first place. We need some sort of interface where the player can give general hints on what to do without a million clicks.

Another scenario to consider is a player with 2 clusters of colonies with 4 colonies in each. One cluster gather all yields in one colony and then another traderoute takes yields from that colony to the port in the other cluster. It also takes stuff like tools back, which is then distributed between the colonies in the cluster as needed for buildings. This means there is a colony, which is both importing and exporting depending on which traderoute is visiting. The benefit of doing this is it makes better usage of units like the trek where you have many cargo slots. The current system is essentially optimized for really fast units with just one cargo slot.

Ideas for any transport system, which takes this into account will be most welcome. Ideally multiple people will propose something because often the best result will be a mixture of multiple ideas and even discarded ideas can be useful because parts of them might be useful or they reveal issues, which makes somebody else think and come up with a design to solve those issues. This means even wild ideas are welcome.

Don't think about programming issues for the time being. Proper gameplay comes from designing the gameplay and then match the programming plans accordingly. Plan programming and then plan gameplay accordingly will never reach gameplay greatness.

 

[Alignn]

The domestic market definitely makes things more difficult for the transportation network, but even if that was resolved by virtual traderoutes (teleporting stuff around) or making the domestic demand apply globally (no need to transport in the first place) there'd still be a complicated web of construction materials being produced and consumed all over, which the current system doesn't handle well - as far as I can tell the auto-transport AI doesn't look ahead of its next destination, so a "main trunk" route with branches off it isn't possible (e.g. bring 300 stone & tools to trunk city A, 100 goes to branch city B, and 100 to the next trunk city C...), nor transporting from a single source to multiple destinations (e.g. pick up food & tools in port, bring tools to city inland, and continue on to desert city to drop off food). Sounds like the original game's route system would at least handle that, but with the sheer variety of goods it would become very troublesome late game, as each connection from one city to the next involves potentially 20+ different goods. Multiple carriages per route would still struggle just to move everything, even if they were perfectly intelligent.

What if we had something in between the virtual routes suggestion and the current physical routes, where you can assign a transport to a given route and the game calculates how much it could transport per turn - and then that amount of stuff just teleports without having to physically move anything? Then we wouldn't have the issue of 101 stone blocking your wagon train from loading important stuff like food, and the domestic market demanding a wide variety of goods isn't such a big problem anymore. Would probably also improve performance during that step where my six carriages decide to roll out and my fps tanks for minute while they figure out what goes where.
The downside to this is that physically moving the goods is part of the charm. Teleporting sure is convenient, but a game of Colonization where stuff just appears in my harbor for me to export without me having to move it there just seems like a lesser game for it, even if I put in the same amount of work as the current system.

Maybe some sort of priority system would help. Prio 1: bringing food to starving cities, prio 2: move construction goods, prio 3: transport finished goods to port cities for export, prio 4: raw materials to where they're needed, prio 5: satisfy domestic market...
It's hard to even envision what this would look like - would I assign priority to a specific type of goods (e.g. food), or to a city (prioritize that starving city), or some combination (bring primarily food to desert city, but if there's space also construction goods, and after that raw materials)...?
Priorities fluctuate constantly during play, too - food isn't nearly so pressing at 100 as at 10. And moving finished goods to the export port is much less important if it's out of storage space and no ship is ready.

I'm starting to see why logistics is a well-paid profession.

 

[Nightinggale]

Maybe some sort of priority system would help.

There is a priority system, though it's very well hidden. Automated transports grant points to each possible trade route and then it will do the one with the most points. If the destination is using the feeder service, the points will double or something like that. This means if you set a colony to maintain say 100 food or 50 ore or whatever input cargo the colony need, the transports are more likely to pick that delivery than it is to just move cargo to the port for export. It's a bit crude, but it's significantly better than no priority at all.

The downside to this is that physically moving the goods is part of the charm. Teleporting sure is convenient, but a game of Colonization where stuff just appears in my harbor for me to export without me having to move it there just seems like a lesser game for it, even if I put in the same amount of work as the current system.

I kind of agree with this statement. If it's something, which just happens behind the scene, then it's no longer Colonization. Actual moving units are really important because they need protection as they can be raided, killed by animals or similar. If moving happens behind the scene, then moving gold from a far away mining outpost will be less of an issue.

 

[Oberon4278]

So here are a few thoughts.

One is that we might benefit from developing a sort of goods flow visualization dashboard. Data visualization tools exist to display complex data in ways that are pretty intuitive. Maybe if someone put together a mockup of a data visualization tool for visualizing supply, demand, and city distance, just the act of making it easier to picture what all is going on would help us think more clearly about the problem? Maybe it would also end up being a huge waste of time, but whatever. I could take that on.

Another is, maybe we could approach this the same way people would have in the real world way back when. There wasn't necessarily master planning going on then, but people still more or less got what they needed just by communicating with one or two neighbors. In game this would work by assigning each automated wagon to a single city. The wagon would go to the city and pick up whatever is most in excess at that city, then at the same time it would "write down" whatever two things (or however many slots the transport has) the city "wants" most. Then it goes to the nearest larger city, drops off the excess goods, and if the goods it wants are there it picks them up and brings them back home. Rinse and repeat.

There is a special case that needs to be addressed in determining city size. You may have an inland city grow larger than its nearest port city, in which case the goods would flow toward the inland city and stop. This is clearly not what we want. You'd need a special check for non-port cities where, if they look at nearby cities and determine that they are the largest around, they designate the nearest/largest port city as their "upstream" city.

To determine what a city "wants" you'd have construction materials, the domestic market, and then some sort of weighted list of things that have been taken from the city for other cities, and goods that foreign wagons have tried to pick up but which weren't available. The list would decay over time.

You could also add a threshold, measured in # of turns since the wagon was "really needed," below which a wagon gets "bored" and decides to head to another town. This should result in wagons sort of drifting around until they land in a city that can use them every turn. You'd probably want to make this cumulative for the city rather than for a specific wagon, so you don't run into a situation where a city has more wagons than it really needs but uses each wagon just before that wagon's timer is up and as a result is hoarding wagons while neighboring cities are having their warehouses overflow.

The only thing that wouldn't address is "tendril" trades, i.e. longer trips between clusters of cities. You could address this in a couple ways. One would be by making particularly large debits of wanted goods trigger a search for more distant cities that might be able to provide those goods, and then sending the largest / fastest transport to the more distant city to pick them up. You could also make it so that if a city determines that it's the end of the line logistically, it looks for other end of the line cities and tries to figure out what goods to send back and forth to them. These two operations are the only ones that would require knowledge of what other cities have; everything else I've described only requires a city to know what it wants and where it sends excess goods.

And, finally, you'd need a way to determine when you want a city to reconsider what it's upstream city is, as city sizes will fluctuate over time. You don't want it to happen every turn because a city might not even have a wagon in it to send upstream every turn, and even if it did populations can change rapidly when a cluster is growing, but you don't want it to be too slow either. You'd have to play with this one a bit to find the right balance.

 

[Nightinggale]

One is that we might benefit from developing a sort of goods flow visualization dashboard.

This sound really interesting, but it leaves the question of what it should look like. Let's assume we make a brand new screen from scratch for this. What should it look like? Don't think about how to code it. Just figure out what the player would need to know to make this work.

In game this would work by assigning each automated wagon to a single city. The wagon would go to the city and pick up whatever is most in excess at that city, then at the same time it would "write down" whatever two things (or however many slots the transport has) the city "wants" most.

Interesting idea and it got me thinking. If we set a specific unit to have a home colony and it goes to another colony and back and that's all it does, then the world view of the unit will become different as in the home colony is the center of the world. This means other colonies should be viewed as travel distance/time.

To do this, the colony should have a list of colonies in the same area (island/continent). When the unit travels to colony A, it stores the turn it leaves and once it returns, the amount of turns passed will be stored in the list. Each colony on the list will then have something like the last 5 travel times. Whenever the time to a colony updates, the list is sorted with the fastest first. Whenever the unit needs to loop through colonies to find one with surplus food or whatever, it loops the list and stops once it finds something.

The result would be that the unit will prefer the closest colony possible (what we want for gameplay) and it loops through as few colonies as possible to figure out where to go (faster code). It also means when the unit has to travel, it will travel a shorter distance, which makes pathfinding faster (less plots to consider). Since it constantly updates for each journey, changes like new roads will get into the list without actually spending time on actual pathfinding during the planning stage.

We could add a timestamp for when a route between two colonies was last updated and use the unit pathfinding to set the time without actually going there. That way we will not fail to detect a new road for 200 turns. The current code has a problem figuring out the travel distance between two colonies, but it can figure out the distance between a unit and a plot (with or without a colony). This means by actually placing a unit of the type in question on the colony we can access more pathfinding data. We should however be a bit careful here as pathfinding is the main culprit of slow late games. It handles massive amount of data and can't be made fast. Yes it could be faster than it is right now, but it can't be made fast compared to most other tasks.

 

[Oberon4278]

What algorithm are we using for pathfinding? Are we doing any preprocessing? What about contraction hierarchies?

I don't understand how it can have a hard time calculating distance between two points when they're colonies but not when they're a unit and a plot. Don't we have a single algorithm that calculates a route, and it's called any time you need to find a route from A to B? I guess I'd better actually take a look at the code. Where can I find it?

As for finding nearby colonies for the purpose of choosing which will be the "upstream" colony for this one, I'd use the Pythagorean theorem to find the straight-line distances to all other colonies the player has, apply a filter (this is where I wave my hands and mathematical magic happens) to figure out which colonies are in the local cluster, then pick the largest colony from that list. All these operations should run in O(n) unless my sleep deprived state has made me stupid.

 

[devolution]

@Oberon4278
The trade route transport scheduling can be found here: CvSelectionGroupAI::AI_tradeRoutes()
As for the pathfinder, it is yet another A* variant

Note that we probably have a half dozen or so open tickets\issues concerning this topic!

 

[Nightinggale]

I don't understand how it can have a hard time calculating distance between two points when they're colonies but not when they're a unit and a plot. Don't we have a single algorithm that calculates a route, and it's called any time you need to find a route from A to B?

You are right that it should be fairly simple to calculate the travel distance between two plots. However Firaxis decided to place the code in CvSelectionGroup(AI) meaning you have to have a group of units (one or more) to call the code in question. Doing it without any units will require writing new code to do so. Why they choose to do so is likely because it's in the design from civ4, not some Colonization original, meaning there is no need to plan traderoutes or anything like that when the code was planned. Doing it based on existing units is actually not a bad design because by having actual units to read data from, the code has access to check if it has promotions to affect movement cost of certain plots, like half movement cost in forests.

The current code has a problem figuring out the travel distance between two colonies, but it can figure out the distance between a unit and a plot (with or without a colony). This means by actually placing a unit of the type in question on the colony we can access more pathfinding data.

I will point out that I wrote "current code", meaning it's about what options we have available without actually rewriting a not insignificant part of the engine.

Note that we probably have a half dozen or so open tickets\issues concerning this topic!

The most relevant for this topic is ticket #102.

One is that we might benefit from developing a sort of goods flow visualization dashboard.

This sound really interesting, but it leaves the question of what it should look like. Let's assume we make a brand new screen from scratch for this. What should it look like? Don't think about how to code it. Just figure out what the player would need to know to make this work.

I would like to ignore pathfinding issues and get back to this. What would be placed in such a dashboard? Also what would the layout be?

 

[andbruu]

Recognizing that I likely have less experience with TAC, RaR and WtP than you guys, I have a few thought I am going to throw out there. I have no idea what is possible, but here we go:

I am not commenting on path-finding, priority and such, only how I think trade could present itself and how the player sets it up.

Instead of having a separate window for trade, the player should be able to click the resource icons on the main screen, cycling through various settings:
..No icon = no effect
..Green "import" arrow = the settlement will import as much as possible of this resource, regardless of storage capacity.
..Yellow "import" arrow = the settlement will import this resource up to one tenth its storage capacity.
..Marketplace icon = the settlement will import this resource to satisfy domestic demand for 10 turns (on normal speed).
..Hammer icon = the settlement will import enough of this resource to build any single building/unit it is currently able to produce.

Any resources in excess of required import are automatically offered as exports.

Note that all stockpile levels are "dynamic", removing the need to type numbers.
This of course offers less precise control, but should be quicker to manage.

Additional icons might be helpful to offer more control, but would increase the amount of clicking needed.


Regardless of system, an option to copy and paste trade settings from one settlement to another would be great.
With the current system, being able to copy settings for a settlement that needs to maintain domestic demands for all goods, and then edit a few after having pasted it, would speed up the management a lot.

 

[FaarmAnimal]

Ideally the automated transport window should have nested options to make it more manageable.

I usually limit my wagons to two cities and it would be handy if I could click plus or minus icons to show or not show certain cities.

 

[ConjurerDragon]

In my current game I wanted to look up the demands of a city in the domestic demand ledger but I failed to find the entry for Sugar. Did I simply overlook the line for Sugar or can Sugar never be locally demanded?

 

[raystuttgart]

Did I simply overlook the line for Sugar or can Sugar never be locally demanded?

No, you did not overlook it, Yield Sugar is not used for domestic demand.

I currently think it was purposely not configured as a domestic demand because it is a "raw Yield".
All other domestic demands are usually "Final / Produced Yield".

You can get more money if "Sugar" is refined to "Rum".
That is why we did not want to have it as "Domestic Demand".

 

[ConjurerDragon]

In the real world grain, potatoes or rice all can be used to produce booze (e.g. Wodka, Sake, Whiskey) which would sell for more than grain or potatoes.
However it would be a bad idea to produce only alcohol and use up all food for that causing a famine...
Historically there were even laws to prevent that.

So having a domestic demand for sugar, which of course would be far smaller than the amount of molasses turned to rum, would still make sense. Unless we restrict the colonies to a bitter place that only persists on maple syrup and beeproduced honey...
https://www.smithsonianmag.com/history/unsavory-history-sugar-american-craving-180962766/

And sugar is used not only as sugar but for sweets, candy or to be able to drink cocoa as pure cocoa is somewhat bitter.

 

[raystuttgart]

In the real world grain, ...

I fully understand your argumentation considering "immersion and authenticity".
But you need to understand as well, that there are sometimes conflicts considering gameplay and mechanics (logic).

A "raw Yield" will (or at least can) be consumed by the Production process of its "refined Yield".
If we also consume this "raw Yield" by another game feature (e.g. Domestic Market) this might lead to conflicts.

A) One of the 2 game features might be messed up. (e.g. show wrong numbers as predictions)
--> Further bugs are possible as well which would lead to bug reports and efforts

B) The player will loose the opportunity to have even higher benefits by producing and selling Rum
--> We will most likely get complaints.

I am not arguing that you are not right with your points.
I am just trying to explain that we need to consider other aspects as well.

Sometimes we simply need to make compromises and one aspect (in this case "immersion") suffers to protect other aspects (in this case "stability" and "gameplay").