Rethinking the Trade Routes System

[Spirictum]

Following what we've been talking in the 'I want to help this mod' thread, I started this thread so we can come together with ideas for redesigning the Trade Routes System.

For non-modders:
Simply put, it takes too long when the number of routes per city gets high enough, and the number of cities as well. As an example, if we have 4 players each with 100 cities, imagine each city with the ability to make 10 Trade Routes, every player is allowed to make trade routes with each other, and every city is connected. This means that when you pass your turn, for each of the players (4) we will first organize all 100 cities so the best are first. Then for each of these cities, we will check all cities that it can establish a trade route with (400 possibilities) and organize them from best to worst, but discarding anything that goes over the number of trade routes for the city (10) after evaluating the trade value of each other city. Finally, with the best 10 cities to establish a trade route for each city, it is done. This means it'll take: 4(players)*100(cities)*400(total connected cities)*X(time to organize the best 10 trade routes to make)*MaxTradeRoutes(currently 24, I'll talk about this later in this post) = 3,840,000 instructions for the computer to process (times X, but as X is completely unpredictable, I didn't put it in the result).

For modders:
Guys, this is a sick algorithm really:

Spoiler :

For each Player (in CvPlayer.cpp and using the same values in the example above):
First for all cities of a given player clear max trade routes in each city (e.g. 100*24 = 2400 instructions)

for (pLoopCity = firstCity(&iLoop); pLoopCity != NULL; pLoopCity = nextCity(&iLoop))
    {
        pLoopCity->clearTradeRoutes();
    }

void CvCity::clearTradeRoutes()
{
    CvCity* pLoopCity;
    int iI;
    int iMaxTradeRoutes = GC.getDefineINT("MAX_TRADE_ROUTES") + GET_PLAYER(getOwnerINLINE()).getMaxTradeRoutesAdjustment();

    for (iI = 0; iI < iMaxTradeRoutes; iI++)
    {
        pLoopCity = getTradeCity(iI);

        if (pLoopCity != NULL)
        {
            pLoopCity->setTradeRoute(getOwnerINLINE(), false);
        }

        m_paTradeCities[iI].reset();
    }
}

Then for each city we check its supposed position in an organized list(min 100 max 5050 instructions)

for (pLoopCity = firstCity(&iLoop); pLoopCity != NULL; pLoopCity = nextCity(&iLoop))
    {
        iTotalTradeModifier = pLoopCity->totalTradeModifier();

        pCityNode = cityList.head();

        while (pCityNode != NULL)
        {
            pListCity = getCity(pCityNode->m_data);

            if (iTotalTradeModifier > pListCity->totalTradeModifier())
            {
                cityList.insertBefore(pLoopCity->getID(), pCityNode);
                break;
            }
            else
            {
                pCityNode = cityList.next(pCityNode);
            }
        }

        if (pCityNode == NULL)
        {
            cityList.insertAtEnd(pLoopCity->getID());
        }
    }

Only after this we start calculating trade routes for each city.

Now in CvCity.cpp:
24 cleaning instructions, as iMaxTradeRoutes is supposed to be 24.

for (iI = 0; iI < iMaxTradeRoutes; iI++)
    {
        paiBestValue[iI] = 0;
    }

Then we clear Trade Routes for the city again (more 24 instructions), and we enter the possible trade routes list

for (iI = 0; iI < MAX_PLAYERS; iI++)
        {
            if (GET_PLAYER(getOwnerINLINE()).canHaveTradeRoutesWith((PlayerTypes)iI))
            {
                for (pLoopCity = GET_PLAYER((PlayerTypes)iI).firstCity(&iLoop); pLoopCity != NULL; pLoopCity = GET_PLAYER((PlayerTypes)iI).nextCity(&iLoop))
                {

So for each Player(4) for each city(100 of each):

if (pLoopCity != this)
                    {
                        if (!(pLoopCity->isTradeRoute(getOwnerINLINE())) || (getTeam() == GET_PLAYER((PlayerTypes)iI).getTeam()))
                        {
                            if (pLoopCity->plotGroup(getOwnerINLINE()) == plotGroup(getOwnerINLINE()) || GC.getDefineINT("IGNORE_PLOT_GROUP_FOR_TRADE_ROUTES"))
                            {
// BUG - Fractional Trade Routes - start
#ifdef _MOD_FRACTRADE
                                iValue = calculateTradeProfitTimes100(pLoopCity);
#else
                                iValue = calculateTradeProfit(pLoopCity);
#endif
// BUG - Fractional Trade Routes - end

                                for (iJ = 0; iJ < iTradeRoutes; iJ++)
                                {
                                    if (iValue > paiBestValue[iJ])
                                    {
                                        for (iK = (iTradeRoutes - 1); iK > iJ; iK--)
                                        {
                                            paiBestValue[iK] = paiBestValue[(iK - 1)];
                                            m_paTradeCities[iK] = m_paTradeCities[(iK - 1)];
                                        }

                                        paiBestValue[iJ] = iValue;
                                        m_paTradeCities[iJ] = pLoopCity->getIDInfo();

                                        break;
                                    }
                                }
                            }
                        }
                    }

It finally loops for each available trade route (10) and assign the cities in order, discarding the overflow, which gives a maximum of 3955 instructions (the city itself is discarded in the initial if statement, and any city of another team is discarded if there already exists a trade route between this player and that city).

After that it loops through Max Trade Routes (24) to calculate the yields of all set Trade Routes

------------------------------------------------------------------------

My plan is to rethink the entire system, although I'm not full of ideas. I prefer to try to transform an idea into code then to give an idea by myself. And I know this may take some time, so while we discuss ideas here, there are a few stuff that I can adress here that maybe could make the current system a little better:


Regarding process speed:
I can't see a reason to loop through Max Trade Routes when a simple loop through Trade Routes is enough. There are several loops through Max Trade Routes that are only concerned with actual trade routes, and ignore each entry that is NULL (each index between Trade Routes and Max Trade Routes -1). This is just a tip of the iceberg in process time, but maybe it'll improve a bit and it's easy to fix. The only thing that pops to mind against this idea is multi-processing. Maybe iTradeRoutes can change between calculating which will be the cities to make routes and assigning the yields to each route. I'm unfamiliar with multi-processing, I've started to learn about it this semester on college.

Regarding gameplay:
The problem with Teams: Consider this part

if (!(pLoopCity->isTradeRoute(getOwnerINLINE())) || (getTeam() == GET_PLAYER((PlayerTypes)iI).getTeam()))

This happens while we're checking if the city being looped is a possible candidate to establish a trade route with (by the perspective of the city object we're currently on). This only fails if the loop city isn't in the same Team as object city and loop city already has a route with any city of object city's owner.

I don't know everything that makes a player a part of the same team of another player, but I remember in LoR that the Permanent Alliance treaty created a paradox, and could be seriously exploited in favor of the player who establishes such contract, because on one hand your perm buddy is in the same team as you, so his cities can be used several times for trade routes just like your own cities, but on the other hand they were considered foreign cities when calculating yields, which gave them 100% boost in value if I remember correctly. I didn't venture into C2C's code to see if here it's still like this, but it's a mistake that can propagate to many other forms of contract (like Vassal State, or any other possibility that exists or that may exist in the future that could mess with team and foreign identities regarding Trade Routes).

The problem with Order:
This one I've seem in Realms Beyond forums a long time ago. Because of the same order of players being looped for any city, Player 1 has the most advantage while the last player has no advantage. The advantage comes in the form of Trade Routes Slots being filled with your own cities rather then with foreign cities. Imagine a city belonging to player 1 being object city. When we loop through the possible cities to make a route with, we first go through all Player 1 cities and only then we check foreign cities. This means that if any ties occur between a national city and a foreign city, the national will be preferred as it got the slot first. As national cities never leave the pool of available cities, this can mean an increment in for player 1, while the last player will first loop through all foreign cities and only then he will check his own cities, and in case of any ties, the foreign cities will be preferred because they are already filling the available slots.

This isn't gamebreaking, but can be used in Player's 1 advantage for competitive games. The people in the Realms Beyond forums reached that conclusion and fixed this for their tournaments.

Regarding gamedesign:
I feel confused with the concept of Max Trade Routes. Everytime the program checks what's the number of Max Trade Routes it gets a DEFINE and sums it with a modifier. This modifier comes from traits. Now, I'm unaware if there are active leader traits that give trade routes or max trade routes, but this seems wrong to me. Imagine I want to test C2C with no trade routes. Then I'd go to global defines to change it there to 0. But suddenly one or more players have trade routes and others don't. Unless I'm aware of this strange concept in CvPlayer.cpp, I'll believe the game is broken. And if I don't discover there are indeed trade routes in the game I tried to make without them, my test will probably be fake and I won't know it. In vanilla (or maybe LoR) Max trade routes were 8, and if I remember correctly only one civilization could get all 8 because of a UB. But every other civilization had Max at 8, even if it couldn't achieve it in game.

 

[Thunderbrd]

I'm intrigued by everything you've said. I have ideas for streamlining and improving and I can answer your questions. I'm a little short on time so I'll have to spend a little time this evening to respond to this.

 

[Dancing Hoskuld]

My understanding was that when the code was profiled to see where the delays or most time was being used Trade Routes were not even in the top 100! Thus work on them was not cost effective as you would not get a noticeable improvement in game turns. However that should not stop someone who is interested in the mechanism from working on it especially if they are going to expose some of it to XML and Python programmers

My understanding about the use of Max Trade Routes rather than just looping through Trade Routes is that sometimes a trade route is removed by setting the value to NULL rather than removing it. This may be because an array is being used rather than a linked list of objects. It is about when the housekeeping on the array is done. If it is a linked list or similar (we only had arrays and linked lists back when I programmed) then my comment is void.

 

[Thunderbrd]

1) As for the speed of that algorythm, it's amazing how fast these things can go. This happens once in a round for a city usually otherwise it would need to be heavily cached. But there are ways we can speed it up I think and I'll have to get some time to look at it with a technical eye. In AND, they did remove trade entirely out of such suspicion but when Alberts profiled it, we found it wasn't quite as heavy as it seemed due to the infrequency in which this check is made.

2)Max trade routes is a way to index the 'array'. Perhaps it could just be the number of existing routes. I'd have to look closer to know for sure. However, the traits tag does expand or retract the max amount from the base for very understandable reasons. It's quite powerful to expand the max trade routes if you have exceeded the number of trade routes you can have, which actually happens rather easily in C2C. I also added some visibility somewhere on what the maximum amount currently is I think... can't remember though. Suffice it to say, max trade routes haven't been used yet because it's still one of my new tags that haven't been put to use since I haven't had the time to implement the new trait structure (but I'm getting there.)

3) I don't even think a permanent alliance or vassalage counts as a team bond. I THINK the only way to establish a team bond is at game setup, such as in the manner we are playing a team game in the progressive.

4) It would probably be good if we could enforce that some trade routes were to the nearest cities and some to the most profitable ones. And give some player capacity to change which cities he's got trade routes with and extend some AI for that when there are some good reasons to make a technically sub-optimal adjustment.

 

[Spirictum]

My understanding was that when the code was profiled to see where the delays or most time was being used Trade Routes were not even in the top 100! Thus work on them was not cost effective as you would not get a noticeable improvement in game turns. However that should not stop someone who is interested in the mechanism from working on it especially if they are going to expose some of it to XML and Python programmers

How exactly do you do this profiling you mentioned? It seems pretty interesting. I understand that Trade Routes may not be so bad at all compared to other stuff, but my motivation comes from 2 reasons: first I love ever bigger maps, and the Trade Route algorithm suffers a lot from this increase, and second it comes from a time when it was probably one of the more serious mechanisms that slowed down Turn Passing, as I first delved into this when I was still playing Legends of Revolution mod.

My understanding about the use of Max Trade Routes rather than just looping through Trade Routes is that sometimes a trade route is removed by setting the value to NULL rather than removing it. This may be because an array is being used rather than a linked list of objects. It is about when the housekeeping on the array is done. If it is a linked list or similar (we only had arrays and linked lists back when I programmed) then my comment is void.

So checking for NULL as a break isn't a good idea, but iTradeRoutes isn't the number of actual established Trade Routes, it's the number of available Trade Routes. It comes from getTrade Routes which doesn't check the slots in the Trade Route array, but rather values in the Global Defines, Player specific Trade Routes, extra Trade Routes if the city is coastal (Great Lighthouse effect), some unspecified extra Trade Routes, and one 'WorldTradeRoutes' from Afforess. These are all integers that say how many possible Trade Routes a city can have, and not an actual counter of established Trade Routes. Why not loop with iTradeRoutes instead of MAX_TRADE_ROUTES?

1) As for the speed of that algorythm, it's amazing how fast these things can go. This happens once in a round for a city usually otherwise it would need to be heavily cached. But there are ways we can speed it up I think and I'll have to get some time to look at it with a technical eye. In AND, they did remove trade entirely out of such suspicion but when Alberts profiled it, we found it wasn't quite as heavy as it seemed due to the infrequency in which this check is made.

That profiling thing seems pretty interesting indeed, now I'm really curious. But I didn't understand the infrequency you mentioned. As far as I explored the code, update Trade Routes is done for every city of a given player when their turn is starting, so, if you play a single player map, you'll have to wait for all players to update their trade routes (amongst other stuff of course) so you can start playing your next turn. It seems frequent enough to me.

2)Max trade routes is a way to index the 'array'. Perhaps it could just be the number of existing routes. I'd have to look closer to know for sure. However, the traits tag does expand or retract the max amount from the base for very understandable reasons. It's quite powerful to expand the max trade routes if you have exceeded the number of trade routes you can have, which actually happens rather easily in C2C. I also added some visibility somewhere on what the maximum amount currently is I think... can't remember though. Suffice it to say, max trade routes haven't been used yet because it's still one of my new tags that haven't been put to use since I haven't had the time to implement the new trait structure (but I'm getting there.)

Yeah, in the loop through maxTradeRoutes (maybe with the exception of the cleaning routine) it's only concerned with iTradeRoutes, discarding every entry after iTradeRoutes because they are all NULL (but looping through them anyways). But about maxTradeRoutes, my issue is just a conceptual one. I agree it's pretty powerful to be able to have more Trade Routes then other civilizations, but I think MaxTradeRoutes tag in Global Defines will become misleading. Maybe creating new tags instead, like Normal Maximum and Total Maximum (better names are welcome ), one for your mechanic, as it'll limit all those civs which don't have the trait bonus, the other for modding purposes, so we can set a test game with 0 routes if we want to, without having to resort to messing in source, or having to avoid game elements that use this mechanic.

3) I don't even think a permanent alliance or vassalage counts as a team bond. I THINK the only way to establish a team bond is at game setup, such as in the manner we are playing a team game in the progressive.

I'll have to check the code, but I'm sure it used to happen back in the times of LoR, because I witnessed it as a player. Many routes scattered through my cities with Paris, my Perm buddy (a city-state yes, perm alliance is so powerful!), all of which were considered foreign for purposes of value, and all gave a lot more then routes with myself.

4) It would probably be good if we could enforce that some trade routes were to the nearest cities and some to the most profitable ones. And give some player capacity to change which cities he's got trade routes with and extend some AI for that when there are some good reasons to make a technically sub-optimal adjustment.

Like dividing into 2 categories? If we manage to let only a few routes be the best possible, then we may keep the algorithm, for as long as this number doesn't try to increase again. I don't think this is the best plan, but without a better one it's a great way around the problem. Most of the routes will be done to nearby cities, that seems easier to process.
Giving the player access to choose makes me believe it can only be fair if some random aspect is thrown into the AI's choice.

 

[Spirictum]

All this gave me some ideas

What about tying it to diplomatic deals? What if some diplomatic deals give access to establish trade routes with the player you are making these deals with? Like Open Borders for example, giving a fixed number of route slots with that player. For the number to be small these international routes should give a larger bonus then they give now. Once again this is still only being better by reducing the number of times the process has to calculate, and not making the algorithm better.

But I have another idea for that, that if doable, may make things a lot easier. Is Distance really a strong factor for decision in value of Trade Route? Right now in the base it only values the worst of population and distance. If the target city is close, it doesn't mind how big the city is, if it's far away it better be big, otherwise it'll be a poor route. Then when applying the modifiers there is the Population Modifier, but no distance modifier. What if we downgrade the concept of distance's value (together with any other category that is dependent on both cities), and start making the route search by only checking the importance of that city? This way, if we have the ordered list of best to worst cities to trade of every civilization prior to calculating routes, it'll be a simple affair of choosing from top to bottom in the stack of trade cities of the partners we can trade.

If we manage to do like this, it won't matter much the number of routes nor the number of players for processing, like it does now. I believe we can even rely on recalculating these lists on load without needing to save its data in the save file. Load will be slowed down, but turn passing will be a lot better.

 

[Thunderbrd]

What if we downgrade the concept of distance's value (together with any other category that is dependent on both cities), and start making the route search by only checking the importance of that city?

Trade routes are about to not be all about economic benefit. I think we should have some very strong distance factors since you'd first trade with your nearest neighbors. Then you'd have a layer of trade routes that would care more about reaching out to the most profitable cities to trade with. The first is a local interdependancy web. The second is an outreach for the exotic.

I'm not sure how I feel about this getting into the diplo screen... I avoid this screen as much as possible.

 

[Spirictum]

Trade routes are about to not be all about economic benefit. I think we should have some very strong distance factors since you'd first trade with your nearest neighbors. Then you'd have a layer of trade routes that would care more about reaching out to the most profitable cities to trade with. The first is a local interdependancy web. The second is an outreach for the exotic.

I'm not sure how I feel about this getting into the diplo screen... I avoid this screen as much as possible.

Independently from choosing to go with diplomacy and/or separating routes into categories, whenever we have to do a search for the best available route, we get stuck with an algorithm that gets a lot worse for not so big increases in city number. The distance factor, if strong, has a huge role in choosing the best city, so it forces us to recalculate it for every city as distances are dependent on both object city and target city (like overseas factor as well).

If we can avoid the distance factor when choosing (and others that are dependent on both cities), we may use the best to worst trade cities list of each civ to pick the best available route.

Let's suppose I can establish a Trade Route with the Aztecs. Using our current system, after we arrange all our cities in order from best to worst trade city, we get our best city and loop through all available cities from every player that fits the needs to be a trading partner. Only after all this, if Tenochtitlan is good enough it'll be a route in one of our cities.
In the way I'm suggesting, as the Aztecs update their best to worst trade cities' list in their turn (or out of it, by functions which already force an update in routes like when roads are destroyed, cities are conquered/destroyed, or blockades are formed), when it's our turn, if we can establish a route with the aztecs, the only thing we will do, after ordering our own cities from best trade partner to worst, is get the aztec's already ordered list and pick the 1st city there (if more are needed we just keep picking from top). This simply kills the nest of loops.

Distance could still factor in, but in a smaller scale so to not become a strong hidden concept that only players who venture into the code know how to manipulate. If it yields just a bit, it won't matter when choosing, which kills the problem. If all factors (or at least only the important enough) that make a city a better Trading Partner are restricted to that city and don't depend on anything else, we don't have to calculate every possibility for every city, we just pair them by popping them from the Trade Cities' Stack of each player and it's done.


And I didn't understand what you meant by Trade Routes aren't all about economic benefit

 

[Thunderbrd]

If we can avoid the distance factor when choosing (and others that are dependent on both cities), we may use the best to worst trade cities list of each civ to pick the best available route.

Which will completely screw up the planned Ideas project where trade routes play a big role in the spread of ideas and cultures. And disease spread will seem completely happhazard as well. There must be a distance factor so that these trade route based influence spreads will radiate in a logical geographical manner. This is what I meant by trade routes not just being all about economic influence.

However, what can be done is to run a distance check and arrange all other cities in a vector that orders them by distance. It can be cached and will only be re-calculated when a city is destroyed or founded. This vector could then be used to speed up the process you're talking about.

Let's suppose I can establish a Trade Route with the Aztecs. Using our current system, after we arrange all our cities in order from best to worst trade city, we get our best city and loop through all available cities from every player that fits the needs to be a trading partner. Only after all this, if Tenochtitlan is good enough it'll be a route in one of our cities.
In the way I'm suggesting, as the Aztecs update their best to worst trade cities' list in their turn (or out of it, by functions which already force an update in routes like when roads are destroyed, cities are conquered/destroyed, or blockades are formed), when it's our turn, if we can establish a route with the aztecs, the only thing we will do, after ordering our own cities from best trade partner to worst, is get the aztec's already ordered list and pick the 1st city there (if more are needed we just keep picking from top). This simply kills the nest of loops.

Much as I hate the diplomacy screen, I like this. The challenge is to then make the AI work with it in a manner that is less delaying than what we have now.

 

[Spirictum]

Which will completely screw up the planned Ideas project where trade routes play a big role in the spread of ideas and cultures. And disease spread will seem completely happhazard as well. There must be a distance factor so that these trade route based influence spreads will radiate in a logical geographical manner. This is what I meant by trade routes not just being all about economic influence.

You'll have to tell me more about all of this. Is anything already implemented (like trade routes spreading culture or disease)? It seems a pretty good idea. But as long as these factors don't mess with the value of a Trade Route, or if they do, they only rely on information of the city itself, and not something related to 2 or more cities, we can use the method I suggested. Factors restricted to the city itself can be pre-calculated and cached so we can skip the nest of loops almost entirely.

However, what can be done is to run a distance check and arrange all other cities in a vector that orders them by distance. It can be cached and will only be re-calculated when a city is destroyed or founded. This vector could then be used to speed up the process you're talking about.

This seems to be the best solution if we wish to keep the current system of calculation, but we would have to decide what to do with the overseas factor, which also needs info on both object and target city, and not just one of them. But then, a vector of distances would need to be stored for each city, so it's a great cache there. Also, distance wouldn't be so useful, as we would have to cross it with population to give the value. A vector of Trade Value would be more reasonable, one for each city with the order of values of every city this city can make a trade route with. Even so, I believe saving a list of all trade cities available for all cities in the game won't be a good idea.

Much as I hate the diplomacy screen, I like this. The challenge is to then make the AI work with it in a manner that is less delaying than what we have now.

What's funny is that this example can be implemented in both the diplo related option I proposed (which is actually a Trade Routes by Slots system, based on the Category Separation that you proposed of Local Routes and Far Away Routes) or in the one we use now (Routes are like points spend on getting the best available out there).


What is really important to change from a system that for each entry checks every other available entry from the pool of entries and orders them to pick the best possible, to a system that only pairs the top entry of a stack with the top entry(ies) of another stack, is making all factors that increase/decrease the Trade Route Value of a given city be only related to the city itself. E.g. Population, Total , Connection to Capital (which I believe is a flag), Total , Total , Adjacent to River, Be Coastal, Having certain buildings like Market, Harbor, Lighthouse. Now examples that won't work because they depend on more then the city itself: Distance (between 2 cities), Overseas (each city must belong to a different landmass), any rate between cities (rate of , , ), any algorithm that enhance routes based on having other routes to the same place on nearby locations.

I know Distance seems a pretty strong factor to still be needed to calculate the value of a route, but if we can relegate it to a second importance (like influencing just a bit the value of a route) so its influence isn't important enough, then we forget about it when establishing a Trade Route, and only then we apply its change in the value, which will be a simple process.

 

[Thunderbrd]

A third idea I'd been mulling about with too... what if outside of just trading with the closest first as a default, which may not at all be the most profitable, you could use a merchant unit, built in the city, to go and establish a 'forced' trade route with a more distant, more profitable city? This could be used to overlook much of the problems you'd introduce with the diplo screen interplay. Open Borders or other unit passage agreements would be all that would be required to establish the route with a unit. It would make for some unique unit AI work but that wouldn't be worse than any other unit, probably simpler. I wanted to do this and also make it so that to maintain that route, the unit must continue to just repeat auto-moving from one city to the other and back again. (If killed the route reverts to distance based default.) There were other angles of this plan as well if you like the idea. Certainly gives bandits something to prey on and the law/military something to protect.

You'll have to tell me more about all of this. Is anything already implemented (like trade routes spreading culture or disease)? It seems a pretty good idea.

Take a look at the thread for the Modder's guide to the advanced outbreaks and afflictions system. Explains much. There's also the thread for the Ideas project.

but we would have to decide what to do with the overseas factor,

The distance eval could just be numerically adjusted (potentially significantly) for being overseas and how many ocean tiles separate the land masses.

What's funny is that this example can be implemented in both the diplo related option I proposed (which is actually a Trade Routes by Slots system, based on the Category Separation that you proposed of Local Routes and Far Away Routes) or in the one we use now (Routes are like points spend on getting the best available out there).

Yeah I realized that... interesting.

I know Distance seems a pretty strong factor to still be needed to calculate the value of a route, but if we can relegate it to a second importance (like influencing just a bit the value of a route) so its influence isn't important enough, then we forget about it when establishing a Trade Route, and only then we apply its change in the value, which will be a simple process.

I'm thinking if we can keep things pretty simple for base trade routes then give the units the more complex calculations and cache those results until the unit gets to its 'best target' destination and enacts its mission to establish a route, you'd cut down on the weight and fractionalize out how often the calculation takes place and for those distance would be only a factor in establishing the calculated route really, no diff than many units have to do.

 

[Dancing Hoskuld]

A third idea I'd been mulling about with too... what if outside of just trading with the closest first as a default, which may not at all be the most profitable, you could use a merchant unit, built in the city, to go and establish a 'forced' trade route with a more distant, more profitable city? This could be used to overlook much of the problems you'd introduce with the diplo screen interplay. Open Borders or other unit passage agreements would be all that would be required to establish the route with a unit. It would make for some unique unit AI work but that wouldn't be worse than any other unit, probably simpler. I wanted to do this and also make it so that to maintain that route, the unit must continue to just repeat auto-moving from one city to the other and back again. (If killed the route reverts to distance based default.) There were other angles of this plan as well if you like the idea. Certainly gives bandits something to prey on and the law/military something to protect.

Except there are plenty of examples of major trade occurring in history where one of the nations would not let any foreigners in. Ancient China, at various times, is probably the best example. The actual trade happened at the border trade posts even though they were actually equivalent to trades between cities. Thus trade existed without Right of Passage or Open Borders treaties.

The other problem with having a unit is Hidden Nationality units. They will always attack foreign units if the nation is run by AI no matter what treaties you have.

 

[Thunderbrd]

The other problem with having a unit is Hidden Nationality units. They will always attack foreign units if the nation is run by AI no matter what treaties you have.

Thats exactly the point. This was historically a problem for traders. Amnesty would keep them from being attacked by the foreign units of the nation you're trading with though. At least in their borders.

Except there are plenty of examples of major trade occurring in history where one of the nations would not let any foreigners in. Ancient China, at various times, is probably the best example. The actual trade happened at the border trade posts even though they were actually equivalent to trades between cities. Thus trade existed without Right of Passage or Open Borders treaties.

You could argue that against our current rules structure too. You can't get a trade route with a foreign nation's city now if you don't have an RoP at least.

 

[nliv007]

Is it possible to remove the city to city trading and rather have trade based on a number of cities in a defined proximity? I know its not perfect, but historically trade routes didn't generate the money as much as taxes on any trade that came through the city - no matter where it came from and where it was eventually going. Ports were particularly profitable because they taxed everything moving through the port even though a lot of the goods would be headed to non-coastal cities. Rivers less so, but for the same reason.

I think the whole city to city trading should be more focused on size of cities. A larger city will tend to export large amounts of goods to a small city and import only specialized goods. Maybe trade based on what the city is bringing in (particularly luxuries) and its relative size to both neighboring cities and the civilization/world as a whole.

A large city needing lots of luxuries would have high taxes due to the valuable trade whereas a small city that didn't need luxuries (high happiness ratio) would have a lower base trade and a lower population trade.

Does that start to make sense?

 

[Spirictum]

A third idea I'd been mulling about with too... what if outside of just trading with the closest first as a default, which may not at all be the most profitable, you could use a merchant unit, built in the city, to go and establish a 'forced' trade route with a more distant, more profitable city? This could be used to overlook much of the problems you'd introduce with the diplo screen interplay. Open Borders or other unit passage agreements would be all that would be required to establish the route with a unit. It would make for some unique unit AI work but that wouldn't be worse than any other unit, probably simpler. I wanted to do this and also make it so that to maintain that route, the unit must continue to just repeat auto-moving from one city to the other and back again. (If killed the route reverts to distance based default.) There were other angles of this plan as well if you like the idea. Certainly gives bandits something to prey on and the law/military something to protect.

I love this idea of giving merchants a role in establishing Trade Routes, but this has to be carefully thought. I'm not sure it would be nice to need to have it go back and forth to keep a Trade Route active between 2 cities. It seems an overwhelming task for a human player, for a small compensation (a single Trade Route). However, if the merchant has to click a button on the foreign city to 'sell goods' and back home it has also to click a button on the domestic city to 're-stock' (and pay taxes), much like the merchants did back in Age of Empires 2 (get to the foreign harbor, loads payment for cargo, get back to domestic port and unload the payment), then this seems more appealing, as it actively shows the player something is happening. But in this case my concern will be the size it can get. Let's make a comparison: If a single merchant in the proposed system corresponds to a single Trade Route in the current System, 100 cities with 10 routes each would mean 1000 merchants going back and forth. Let's not forget the cap of 8192 units for a given player at any time.

Take a look at the thread for the Modder's guide to the advanced outbreaks and afflictions system. Explains much. There's also the thread for the Ideas project.

I will, probably tomorrow though.

The distance eval could just be numerically adjusted (potentially significantly) for being overseas and how many ocean tiles separate the land masses.

If we care about distance when giving value to a Trade City (and in extension, about any factor that relies on both object city and target city), then calculating the overseas factor the way it is now seems better then doing an adjustment in the distance value. Right now it just checks if both cities are in the same Area().

I'm thinking if we can keep things pretty simple for base trade routes then give the units the more complex calculations and cache those results until the unit gets to its 'best target' destination and enacts its mission to establish a route, you'd cut down on the weight and fractionalize out how often the calculation takes place and for those distance would be only a factor in establishing the calculated route really, no diff than many units have to do.

This is a great way to not let a scenario like I said above happen, of a 1000 merchants going back and forth. If only a few are allowed to do this, and the rest of the mechanic uses simple algorithm to calculate the minor routes, I believe it'll work nicely.

The other problem with having a unit is Hidden Nationality units. They will always attack foreign units if the nation is run by AI no matter what treaties you have.

I agree with T-brd on this, Bandits did force merchants to get men to protect them, and that would be interesting to see in CIV.

Except there are plenty of examples of major trade occurring in history where one of the nations would not let any foreigners in. Ancient China, at various times, is probably the best example. The actual trade happened at the border trade posts even though they were actually equivalent to trades between cities. Thus trade existed without Right of Passage or Open Borders treaties.

Is it possible to remove the city to city trading and rather have trade based on a number of cities in a defined proximity? I know its not perfect, but historically trade routes didn't generate the money as much as taxes on any trade that came through the city - no matter where it came from and where it was eventually going. Ports were particularly profitable because they taxed everything moving through the port even though a lot of the goods would be headed to non-coastal cities. Rivers less so, but for the same reason.

I think the whole city to city trading should be more focused on size of cities. A larger city will tend to export large amounts of goods to a small city and import only specialized goods. Maybe trade based on what the city is bringing in (particularly luxuries) and its relative size to both neighboring cities and the civilization/world as a whole.

A large city needing lots of luxuries would have high taxes due to the valuable trade whereas a small city that didn't need luxuries (high happiness ratio) would have a lower base trade and a lower population trade.

Does that start to make sense?

Now we are talking real concepts! My knowledge about Trade, Trade Routes, Merchants and the like is minimal. The only thing I know about it is what CIV has taught me about it. I'd love to see more of these ideas that are completely outside of the Programming World, that try to analyze the real mechanics that exist or have existed in our World. With that I'm sure we can come with a better system, not only in memory consumption but also in concept. Easier to learn and to manage. And more realistic as well.

 

[nliv007]

I love this idea of giving merchants a role in establishing Trade Routes, but this has to be carefully thought. I'm not sure it would be nice to need to have it go back and forth to keep a Trade Route active between 2 cities. It seems an overwhelming task for a human player, for a small compensation (a single Trade Route). However, if the merchant has to click a button on the foreign city to 'sell goods' and back home it has also to click a button on the domestic city to 're-stock' (and pay taxes), much like the merchants did back in Age of Empires 2 (get to the foreign harbor, loads payment for cargo, get back to domestic port and unload the payment), then this seems more appealing, as it actively shows the player something is happening. But in this case my concern will be the size it can get. Let's make a comparison: If a single merchant in the proposed system corresponds to a single Trade Route in the current System, 100 cities with 10 routes each would mean 1000 merchants going back and forth. Let's not forget the cap of 8192 units for a given player at any time.

There was a civ-like game that did this and it didn't work very well. Albeit it had its own setup, but worth checking out before going down this route. I'll try to find it.

Maybe make a separate class of units that don't have any affiliation with a civilization (and are friendly with all) which you can create and goes out and establishes a trade route with another city to the one it was created from. A mix of a civilization promoting the private sector of trade, but also not having full control.

 

[Thunderbrd]

It seems an overwhelming task for a human player, for a small compensation (a single Trade Route).

It would be a way to get more from the trade routes that already exist but something you could easily ignore as well. It would also be a way to strategically blend Ideas with the Ideas project, or avoid having to trade with local cities that have disease issues you want to avoid. And a way to strategically improve your income if you have the patience to use the system. You should be able to simply automate the whole thing once they are trained, which, to be honest, would be the best way to go in most cases because you wouldn't as a player know what the best cities to trade with are and letting the automation figure it out is like letting the merchant rely on his own professional knowledge to select a city to trade with.

100 cities with 10 routes each would mean 1000 merchants going back and forth. Let's not forget the cap of 8192 units for a given player at any time.

You would only need as many as you want to take from the core trade routes to create a forced trade route that can be more profitable. You would not be getting pick up and drop off amounts, but rather the movement between the cities would just be there to justify the constant trade route connection and if interrupted violently, the route is cut.

But to use your trade routes, they'll just default to the closest cities in order if you don't use a merchant to take one of those defaults and trade it in for a hand selected route. And if the hand selected route is interrupted, then it will go directly back to the default. It might even be good to not try to use all the distant routes you can (for reasons to come in the Ideas project where these default routes, usually being within your own national border, will find their 'ideas' unifying by each trade route strengthening the simliar thinking a naion often shares.)

Thus in the city AI, you'd space out the priority level of when such a merchant for this purpose would be trained and deployed. And you'd probably end up wanting to leave about half or so (maybe varied by the leader personality eventually.)

If only a few are allowed to do this, and the rest of the mechanic uses simple algorithm to calculate the minor routes, I believe it'll work nicely.

Exactly.

Now we are talking real concepts! My knowledge about Trade, Trade Routes, Merchants and the like is minimal. The only thing I know about it is what CIV has taught me about it. I'd love to see more of these ideas that are completely outside of the Programming World, that try to analyze the real mechanics that exist or have existed in our World. With that I'm sure we can come with a better system, not only in memory consumption but also in concept. Easier to learn and to manage. And more realistic as well.

We use so many tags for trade routes, adding subtracting, manipulating the value of each, I would not advocate a whole new trade system. That would be, imo, too reinventing and would destroy a lot of the work I've done on traits that are soon to be manifest.

 

[Dancing Hoskuld]

Is it possible to remove the city to city trading and rather have trade based on a number of cities in a defined proximity? I know its not perfect, but historically trade routes didn't generate the money as much as taxes on any trade that came through the city - no matter where it came from and where it was eventually going. Ports were particularly profitable because they taxed everything moving through the port even though a lot of the goods would be headed to non-coastal cities. Rivers less so, but for the same reason.

I think the whole city to city trading should be more focused on size of cities. A larger city will tend to export large amounts of goods to a small city and import only specialized goods. Maybe trade based on what the city is bringing in (particularly luxuries) and its relative size to both neighboring cities and the civilization/world as a whole.

A large city needing lots of luxuries would have high taxes due to the valuable trade whereas a small city that didn't need luxuries (high happiness ratio) would have a lower base trade and a lower population trade.

Does that start to make sense?

Some of this is covered in "housing" although not as well as I would like. The better the infrastructure and the better the availability of luxury goods the better the housing and the more money you get.

I would have extended it a bit locally produced luxuries would allow lesser elite housing. Luxuries produced elsewhere in the nation, ie not in this city, would allow elite housing but to get high elite housing would require luxuries from outside your nation. I would also include copper (ore) as a luxury if you don't have copper working and so on.

 

[Spirictum]

Take a look at the thread for the Modder's guide to the advanced outbreaks and afflictions system. Explains much. There's also the thread for the Ideas project.

It's probably under my nose, but I don't seem to find it. I tried to ctrl+F 'affliction' and 'outbreak' in both 1st pages of the Modders Documentation and V37 Player Guide threads but didn't find anything.

We use so many tags for trade routes, adding subtracting, manipulating the value of each, I would not advocate a whole new trade system. That would be, imo, too reinventing and would destroy a lot of the work I've done on traits that are soon to be manifest.

I understand your concern with work thrown away, so let's keep the city to city Trade Route like CIV always did.

You would only need as many as you want to take from the core trade routes to create a forced trade route that can be more profitable. You would not be getting pick up and drop off amounts, but rather the movement between the cities would just be there to justify the constant trade route connection and if interrupted violently, the route is cut.

'As many as you want' only exists in the vocabulary of players who aren't worried with competition . But now seriously, this way of designing often lets some smart folks to break the system, and that's something I love to discover how to do. I'm not advocating against this approach, I do think it should be a choice, and several factors should play in to help in your decision, but it's something we must be careful about, so not to let someone play in an unintended way, and get a huge advantage because of that* (e.g. by maxing these special routes, or avoiding them altogether, and because of that gaining a huge advantage we didn't think about when designing the mechanic).

Spoiler :

* Some examples on LoR: The Troops' Shield (tricks the AI into thinking there is no way to reach your cities), National Wonder Doubling (tricks the Game into allowing you to have multiple copies of the same National Wonder), Cultural Victory without a Hint (tricks the other players by hiding your Legendary Cities through Espionage until the very end), 100% Withdrawal Units (this doesn't even need an explanation), the Recon's Carpet for Oil Finding (tricks the Events' System into maxing the chances of getting the Oil Event early on), and there is probably more that I don't remember now, anyone interested I can explain more about them.

So, what we can see so far:

- Routes chosen by proximity
In this case I'm still unfamiliar with the algorithm used to make proximity searches in the map. I know it's used to create the graph of connection between cities, so it shouldn't be hard to find and probably not hard to understand either, I'll take a look at it later. I'm presuming it's a better algorithm then the one used now (Routes chosen by Best to Worst), and it probably is indeed.

- Routes by Category
Well this is simple, using a method to calculate each category

- Routes created by Merchant Units
Here there is more to discuss. At first, a merchant can be trained by a player, and used to create a forced Trade Route between 2 cities, one which is yours, and the other which is anyone's. For that, a Merchant has to move back and forth from Object City to Target City (to use the same terms I've been using so far).

- Routes by Pairing best Cities
This is the method I suggested to avoid most of the loop nest in the current system, it avoids factors that are dependent on more then one city, and only accounts for factors that are a property of or are related to the city itself and nothing else.

- Routes by Slot
Another one I suggested because it goes well with the Pairing system. Routes would be tied to slots given by other elements of the game like Units, Diplomacy, Buildings, but especially concerning foreign Trades, it would give different Slots per each other Nation, instead of simply "Foreign" Slots.


These aren't excludent of each other of course, several can be used at once. What I would like to focus now is the Merchant idea, which to me has the most potential, but still needs a lot of info to be more tangible. As T-Brd started this idea, I wish that he comes with everything he has already thought about it so to guide the discussion of this topic.

I already mentioned the idea of having the Merchant do something actively, like having a button to press both in Object City and Target City, so to simulate the trade to the player, instead of just putting the unit to patrol back and forth (BTW, I never use the Patrol command, does it make the unit go back and forth infinitely?). So how exactly the Route is established? What's the trigger? Will it give its profits every turn, or only when the Merchant comes back? And how is a Route like that interrupted? These are a few questions that come to my mind thinking on this system.

 

[Thunderbrd]

It's probably under my nose, but I don't seem to find it. I tried to ctrl+F 'affliction' and 'outbreak' in both 1st pages of the Modders Documentation and V37 Player Guide threads but didn't find anything.

It's not linked there. It's in the main C2C forum pool of threads. https://forums.civfanatics.com/thre...-combat-mod-outbreaks-and-afflictions.613839/ and for the Ideas project: https://forums.civfanatics.com/threads/c2c-ideas-project-development-thread.562018/