[RaRE] RaR Extended: a combined modmod for the RaR modification

Noticing your comment here
However considering that there appears to be absolutely no interest in RaRE at all (at least for developing), I fear people have moved on.
I would like to raise the question in which direction RaRE should move.

As far as I am concerned, there are two areas which desperately need some improvements:
1) usability
2) AI

No. 2 I regard as the most important, but up to now this is just my personal opinion.

What do others think?
 
My plan for improvements: (in no particular order)
  1. Make a game option to switch between 1 and 2 plot city radius
  2. Figure out a good way to make automated transports handle a mixed land/sea transport network
  3. Feed construction automatically with needed yields (such as tools)
  4. Make the AI figure out NOT to clear all forests
  5. Figure out a way to switch between RaRE and M:C without resetting the game setup (they likely have to match in numbers or something)
  6. Improved GUI for plot modification, such as draining marsh. Ideally it should tell yield changes from the action
  7. Change light to dense forests without clearing the light forest first
  8. Ability to build improvements without clearing forest first (good when mixed with yield change display)
  9. Apply post 2.4 RaR python fixes
  10. Upgrade JIT arrays once I'm done updating them in M:C (provides more features and smaller savegame files)
  11. Improve learning by doing to not forget everything the unit learned in a profession by switching it to a different profession for one turn (happens often with fishermen and pirates)
  12. Make a unit plot/profession memory, which allows fishermen to resume the task you assigned to it once the pirate has moved out of the plot
  13. Make the AI try to make the same units work on the same thing each turn. Right now it is perfectly happy assigning a new unit as farmer each turn rather than letting one stay long enough to become an expert
A number of those (particularly #2) deals with vanilla issues and a solution would work for M:C as well. #3 is already solved in M:C and I applied it to a local version of RaR 1.9 where it works well. This mean at least half the items on this list involves moving code between RaRE and M:C.

However my motivation to do this comes from not wanting to see RaR(E) die and not from a lack of things to do (M:C could keep me busy for a lifetime). It is a bit tricky to work on it and make it feel alive if you are only one person. This mean I will probably not start on this list until somebody else also starts to do something.
 
Well, as far as I understand it this means improving usability is your prime target.

There is nothing wrong with that, of course, I just want to make sure I got the right impression?
 
Well, as far as I understand it this means improving usability is your prime target.
4, 6, 8, 11 and 13 is AI or can be used to improve the AI. Also M:C has debug code to look inside AI cities (though the GUI is a bit buggy as it shows production bonuses of the human player, not the city owner). Adding that to RaRE will make it easier to figure out what the AI is actually doing and that way figure out specific improvements the AI could benefit from.

I'm not sure if I would say that I value the AI to usability the highest because the answer would sort of like be both. Other improvements, such as smaller savegame files doesn't even fit into any of those two categories, but if it can be added by simply copying a file from M:C, then why not? :)

Besides I don't think we need to have 100% the same goal. For instance if one person improves say AI ship movement and another person improves the pioneer action help popup and a third improves the pedia, those updates will work well together even though they do not share a common goal.

Big changes like adding units or something like that would require a common vision. This makes it important to announce what you want to do, but so far I don't think any of the proposals really conflicts with each other or some shared goal.

There was a lot of proposals for improved combat at some point. I would love to get those into RaRE.
 
Actually I'm also interested in improving RaR
I'm a fairly experienced modder, but have free time to mod very sporadically, and already have quite a lot on my plate with Rhye's and Fall: Europe.
So don't expect regular updates from me, but I definitely would like to fix/improve whatever I find when I'm in the mood for Colonization
 
I just had an idea, which on it's own makes it worth it to code on RaRE. When the game desyncs, it should write to a log file even if logging is disabled. Ideally it is a file used for only this purpose.

This file will then be a kind of savegame, except it is written in plain text, making it human readable. The files from two computers can then be compared using diff and any differences will stand out. This will allow to tell not only that the game desynced, but also which data made it desync. It will not tell what went wrong, but at least telling that it is related to a specific variable is far better than the current system, which is "something somewhere in the dll or python".

Such a log would have to contain anything, which could cause a desync, meaning it's all plots, players, units, cities, random seed and possibly other stuff as well. It will be a big txt file, but if it helps removing even rare desyncs, then it will be worth it.
 
I added the post 2.4 hotfixes that ray posted for RaR.

I have been thinking what to do with RaRE. I have been waiting for other people to show up before doing anything. Maybe I should do it the other way around. The lack of activity might scare people away because they think precisely the same as what I'm thinking.

I came up with a new idea I would like to see in RaRE. The pathfinder is by far the single most slowest part of the code in late games, but modding it is tricky because it is handled by the exe. However I noticed that K-mod has an exe replacement for pathfinder for BTS. I wonder how much work it would be to move it into RaRE.
 
Improving pathfinding would be a very good thing, but this is way beyond my skills, so it would have to be let to you.

I have to apologize for having been so silent.

I have to admit that I was deeply demotivated during the past weeks. Actually, I was thinking about switching back to a previous release of RaR (currently I have still not installed 2.4), as meanwhile the AI is lost even more than before.

Actually, it turns out that pressing more and more "things" into the game has been quite counterproductive. Unfortunately, one almost can't improve this without repairing the AI.
I really think this is the most important thing to do. Maybe we should start some kind of "brainstorming" of what and how to repair.
 
I agree. Fixing the AI should have a high priority. However at the moment I'm not sure what to do about it. I think once I have finished CivEffects in M:C, I will make a 1 plot radius switch (I don't like the 2 plot thing) as well as some traderoute automation. With that installed, I will start playing RaRE and spy on the AI to figure out what it does and how it does things wrong. It's quite possible that I will have to write cheat code to allow spying way more than normally allowed to figure out what the AI is doing, but so be it. Once we have a good understanding of how the AI behaves poorly, we can start to talk about what to do about it.

We have to remember that we have the entire server history in git. We can go back in time and branch out from RaR 2.3 if we like. I would prefer to fix the AI issues first, but it's nice to have as a fallback plan in case the AI becomes too tricky to fix.
 
I agree. Fixing the AI should have a high priority. However at the moment I'm not sure what to do about it.

As far as I am concerned I think that improving trade routes would be most beneficial, as it not only helps the AI but would make life easier for the human player as well.
When we can teach the AI how to distribute yields properly, in time, in the right quantities, a first step is made to enable the AI to do proper production and trading, too.
Conceptwise, this shouldn't be too difficult. It's only that I myself am too weak a programmer to do it just by myself.
 
As far as I am concerned I think that improving trade routes would be most beneficial, as it not only helps the AI but would make life easier for the human player as well.
When we can teach the AI how to distribute yields properly, in time, in the right quantities, a first step is made to enable the AI to do proper production and trading, too.
Conceptwise, this shouldn't be too difficult. It's only that I myself am too weak a programmer to do it just by myself.
Try the develop branch in M:C. You can set a city to import the yields it need to finish the building it is constructing (well, for anything in the queue/finished but waiting for yields). It can also be set to supply your buildings, like cotton for the weavers. You just set how many turns you want to stockpile for and the automation will deal with the rest. You build a new building with an extra working slot and the import threshold increases according, fully automatic.

There are a few issues though. It doesn't scale as well as I would like. It has no sense of distance meaning if one city needs tools from another city, it doesn't care if it takes 2 turns or 15 to get them. I'm not sure what to do about that. This is a general feeder issue.

The other issue is that even though it's written with the AI in mind, the AI doesn't activate this yet. The idea is that a single plotgroup (defined as cities connected by road, not blocked by lack of open borders or hostile units/owners) should kind of act like one big city. The AI would be perfectly happy mining ore from a bonus ore and then transport the ore to a different city, which has ironworks. I have a few ideas on how that can be done, but I haven't actually done much towards implementing it.
 
Another thing which I noticed yesterday is balancing. Oh Lord, we're in dire need to do something in this area.

There are a few issues though. It doesn't scale as well as I would like. It has no sense of distance meaning if one city needs tools from another city, it doesn't care if it takes 2 turns or 15 to get them. I'm not sure what to do about that. This is a general feeder issue.
Yes, I understand. This is one of the most complicated things to properly set up.
Actually, I am thinking about some kind of "cascading" transports. Say, we have four cities:
A(10) ---- B ---- C(5) ---- D(3) (parentheses indicate offer and demand)
where A provides a certain yield which is needed in say C and D, it may easily happen that the bloody transports go from A to D to return to A to pick up the load for C... and so on.
The human player would try to transport from A to B, from B to C, from C to D.

I don't know if plotgroups would allow for some kind of investigation if cascading is possible. Does the plotgroup provide information about the distance to the next city?
If not, I think a vector for each city containing the information about the minimal distance (in turns) to each other city might be a solution.
 
I don't know if plotgroups would allow for some kind of investigation if cascading is possible. Does the plotgroup provide information about the distance to the next city?
No, a plotgroup is actually a very simple concept. It is an array in CvPlot, which tells the ID of the plotgroup for each player. Each plotgroup starts from a city. When adding a plotgroup, all plots next to it are checked if they are valid (road, no enemies etc) and valid plots are added using the same plotgroup ID. They too will check neighbour plots should be added and so on. It can then be used to check if two cities are connected by checking if they have the same plotgroup ID.

In short, it's a bool check to tell if there is a valid road connection. Since it is comparing two ints, it is nearly instant (much faster than any alternatives), but it doesn't have travel time or anything fancy like that.

Come to think of it, maybe it would make sense to make a plotgroup system, which spreads on railroads only. If two cities are in the same group, then a train can travel between those two. If they are in different groups, there is no point in calling the regular pathfinder to try to find a route, because there aren't any.

If not, I think a vector for each city containing the information about the minimal distance (in turns) to each other city might be a solution.
I thought about that too. The idea is good, but what if you build/destroy/upgrade a road? Maintaining the correct values could be a nightmare.

I have been wondering about a simple solution. If two cities are in the same plotgroup, the distance could be estimated to be max(diffX, diffY). Sure it wouldn't be optimal with U shaped roads, but it's fast, simple and will not have to rely on a cache, which can be outdated.

Another concept I have been thinking of is declaring certain cities to become hubs. Each non-hub city will then have a pointer to a hub city, which will be the closest one (maybe you can pick from a list). A transport can be assigned to a hub (home city concept) and it will then deal with transports between the hub and the nearby non-hub cities. Other transports can then be assigned to inter-hub transport.

This would require some clever coding where a yield can go A-Hub1-Hub2-B, which I haven't really come up with yet. However if this works, then automated transports will become more efficient because if a city imports, the yields will most likely come from another city using the same hub. Long distance transports will gather up plenty of resources between a few selected spots meaning if 3 cities each send 2 units units of yields, they gather up to become 6 and that will fill a train. The current system makes wagon trains and trains the best transports because the inability to really use the cargo capacity makes speed more important than size. A trek will not really be that useful, but it could be between hubs.

Sea transport
Another issue I would like to look into is to make some sort of automated transport, which will only go between cities on different islands/ cities not connected by road (plotgroups?). If I have say 5 port cities on one island and make a city on a one city island, chances are that if I automate a ship, it will go between the same cities as my land transports. This makes using the small islands way more complex than it has to be and I end up not really using them.
 
I thought about that too. The idea is good, but what if you build/destroy/upgrade a road? Maintaining the correct values could be a nightmare.
I don't think so.

In CvCity::doTurn() we touch each city anyway.
What would be needed is a "marker" (like bool bCheckRoutes) on the city level which is set when the route calculation has been done. As long as it is set, there is no need to recalculate.
Next we would need a general marker on the player level which will be set in case of clearing a feature or building of some kind of road.
In case of pillaging a road, this marker would be set for all players.

After all, I think that the resulting number of calculations would still be significantly less than currently when all transports are calculating what might be the worst thing they can do.

I have been wondering about a simple solution. If two cities are in the same plotgroup, the distance could be estimated to be max(diffX, diffY). Sure it wouldn't be optimal with U shaped roads, but it's fast, simple and will not have to rely on a cache, which can be outdated.
This idea I don't like. :p
I have made mountains impassable which would cause some conflicts with that approach.
And even if we're not thinking about mountains, this could go really south in case of larger inland lakes or even oceans.


Another concept I have been thinking of is declaring certain cities to become hubs. Each non-hub city will then have a pointer to a hub city, which will be the closest one (maybe you can pick from a list). A transport can be assigned to a hub (home city concept) and it will then deal with transports between the hub and the nearby non-hub cities. Other transports can then be assigned to inter-hub transport.
That is the typical human approach. :)
Good idea, but it would require additional calculations to determine when the bigger, slower transport trumps the smaller, quicker one.

Sea transport
Another issue I would like to look into is to make some sort of automated transport, which will only go between cities on different islands/ cities not connected by road (plotgroups?). If I have say 5 port cities on one island and make a city on a one city island, chances are that if I automate a ship, it will go between the same cities as my land transports. This makes using the small islands way more complex than it has to be and I end up not really using them.
Why not having a "city based" automization, too?
As far as I see it, this could be done with the current transport manager already. All which is needed would be a "transport everything" button on top of the yield list.
As you do it now already, you select source and destination city, but instead of selecting the different yields to be transported now you just select "transport everything" and the transport runs back and forth between both places.
Source and destination then of course should be excluded from the "totally automated" transport calculation to avoid unwanted transportations.
 
My plan for improvements:

[*]Figure out a good way to make automated transports handle a mixed land/sea transport network

We probably already discussed this question in RaR. However, I will ask again.

To my opinion the "automated transports" will be much easier if each transport unit could transport many goods of different types.

Now in vanilla and RaR 2.4:

UNIT_WAGON_TRAIN has
Code:
			<iCargo>2</iCargo>

Thus, my wagon is full if I transport only 1 bottle of rum and 1 piece of wood. I have no more cargo space for 1 piece of food or something else.

At the same time at normal speed the CargoSpace of each Cargo-place is equal to 100.

It means that during goods transportation (distribution between cities, selling/buying in Europe, etc.) we don't use about 98-99% transport space. As result intead of 2-3 wagons we must build 20-30 similar transport units.

Are you plan to correct this situation and propose your own variant, when each transport unit (wagon or ship) could transport as much various goods as possible?

P.S. I don't propose to break DoANE code, I'm asking about your own variant.
 
Thus, my wagon is full if I transport only 1 bottle of rum and 1 piece of wood. I have no more cargo space for 1 piece of food or something else.
That's another issue unrelated to distance between cities.I wouldn't mind solving this, but technically it's two problems and they should have two solutions.

P.S. I don't propose to break DoANE code, I'm asking about your own variant.
My experience tells me that implementing something is likely to be best if it is recoded. Even some of the stuff I have coded myself I recode for other mods based on experience with the first implementation, which turns out not to be perfect. Well it depends on what it is. My idea regarding making one dynamic Domestic Advisor, which fits both M:C and RaR naturally didn't require recoding when moved between mods ;)
 
Why not having a "city based" automization, too?
As far as I see it, this could be done with the current transport manager already. All which is needed would be a "transport everything" button on top of the yield list.
As you do it now already, you select source and destination city, but instead of selecting the different yields to be transported now you just select "transport everything" and the transport runs back and forth between both places.
Source and destination then of course should be excluded from the "totally automated" transport calculation to avoid unwanted transportations.
I think it is missing the point where an automated transport can handle multiple one city islands without too much setup.

However the transport everything is a good idea. I wonder if we can simply make it an enum value YIELD_ALL = -2 or something like that.

It goes back to my old plan of making a list of cities where the unit visits all of them in order and it can figure out what to pick up or drop based on city settings. As cool as it sounds, I haven't figured out precisely how to actually do it.
 
Don Senglar posted about the game recommending all plots for cities in the RaR forum: http://forums.civfanatics.com/showthread.php?t=438486&page=12

I kind of like the approach where the threshold is calculated dynamically to fit each map and it is a likely candidate for adding to RaRE.

However the whole discussion made me think about XML. The problem is that we do not agree on every single XML setting. Still it would be beneficial if we could place everything into git.

I started thinking about adding more tags to XML files where game settings would then select which one we would like to use. However this will require a fair amount of DLL modifications and it would need DLL work for each tag, which would be selectable.

I then came up with this:
Add an XML file like globalDefineALT.xml, except the new one will be ignored by git. This will allow us to have personal XML settings. Say we add a tag called PersonalXMLprefix. We can then write the name of a directory with our personal XML settings.

On XML load, PersonalXMLprefix is used as a prefix to all XML paths. If the load fails, the file is loaded without the prefix. This will allow adding modified files and then everything else with be the regular RaRE files. This will allow one filebase and still allow individual settings.

If we take this one step further (which requires a bit of DLL coding), the default path is read first and then the custom afterwards. When the custom files are being read, it will preserve the original settings for anything not mentioned in XML, making the result a hybrid of the two.

This will allow something like:
Code:
<UnitInfo>
	<Type>UNIT_COLONIST</Type>
	<iMoves>4</iMoves>
</UnitInfo>
This will then set colonists to have 4 moves while everything else is set by the default XML files. In other words, it should be fairly simple to have a number of personal modifications while the overall work is shared.
 
Don Senglar posted about the game recommending all plots for cities in the RaR forum: http://forums.civfanatics.com/showthread.php?t=438486&page=12

Just to clarify:

That issue does not exist in RaR base mod.

That issue only exists in Don Senglar's private modmod of RaR.
(Most likely due to his XML balancing changes to Yields and Terrains.)

Please differentiate between issues that really exist in the base mod and issues caused by modmodding. :thumbsup:
 
I made the first dedicated RaRE commit :high5:

I wanted to include a switch between one and two plot city radius. However looking into it, I realized setting that at runtime could result in a performance issue. Instead I decided to make the switch at compile time, which completely eliminates performance issues regarding switching. Details are in the newly added compile options.txt.

Obviously savegames made with one setting isn't compatible with the other. I added an int to the savegame telling which one is used to make the savegame and if it is the wrong one, a popup will appear telling which flags the savegame was created with. With expansion in mind, I added a system where more flags can be easily added to the savegame mismatch detection.

The game will read different XML values depending on compile options. To be precise there are options for 1 plot and 2 plot for city zoom level and min distance between cities.

Python needed updating too. However that isn't a major issue because I recoded it into asking the DLL for the city size. This mean the city drawing code in python is written to handle both sizes without any issues.

There is a single issue though. Due to game, the popup will not appear on top of the game in fullscreen. Instead the game freezes and you have to alt+tab to see it. I looked around the code for a solution for that, but it looks like the code for writing code on top of the screen in fullscreen is hidden in the exe :(
 
Top Bottom