Well internally the way it works is relatively simple.
It always calculates a score for each of the improvements, and then decides which improvement is best based on that.
What's unfortunate is that this calculation doesn't care about a lot of information humans find meaningful, such as trying to find the right balance between production and food.
Basically if it considers the food+housing on a farm more important on one spot, itll consider it more important on all spots.
Modding wise, unfortunately there's not too much we can do about it as far as I'm aware. Usually calculations like these are tied to the desires for the different yields, which you can alter through AiFavoredItems. However, it appears this particular calculation is rather insensitive to these desires or may not even use them at all. I've tried making Gilgamesh stop building his ziggurats all over the place to no effect.
I've not done very extensive testing though, and it may still be possible. A few places you look at:
- Using the 'improvements' ailist system to give a direct boost to certain improvements
- The yield desires in Aifavored items, look for YIELD_FOOD etc.
- The Build City Improvement behaviortree
- Maybe the city specialization favoreditems (BUILD_FOR_PRODUCTION in leaders.xml)
- There might be an AiList system for improvements that could give direct boosts to certain improvements though I can't see it being used in the base game