A creative thinker/modder should be able to overcome the challenge, though, don't you think?
No amount of creativity can overcome the complete absence of certain access points. We CANNOT DO a lot of things that people want, because we simply don't have access to the internal mechanisms through which those are decided. We have a very, very limited set of XML/SQL abilities, and a moderate array of Lua functions and triggers, and that's it. And as I said before, even if you CAN do something in Lua, making the AI actually use it correctly is often impossible. At best, you can hard-code the AI to perform certain actions at certain times, which is not even close to the same thing, especially if you want your mod to work well with various other mods already out there.
For instance, we can't make two AI-controlled civs friendly with each other. We can adjust each civ's relationships with each city-state, and we can make two civs go to war with each other (or declare peace), but we can't modify the relationships any finer than that, because we have no access to the code that handles those behaviors. When the DLL is released this will likely change, but in the current environment, any idea that involves adjusting those sorts of relationships is doomed to failure, no matter how awesome you think the idea is.
This is why I suggest learning how to mod; even if you know you'll never be good enough to make the mod do exactly what you want, you'll at least get a better idea of what is and isn't possible with our current tools.
Also as to the AI issues: could the AI not be "taught" to take only a portion of population not necessary to other uses?
No. We cannot "teach" the AI anything, because the AI behavior is almost entirely internal to the game engine. We can use Lua to force it to do certain things at certain times, in fairly limited ways, but those don't generally mesh well with existing systems and work far better for entirely new ones. We can encourage it to build certain things by adjusting Flavor values, but those are not only very random, but have no room for moderation. And some things (Specialists, Resources, etc.) have no Flavors at all.
For instance, take my own
Age of Mythology mod. I introduce a new Priest specialist that adds Happiness, which is very important as I add severe Happiness penalties relative to the vanilla game. I WANT people using Priests, because those Priests are also a key generator in the "Favor" system I use to unlock new divine abilities. However, I give you far more Priest slots than you'll want to actually use at any given time, because using that many specialists would cripple your production and growth. So a smart player knows to fill some fraction of his Priest slots (say, half) in the core cities, and not use them at all in the small border towns that still need all the food and production they can get. You'll very quickly learn how many are appropriate in each situation.
An AI cannot handle this, in MANY ways.
1> The AI doesn't understand that this specialist adds Happiness, because the game's internal decision-making uses only the yields produced, and Happiness isn't a yield. So it won't want to use any Priests, because it sees them as worthless and can't be convinced otherwise.
2> If you give the Priest enough other bonuses that the AI would want to use it, then it'll want to fill every possible slot, because it doesn't understand diminishing returns. It'll still be basing its decisions on the yields other than Happiness, which means that the Priest specialist is now even more powerful than other types.
3> Because their primary benefit is empire-wide, the AI has no ability to judge how many Priests it should really want to use in each city at any one time, even if it did somehow understand moderation, because too many other variables are involved.
No matter how simple they seem, you cannot "teach" the AI how to handle any of the above in a way that's even remotely comparable to what a human would do. So in the end, I have to hard-code the behavior with a few small randomizations added, basing the exact amounts on my personal experience playing the mod. As in, count up the Priest slots in cities of sizes greater than 3, capping the per-city use based on population, pick a number within a range based on your pantheon of gods, and then force-set some of the Priest slots to "on" in certain cities as a result.
If you're careful enough you can get a playable behavior from the AI, but it's not done through any sort of intelligence. The AI has no clue WHY it's now using those slots. This might not seem so bad, but if you were to pair my mod with someone else's balance mod, then the AI would continue to use the exact same number of Priests,
even if that balance mod changed the Happiness in a way that made Priests now worthless, because it would continue to use my distribution logic. While I could add enough variety in the override logic to make the AI seem to be reactive to changing situations, it won't even come close to matching a human.
And again, this is for an entirely new entity (the Priest), while you're asking for a modification to an existing AI logic (which we can't access). At best, you could use the CityCanConstruct Lua function to tell the AI it can't build any more units once it reaches a certain threshold, but again, it won't really understand why, so you'll be limited only to "hard" caps on the number. That's why I suggested finding some other limiting mechanism (like strategic resources) that the AI already knows how to use; it's much easier to weight the existing logic than it would be to try to override it.
)
