I read an article not too long ago on the new AI dev working on GalCiv3 AI. He came in and made the AI substantially better win just a few weeks. I believe he stated that he doesnt like moddable AI bc it is generally much harder to make it competent for the developer. It shouldnt be a modders responsiblilty, I paid Firaxis and they should be the ones to do it right.
And civ AI actually isn't very moddable either, despite their efforts. It allows some tweaks here and there, which while it can provide some minor gains, just isn't anywhere close to what you can accomplish by just coding it better to begin with.
For example they use a sort of priority system for unit moves. There's a list of actions bots can take, like "attack high priority unit", "use promotion", "attack city", "move to safety" where it always appears to just pick the highest priority action available. And modders get to shuffle the priorities around or remove certain actions (on a global basis, not in a way specific to individual units/situations). This offers some miniscule control for modders, but at a gigantic cost.
A simple priority list like that cannot possible come close to clever behavior. It completely ignores some of the most fundamental aspects you need to make proper decisions. Attacking a unit means your unit is now going to have to end the turn on some tile. This can range from perfectly fine (no enemies around) to absolutely horrendous (you just put your 6 promotion army with 10 health next to an enemy city). A priority list like that doesn't care, it just sees you can attack a unit, then does it because it's at the top of its list, goodbye army. Allowing modders to shuffle priorities around is completely nonsensical when they should've just made something that actually takes in important information and makes meaningful decisions. I mean yeah, mods can improve the game, turning "chase target" off completely made the AI more likely to take cities because it'd just get distracted all the time otherwise. But how is that at all relevant when melee units frequently completely ignore cities on 0 health because they literally don't care at all when city attack is not at the top of the list (and if you do put it on top, they will ignore all defenders/promotions/pillaging/running away).
The systems in place for 'operations', which are groups of troops with some goal, production choices logic, diplomacy, all of that is build on similar systems which are mod-wise exposed enough to be able to tell there's little else going on (otherwise my changes wouldn't do anything), but which aren't exposed enough to actually be able to do something meaningful with it.
In contrast, a game that does AI moddability right is Dota 2, it just has its own internal c++ bot code, but allows modders to turn the entire thing or sections of it off, and use your own lua code instead, complete with a decent API that allows you to write whatever you want. The bots written by the devs aren't constrained by moddability and also seem significantly more impressive, especially considering it's a harder game to code for. And modders still get the best too. You get to completely write your own thing, and people can download your bot using ingame tools and play against it. No dll hacks required.