Pardon the techno-speak sure to creep into this post.
Within the last few months, I designed and coded the plug-in, modular architecture for customer written "policy" for my company's medical application. The policy itself is somewhat simpler than an AI is, but the part that made it customizable by the customer was significantly more difficult than a plug-in AI system would be. It took me, working alone, about a month to get the design and code complete, including one major revision to cover things I missed the first go around. For a modern programmer, this stuff isn't Hello World simple, but it's hardly rocket science.
So we are not talking double the work. We are talking about a small addition to the work. The trick, however, is to plan for this from the beginning of the application. I was able to move so quickly because we did not have existing "policy" throughout the application that had to be worked around. Plus, you get some of your development time back, because such an AI is easier to debug and tweak. (It's easier to split up the work, as well.) It
does require that the AI developer(s) communicate with the plug-in developer(s) to get the interface correct. If these developers are not the same developers, that part could hit some snags. (An AI developer that already understood plug-ins or quickly learned plug-ins would have no difficulty, because he would already have a sufficiently abstract view of AI development to know where to draw the lines.)
Moreover, if the modular part is sufficiently robust, the AI can be less than stellar and still be superior to what we currently have. For example, assume that the developers spend another week or two on the modular part so that not only can it be changed, but that it can support multiple sets of AI's during a single game--or even change during the game. IIRC correctly, we have seen essentially three barbarian AI behaviors during CIV3 development: 1) Beeline for cities, workers, or settlers in stacks. 2) Attack a nearby city. 3) Wander around aimlessly looking for a easy pick. I also seem to recall some game reports where players took advantage of a known AI strategy by moving a single defender around. Now assume in Civ4 that a given barbarian stack will approach your territory, then pick one of the three strategies at the last moment. It just go more difficult to game the AI.
In summary, good modular design that allows some customization is a good idea--and likely to save development time in the long run--even if you do not intend to let modders loose. The modders contribution is all gravy.