What you did to improve on the pitiful Firaxis AI is simply amazing. I wonder if you are just a genius, or Firaxis are incredibly lazy. Every single one of their otherwise great games has horrible, horrible AI. Basically non-existent. Every Civ game as well as the XCOM games. But if one man alone can create such a good AI, shouldn't a team of pros backed by millions in sales be capable of delivering something similar when the game ships? Instead, Firaxis never even tries. It seems obvious that they simply do not regard AI as important. Which is a complete mystery to me, although I'm afraid their success with players (and reviewers!) proves them right – most people do not actually want to be challenged by the AI. In reviews, this topic usually doesn't even come up!
Its because the game companies today are too lazy/incompetent to actual sit down and code the AI, they make money by putting together a graphical theme park and shipping it off asap so they can start on the next product.
There are a couple of factors at play with something like this.
It's true that the most important aspect is the cost/benefit analysis, which is not entirely the same as laziness or incompetence. You can't really use AI in marketing (or rather, you can just put some boilerplate like "The AI will react dynamically to player action" in your press releases and its fine no matter the actual content). Everyone can lie about their AI, until the game is out nobody is there to verify it. On the other hand, extensive new feature lists, shiny graphics and new content (playable civs and the like) can get people excited early on. Unless there is the rare massive backlash as with No Man's Sky, early hype is much more valuable than hindsight reviews, just look at the success of Civ5 compared to its reception after release. It's just that at every point of the development cycle developers are encourage to allocate resources to things that are not AI.
(Not to mention that being easy is apparently a plus with a large enough audience to make it an overall benefit - but that's another discussion.)
On the other hand, I don't know how many hours karadoc has poured into K-Mod but I'm pretty sure it exceeds what your dedicated AI programmer would put into the game overall throughout its development. And K-Mod is already created on top of all the basic AI that person created. And remember that "basic AI" includes stuff like pathfinding etc. There's a lot of groundwork to be done first before you can even address something like strategic choices.
Lastly, often what is or isn't good strategy is often not known at all, or only very late in the development process. Developers are usually not the best players, and are usually mentally stuck in a way of thinking about how the game is "supposed to" be played. The best strategies only emerge after outside people come in, and only then AI programmers could try to help the AI follow similar strategies. Especially during Civ4's time it was really hard to get large scale beta testing done, and even if you do there is little time to distill information about game strategies that is useful to developers and then have this knowledge filter down to the AI developer, assuming that person isn't currently mired in fixing all sorts of bugs that are discovered at the same time and need to be fixed before release.
I have heard both Brian Reynolds (Civ3) and Soren Johnson (Civ4) talk about this problem at length (Jon Shafer too but he's not the sort of person I'd like to cite as my source). They know their AIs were not very good.
(If memory serves, it was the Better AI Mod itself that recommended this installation method, rewriting the original game files. I stand absolved! /fail)
If that is the case, they were either ignorant or arrogant, in any case there is never a reason to do that. I would also suggest a clean reinstall.
