An average videogame programmer earns roughly 100k USD a year and gets treated like dirt. An AI expert can easily ask for 200k per year, and if Firaxis/2K start giving them lip, they can easily jump ship to a company that doesn't treat them terribly (Google, Facebook, and Intel are always on the lookout for this kind of talent); in real terms, this usually means that if said expert would have to work the same schedules as videogame programmers, they would actually ask for something around 300k per year at least (this includes overtime and crunch that is standard for videogame development). For that price, 2K can just hire two generic programmers to make a superficial OK-looking AI (people don't really have high expectations), spend a bunch on a celebrity voice actors who players will immediately recognize to sell more copies, and still have money left over.
Most AI experts work best when they're working with neural networking architectures (also called "deep learning" by marketing people). In fact, even when such architectures aren't used, being able to rely on heuristics and a static ruleset makes all AI programmers' lives easier; this is why CPP was able to get its decent AI through 4+ years' of work from 2+N programmers (2 main as Gazebo and Ilteroi, N supplement/groundwork as ninakoru, notque, and myself to name a couple), since BNW had a well-established system for unit control. However, all of this goes out the window for a game in active development: fundamental rules can change on a weekly basis, possibly invalidating months' work of neural net training and heuristic development. As such, I would estimate that even with good AI programmers, developing a good AI for a Civ game would require having at least two of said programmers work on the AI from the start of the development cycle. If we are to take Civ6 as an example, it has been in development roughly since the release of Civ5. That's 6 years' development x 2 programmers x 300k USD per year = 3.6M USD extra just spent on AI development... and that's not counting the extra overhead and management costs for a dedicated AI development system. There are way better returns on a 3.6M USD investment for 2K than employing dedicated AI programmers.
That's quite a substantial sum of money. I wonder if they just rely on modders to fix it for free. Do you know if they are going to allow modders to fully access the AI and actually change it?