On top of this, there's the real possibility that the person programming the AI doesn't understand the game well enough to know the difference between a good play and a bad one.
The thing is, though,
nobody knows the difference between a good play and a bad one when the game is still under development. All the AI person can do - all anybody could do at that stage - is ask the development team how they think the AI should act under various circumstances, and then implement a decision tree as best they can.
You can't possibly have well developed AI decisions until:
(a) the rules are finished; and,
(b) the game has been released into the wild long enough for players to figure out the best tactics.
I look at how the AI tries to convert another civ - wave of Apostles, then a wave of Missionaries - and I guess that was the expectation of the development team that a wave of Apostles would clear out the enemy, and then Missionaries would finish off the conversion. What actually happens is the Apostles eliminate themselves using their last charge to convert cities, and the Missionaries get killed in religious combat, wiping out more religious pressure than they spread. In this situation, even some playtesting may have identified this issue, but there may be little time for playtesting of the final rules depending on when the rules are finalized (and in Civ it appears they continue to get tweaked right up to release, which likely also explains many of the variances between actual game play and UI information).
What I think we should be able to expect is an AI that can make use of all of the game systems on release, and then over time continuous improvement of the AI decision making as feedback comes in from the field.
Mostly that's what we got with Vanilla, with some notable absences such as airplanes. Then that model seemed to go out the window with R&F, which released with more systems that the AI can't use (has anyone ever seen the AI use the National Society?) and for which there's been no apparent attempt to teach the AI to play better.
It would be great if Firaxis engaged some of the better players to work with them on the AI before development finishes. Absent that the best we can hope for is as much of the source code as possible, so the community can continue to improve the AI itself.