I don't think anyone here imagines for one second that creating a strong AI is an easy task. If it were, then every game would have one. But do you honestly believe that Civ has evolved beyond the capacity of Firaxis to give it even a decent AI? We have evidence to the contrary -- Civ V ended up with quite a good AI, largely because because more tools were given modders to make one in what ended up as the "Common Patch." It would be hard to argue that Civ VI is so much more complicated a game that it has crossed some threshold of AI complexity. In fact, I don't think that it is ANY more complex than Civ V -- it just has a few more gimmicks like districts, in which you can build pretty much the same buildings that you could build in V.
What's most maddening -- to me, anyway -- is that Firaxis isn't even trying to make it better. Look at the most recent patch notes. They barely mention improvements to the AI, and those few mentions are, to put it charitably, minor ones. Ok, fine. If we the playing community don't demand a more challenging game, then we surely don't deserve one.
One final observation. I for one do nor see a categorical difference between AI's in rts versus turn-based games. Paradox's games all are rts games, as is well known, and the AI in those games range from "meh" to "bad". Meanwhile, the three games I suggested above in this thread that have superior AI all are turn-based games.