Let us not forget who we are here. This is the "civ FANTAICS" forum. We are the people who spend hundreds of hours playing the game, looking under the hood, and crunching the numbers. Many here have have figured out how to beat the game, the next DLC, the next expansion before its even released. We already know pretty much every move the AI can possibly make then complain the game is too easy In a game such as Civ6, only another human could provide any real challenge.
I would guess the majority of the market is made up of the casual player and not the fanatic. The casual player is not looking for the same challenge. The casual player wants to get in, get out, have a little fun and move on to a bevy of other games. The investment necessary to create an AI capable of challenging the fanatics must be considerable as well as counter-productive in light of the market share I described above. I am simply not sure that an affordable AI is currently possible for our needs; not to mention one capable of running on the typical desktop.
The real failure of the current AI is its inability to learn and predict. In a game as complex as Civ6 and with such a broad scope, no AI can hope to match wits with a dedicated human player. The AI is simply a series of algorithms with mathematical formulas looking at hard values weighted against other hard values. Is it any surprise that the dedicated human mind can beat that? We are far more complex, nuanced and unpredictable. We can and do look under the AI's hood, but the AI cannot peer into the mind of its human opponent.
That said, the sheer complexity and scope of the game is, to me at least, the real enjoyment. Every game is different. Even if I hand pick all the same conditions from one game to the next, they are different. In my opinion, that is the real testament to Civ6 and the developers.