To me, it's all about the ability to win. The Civ 4 AI always knew how to win on higher levels. The Civ 5 AI kind of knew how to win (but when going science and a spaceship part was built somewhere in the tundra, well.....
The Civ 6 AI just doesn't know how to win. At all. Period. This alone is why people don't finish games.
And remember, steam achievements are deceiving. Civ 5 didn't count achievements if you used *any* Mod (well, other than the mod achievement lol). Well, what's more important, an achievement, or a fun game using a mod where the AI is better. Civ 6 apparently allows achievements with mods, and you still have that few finishes.
I am currently playing a Civ 4 Mod that I never played before, Fall From Heaven 2. I have played so many hours of Civ 4 it's sick, and this has totally renewed my interest. The point of the rambling is the following: the modders have reprogrammed the AI to not only understand the mechanics of the game (including the Armageddon Counter), but also how to cast spells and so forth. And, the AI can cast a spell, create a unit, and use the unit *with purpose* shockingly! Compare this to Civ 5 where without a Mod to this day an archer can't move one tile and shoot.
So, it's a combination of factors, but the AI seems to not only be getting dumber, but to have less ability to be changed to being smarter.
I've had all three games at release. So, rankings for the vanilla versions:
Civ 4 AI: 8 (relative to the other two)
Civ 5 AI: 3
Civ 6 AI: 1.2 (the .2 is for those little early surprise wars at Deity)
A final note. My level of play for Civ 4 took a long time to get to Emperor, and still Monarch is pretty much the best level. Civ 5 it was Emperor and then to Immortal. Civ 6 I've already beaten Deity. This is always on a large map with a lot of Civs, so levels are relative to game settings. The only reason I say this is at this rate by the time Civ 7 comes out it won't even be interesting to me, and I bet there are *many* others that feel the same.
Firaxis, I know that I'm getting older, and that I don't matter as a result. But consider this. When I bought Civ II it was just for me, one purchase. When I purchased Civ 6, it was 4 copies for me and my children. Civ 7 will not have those 4 purchases at the rate this is going.