Coming from a more technical background and having done some modding on this game, I understand why they cannot improve as much as we want them to, so I'm not particularly disappointing that they can't improve it all the way up to deity level players. And without that deity level player, you will need some form of handicap to make the game interesting for the best. So you might as well have that handicap in place for the entire spectrum of skill. I don't expect that this will change at any point.
On top of AI just being extremely hard to do for any strategy game, civ has some additional difficulty factors when it comes to how civs build their empires. The main one is that the AI developers, will not know what the game actually ends up looking like. During the development process, the costs, effects, requirements, etc of all the little building blocks they have (buildings, units, techs, etc) will change. And worse (for the dev), they will change with new expansions and player mods. So the system they chose is one that has to still work if completely new buildings pop up, effects change, costs change, requirements change, etc. Humans can and will adapt to new or changed stuff, and the AI will also have to.
It makes far more sense for the company to build such a 'flexible' AI alongside the development process, rather than wait for every single balance change etc to be done before even beginning to build an AI that would then still fall on its face if content is added/changed. We're not talking about the AI just being bad at choosing when to build new stuff, they probably just wouldn't ever build it all if it wasn't accounted for.
Additionally, the CIV team has put a big stake in the different leaders having noticeably different personalities. All civs have their own unique, subtly different playstyles, and that is really noticeable, flavorful and fun. A more 'competitive' based AI likely wouldn't have a lot of freedom there, itd be very hard to notice differences.
And the last factor is that they make their AIs not just as competitive machines, but are made more to emulate what we kind of expect from real world empires. If the AI were truly competitive based, then the correct strategy to win would most likely be to just all instantly band together and DOW the real player.
So, I think the 'flavor' based system they picked for the AI is actually a decent choice, although it has its failings. For those unfamiliar, the empire building AI part of CIV 5 is based around flavors. Every building, tech, civic and unit has certain flavor values applied to it, say a granary has a food flavor attached. Every leader has its own preferences for these 'flavors' and will prioritize buildings accordingly, on top of that, there are 'strategies', which are triggered under certain conditions and will change the desires for certain flavors, these are things like: 'My land has few improvements' -> increase the 'worker' flavor. 'I'm doing the space victory' -> increase the desires for 'science' and 'spaceship'.
This allows great flexibility, because adding a building just requires setting the flavor values of this building appropriately, and the AI will handle it like any other. And on top of that, it is an efficient usage of performance (the bulk of the ai turn times actually goes into pathfinding, hence the slowdown near the end of game).
Essentially all the AI does, is look at its current options, look at its desires, and pick whatever building option best matches it desires. It works pretty well, but does have some huge flaws, which I hope will be addressed directly, or which will become moddable.
The worst is that the AI cannot do planning. At all. Even on the simplest things, it just doesn't. You might say the AI is always surprised at what it encounters, if an AI builds a wonder that gives it a great person, it is not because the AI decided it needed a great person at this particular moment, with say a tech or building to rush in mind. Nope, that wonder just had a 'great person' flavor attached to it, and this personality likes great persons. And after building it, the AI is basically surprised that a great person pops up. It cannot predict that it will come, theres no plan, and then the ai just does whatever its shortterm based mind tells it do.
This means that the AI is extremely limited in its ability. It can never match a humans ability to craft strategies, because all it does is pick whatever sounds nice. Some humans do basically do the same, so luckily, it does still manage to keep up relatively well with inexperienced players. It might at least have a bit better of an ability to decide what 'sounds nice'.
The other major problem, which can be blamed on the developers. Is that the flavor system is unfortunately extremely halfassed. The flavor values that have been picked for buildings/techs/units/civics look like someone just kind of thought of a number, not really realizing how it plays into the game. Give the library a science flavor of 35, sure, why not, what does 35 even mean anyway? A market gets 50 in the gold flavor? Why not? A factory does a lot of production, why not 100 in production? 100 sounds like a lot.
Honestly the only reason why players can keep up in deity at all is because of these halfassed values. The library is probably the biggest offender. While players realize the library is probably the most important building to build for just about any strategy, partially because of how strong it is, partially because of it leading to the national college; AIs dont really care for it. Its total sum of flavor values is one of the lowest in that era, so any leader who does not have science as his top priority will not build it untill it literally runs out of other options to build. Because of that, there is almost always at least one city that doesn't have it yet, so the AI never builds national college. Even if magically it does gain access, it will likely build in whatever city happens to be the first to see it pop up on the list of available options. I've looked at a lot of AI games, national colleges pop up in maybe about 1% of AI capitals below deity, and that's usually sometime after renaissance. In deity it exhausts the options much faster, so in small empires it does get build sometimes, but even then usually pretty late. Since the AI gets no science bonuses, it's no wonder that good players can catch up.
Changing a single value, library science flavor 35->100, probably puts peaceful deity victories out of reach for 75% of players who can currently manage.
So the college thing demonstrates that the flavor system does not account for things being requirements for other things, which is very noticeable in other areas. If there's a civic that is amazing for gold, but its locked behind another civic that has no influence on gold, then unless someone decided to attach a small gold flavor to the requirement, the AI will never get there if its just basing its decisions on gold. Some of the civic tree openers have terrible flavors, so some trees barely ever get picked.
It also ignores some things humans think of. We realize that happiness is good for growth, and growth is good for science, so at some points a Colosseum may be the best option for science. The AI will not realize this. While some minor attempts have been made to correct this (a granary does have a small science value attached), it really lacks in this area for a lot of buildings.
Are you seeing your AI spam way too many anti aircraft guns? Its simply because the flavor/desire system ends up delivering that result. Are you seeing hundreds of aircraft carriers? Well, its because it has a decently high overall total spread out over the flavors 'offense', 'defense', 'naval' and 'aircraft'. Is the AI being attacked? Ok, we need defense and we lack air power, lets build some carriers. The AI isn't even looking at the amount of aircraft it has, it just looks at flavors and some random factors. The randomness is pretty high too, to a degree where it outweighs strategic and flavor aspects, so that all AIs build units of all types in all situations.
Why is the AI building more cavalry than riflemen? Because the total sum of flavors for cavalry is 23 (10 offense, 6 defense, 7 mobile), and for riflemen its 20 (10 offense, 10 defense). So unless the AI highly prefers defense over mobile, it'll make more cavalry. Catapults (offense:3, defense:3, ranged:8) will be build more than composites (offense: 3, defense:3, ranged: 6), the AI doesn't care that they have different functions or that composites tend to be better overall. The list goes on and on.
It'd be really nice if the devs gave their flavors some additional passes, with actual testing etc. The AI would end up way more powerful.
And as a modder I would love it if they gave the option of creating our own types of flavors for more specific behavior (like, give certain leaders huge boosts to certain techs so they'll do bee-lining)
So that ended up being way longer than I anticipated, I hope it was educational at least!
tl;dr:
The AI in civ 6 can't be much better than civ 5 assuming they use a similar AI system, which they probably will. But, there's a good amount of room for improvement even within that system. Perhaps as much as 1 or even 2 difficulty levels worth.
