I don't agree. An improved AI with more reliable path-finding, better management of units and specially groups of units, more effective in air and naval operations, better in managing and obtaining resources and more focused in the victory conditions that suit the game state and the leader agendas will fit the definition of a better AI of everyone here (I can be proved wrong but that would be a surprise).
That being the case, it is possible to greatly improve the AI without making any compromise to any type of game play. An AI more capable of using the current game systems, more focused and better at using units and resources will be good for challenge and role play and will not be frustrating to casual players due to the big flexibility in difficulty options the game has.
PS: I will name this a "It can't be agreed upon extreme extrapolation" fallacy xDD
Ho, globally, we agree , yes.
but when we go into specifics:
"An improved AI with more reliable path-finding, better management of units and specially groups of units" :
IMO just change the 1 combat UPT rule to n combat unit per tile where n is a finite number, n > 1 and n < stack of doom. Then you can improve your tactical AI on those rules (groups compositions per tile), but it will perform better already (pathfinding and formations partially solved) and you can spend more development time on other areas.
But a lot of people don't wan't to change the tactical aspect of 1UPT combined to 2 tiles ranged units, and while, yes, you can make a better AI for those rules, you may have to spend a lot of development time on it (surely less than what the VP's team had spend on it, but still some time)
And I'm pretty sure that if we can make a good tactical AI for 1UPT in x hours, then we can make a better one for nUPT in less hours.
The question about which way to go is then "how much better do you want the AI to be and when do you want it ?"
"more effective in air and naval operations" :
That one I think is the point everyone agrees on.
"better in managing and obtaining resources" :
IMO, again, more a problem of design than pure AI, strategic resources are highly RNG dependent, without a lot of alternate ways to get them (while the AI may require the strategic before being able to conquest it for example)
So first add other possibilities for acquisition than trade or conquest for a player that has none in his territory: small income in strategic from each mines even without deposits, a new smuggler unit similar in usage than the trader, etc...
Then see if the AI still need improvement here.
"more focused in the victory conditions that suit the game state and the leader agendas"
I disagree on that one and I'd change it to "build an Empire to stand the test of time" or I would accept it as a priority if we first change the rules to create different "games states" than the current ones. OFC, it's just an opinion, part of the RP crowd, I don't mean it should be that way, I'm just pointing again that, yes, opinions may differ on what/how.
A zillion years ago I proposed a "Gamist AI" checkbox to differentiate from the regular "Roleplay AI". Back then the series even had an "Aggressive AI" checkbox but it actively made the game easier because the AIs trashed their research while attempting to warmonger.
Remember that one, used it as the AI was not agressive enough otherwise. And it's a good point about difficulties in balancing AI priorities.