You do realize, that if they do the "perfect" AI, the game would become pretty much unbeatable on higher levels, right?
That statement makes no sense.
The fact the AI is coded to cheat is a direct result of an attempt to make something stupid be tactically and strategically difficult. The difficulty is set by Firaxis, it didn't magically come into being. If they changed the AI, they could just as easily change the cheating (and should have, with the last two expansions).
Do you know all those grand-master chess matches where one of the players randomly spawns extra pieces? How about those football matches where one side has extra players on the pitch? You don't, because it doesn't happen. The game's difficulty is based on the skill of the players. The game is otherwise equal and fair, that's why it's a game. In some games when you know the players are just not at equal skills, you provide some sort of handicap for the inferior player. In this case, the AI's cheating is its handicap.
What bonuses the AI gets is purely set by Firaxis in regards to what they felt met their level of difficulty. Or, did, back when they created it.
The issue here is, the AI fails at everything. There is not one thing the AI does well. It can't fight, it doesn't build well, it rarely has any strategy, it's never successfully prosecuted a war past its own borders. It's just 100% terrible.
The OP's dream would be that the AI at least develop some level of either tactical or strategic ability. As it is, the whole game is just horde mode where some of the zombies are not combat zombies. The difficulty level you choose to play on is purely a result of how much clubbing of mouth-breather AI units you can manage in a period of time.
On the lower levels (king or emperor and down) I think it's less of an issue. You know you're going to win, it's just a function of exploring and building and the AI is off doing its own thing. Emperor seems to be about the balance of where the AI's building failures match their bonuses. On the higher levels, it's mostly just a war game that is pretty terrible in comparison to 4x games, because the AI is unable to wage any semblance of war beyond zerging with "overwhelming" forces.
So, I think it's fair for the OP to complain the AI is terrible. It is. How difficult it would be to improve, I have no idea. I imagine it must be pretty painful, else they would have done it; since the terrible AI is functionally the deal-breaker for everyone I know who loves 4x and hates Civ V. But as for the viability of it? I think it's in their court for who's at fault. Total War is way better, Most of the Paradox games are better. When someone decided on changing the franchise to a new unit method, they should have investigated whether they could actually make it work first.
I would propose the true issue is, a lot of people play on Deity who really don't really want to do it, because of achievements and such. Deity really just enhances all the breakdowns in the game. I think things would be a lot better if the only people who kicked into deity knew they were facing a boring seal-clubbing scrum. Then, there would be no reason to complain about it.