It doesn't fail, it's just incredibly complex to make something with as many components as Civ perform in a way that humans will find complicated for long.
Even if the AI didn't have these issues, it would ultimately be overcome by learning it's tactics and adapting to them, something the AI would be unable to do.
Why are people expecting intelligent dynamic tactics from a machine? This expectation wasn't there when the AI could stack it's way to victory, or defend a city in such a way that it could wear you down over time regardless of your military strength.
How come brute force is regarded as intelligent but finesse isn't?
Given the complexity of the task, Civ performs extremely well. Is there room for improvement? Definately. But given the state of the game at inital release and the state of it now, you can't say that the development team aren't moving in the right direction.
As with any modern game, it develops over time. Civ 5 will most likely be out for a good 4-5 years more before a successor based on the longievity of it's predecessors. A complete overhaul of the tactical AI isn't something that you're going to patch as it's probably a hefty part of the program. The nuts and bolts are easily replaced, the ship can keep on sailing with a couple of loose ones, the AI itself is like replacing a bulkhead.
In the meantime, either tolerate its shortcomings and enjoy it for what it is, or get over yourself and develop adaptive, creative, tactical AI that demonstrates both environmental awareness whilst predicting numerous strategic outcomes of it's actions and learns from it's own compound experience yourself, I'm sure the military would fund you rather well for it's invention.