Take a look at the AI in games like Total War and then tell me the technology isn't really there yet...
Well, Total War would certainly suggest the technology isn't there yet. TW has a simplified, Civ I-IV style diplomacy system that the AI struggles to handle. Its tactics are unsophisticated and barely change with difficulty level - instead, as with Civ, it just gets more units. AI army design is rudimentary and always focuses on the same two or three unit types. It makes little use of agents for much of the game, and when it does use them it can be annoying but exhibits no selectivity in target selection.
It has the same rudimentary understanding that it should flank as Civ V AI, but little more than that - it will charge headlong, will rarely disengage to choose other targets (though on Hard and above it will sometimes do this with cavalry, but only cavalry), will either not keep units in reserve at all or will simply stand around with 'reserve' units while its fighting units are being defeated by enemies with exposed flanks (especially in naval combat, at which the AI is atrocious), it won't use its units' special abilities at every opportunity (fire arrows right at the start of a battle, but never subsequently, for example; and it never uses schiltrons/phalanxes, a particular problem since it's so incapable of responding effectively to enemy cavalry), and it has no idea at all how to execute a siege or to coordinate attacks so that defenders are engaged simultaneously (and the order in which it attacks is heavily stereotyped). AI units will invariably be set to run to the point of exhaustion, and will never withdraw to stop and recuperate (although in Shogun 2 the game pace is too fast for this to matter a lot, still the default run setting means AI units are less capable when they reach combat than they should be). Every TW release is accompanied by fans waiting on a "Better AI" mod.
Tellingly, almost none of this has shown any improvement since at least Rome (although I haven't encountered it doing anything quite as stupid as Medieval II's favoured siege tactic of sending battering rams and siege towers - one by one - to be shot by defending archers while all the infantry and cavalry stands about out of range) despite an increase in AI turn times over the same time period (and, in Shogun 2, a drastic reduction in the complexity of the strategic level). Indeed some of it seems to have got worse - Shogun 2 seems much worse for basic rushes than Rome or Medieval II, where AI units would sometimes actually withdraw if attacked by a superior force, and would actively seek cover to hide in to set ambushes. Empire's AI was so notoriously bad that it's the major reason for the game being considered the weakest in the TW series (and likely the reason Creative Assembly rushed the semi-sequel Napoleon out within a year).
AIs might well be able to do better than Civ V's on technical grounds, but comparing Civ AI with Total War games, which boast what may be the most famously bad AI in strategy gaming, is an odd choice.