Ok, to write it down a bit:
1. Overall the goal of being competitive with minimal bonuses is wrong. AI is just one of the game mechanics to provide challenge and how interesting the challenge is doesn't depend directly on whether it's done through more bonuses or more complex logic. So, all those mentions of "good" AI should be made in quotes.
2. What people usually mean by competent AI is what it appears competent. If AI moves units without visible goal, the players are usually frustrated even if there's actual goal. Nut that's part of the immersion, not game efficiency and immersion is a double-edged sword. For example, Civ5 was of often criticized for AI playing to win too much, while players expect it to roleplay.
3. In general yes, it's possible to make AI more effective in terms of providing bigger challenge with less bonuses (but again, with as complex tactics as in Civ you can't do it without bonuses at all). But it's not just a non-goal (see point 1), but also requires significant developer resources which could be spent elsewhere. It's wise for software development company to prioritize those things.
4. Another important thing is - if you want AI which would look smart utilizing other game mechanics, it needs to be written after they are finished, but in game development all mechanics are developed in parallel, so you can't fine-tune AI with the latest state of the game and you have to make AI more generic or spend additional resources to sync those mechanics. That's a huge advantage modders have - they could polish their mods years after last patch of the game.