I think that in the case of how the AI uses the units it has or building units that make sense (either strategically or thematically) there has to be some level of competence. I don't think many people would really want the AI to act like a smart human in terms of them using their navy to attack the player where they are weak. As for challenge I think it is very hard to scale AI up in terms of difficulty levels. It is much easier to just give them more resources. I think a moderately capable AI with more resources would probably be the sweet spot.