Yesterday someone posted a "lecture" by the AI programmer of Civ 3+4, which gives great insight. I just can't find it any more. Can someone link it here, please?
Basically, they had a huge conflict between making the AI competitive and making them roleplay as historic leaders (which added to the fun for a lot of players, but generally was dropped for a reason).
One example is religion and it's diplomatic modifier. The Civ4 AI role-played and therefore cared about the religion of the human player, but the human didn't need to.
In Civ5, they went away to a huge extent from this roleplaying, historic AI and made it simulate human behaviour, which makes it easier for the AI to act smart.
@stealth_nsk:
The lecture explained that the Civ4 AI did NOT have lists of build priorities ("build temple, then barracks"), but the AI thought about which basic items they need (culture, production,...) and built the buildings that provided this, no matter what they were called. That made the AI quite resistant to patches and mods.