Firaxis had to give the AI a leg up on the harder difficulties because their CPU budget for AI processing wasn't very big. People already complain about how poorly CIV4 runs on the average gaming computer -respectable AI would have to be running constantly in the background (slowing the game considerably), or else take a few minutes at the start of each turn to do all of the processing.
The problem is that AI code is computationally expensive, and it has to run on your main CPU. With graphics processing being handled by a dedicated processor (GPU), AI is the single biggest resource hog in games today.
And then there's how difficult it is to write good AI in the first place, but you could write a doctoral thesis on that issue.