I think this thread is getting mixed up in connotations of words.
There are two types of ways to make an AI harder.
The first is to make it play with a better strategy/tactics. This is ideal since all players play by the same rules, and although some luck is involved each player's strategy and choices play the strongest factor, and superior strategy/tactics will generally win despite some really unlucky situations. Even with these though, all players are equally likely to experience them. So if the default strategy is for the AI, when at war, to immediately send all military units to the nearest enemy city as soon as possible, the difficulty could be scaled up to add batching groups of units to attack cities. As for strategy, if the default is for the AI to go for military all the time, an example of increasing difficulty would be to have each civ go for a victory condition suited for the particular CIV strengths.
Obviously these could further be increased, but the point is made. This type of difficulty increase is best illustrated in Human players by playing against myself as the default and to increase difficulty playing against Maddjin.
While this type of difficulty increase among players is perceived as a challenge, and the ideal way to scale difficulty, it doesn't translate well to an AI, which, if I understand correctly, "thinks" by applying a pre-set number of rules that may have a random element applied to it for variety. With as many possible choices to make, it is difficult to find "best" rules for AI's, because there are multiple ways to win, and the optimal path may change any given turn.
To compensate this, Civ used the 2nd way of increasing difficulty - adding bonuses or handicaps. We look at the AI and say "it seems the 3 city tradition opening strategy is optimal, so for the AI to be competitive we should make they always try for it". This could also apply to other strategies as well. If we do code the AI that way, then there is no variety of surprise in the early game. It will also influence the later game. It also doesn't allow the AI react to counter-strategies to that opener, and unusual choices made by other players. To allow the AI to have a variety of strategies (including sub-par ones) they are given bonuses and humans are given handicaps. While this makes for a more interesting and challenging game, it does not create a fair game because each player (human and AI) are not on equal ground, and strategy is no longer the sole factor influencing the outcome of games. An AI starting with techs, soldiers and a settler will have an edge. This still allows the human to beat it, but the human will have a harder time. The bonuses are given as a way to skip programming AI strategies that require variety. "Well the end result of a good opening strategy is increased research and a few early cities, so we don't care how the AI gets there, so well just start them with the end result and skip the futile programming for them to do it on their own "choices"". These "bonuses" would be the equivalent of cheating for a human because you are altering the rules of the game to inherently favor another person. To apply this to chess, its not cheating for for Gary Kasparov to use his best strategies and wipe the floor with me, but it would be cheating for him to start with extra pieces on the board.