Yes, but an increased difficulty can be achieved by boosting Production/Science/Culture etc, which, in my opinion, is a much more elegant method.
When I play without AI starting units, it does not impede the AI's ability to grow. The game seems to progress very naturally.
There are two possible reasons that I think that the AI may have been given extra starting units:
1) By starting behind, with only one city, it means that if the game's set difficulty level is too easy for the human player, then the game will at least feel challenging for a while longer.
2) To prevent the human player launching an early war and taking out an AI player. Knocking out a neighbouring rival and taking their city so early would make the rest of the game too easy. So the extra units are there to make sure the AI survives the Ancient era and can get going.
Neither of these reasons persuade me to give the AI the conspicuous starting advantage of extra units. Firstly, I think it's intrinsically better to have an AI that can roughly match one's capabilities, rather than one which is slightly worse, but which is given an offset, with the extra units, to compensate. Secondly, I'm not generally very warmongery and so don't immediately set about conquering my neighbour. In any case, my "Tough AI" mod would make it harder to do so.
A wonderful topic that comes up every so often on here...
They
do get boosts to production/science/etc. Up to like, +80% on deity. But this doesn't really matter because they are not capable of
playing the way humans can - namely we know to rush certain things and what to skip. For example, many players on higher difficulties will either commit to religion or ignore it entirely, and thus not waste precious early hammers on holy sites. The Ai doesn't know about this, so they spend
a ton of time on holy sites. One civ that is almost always very strong on any difficulty is AI Korea, and even though they can't use Seowons (they will fill districts around them) Seondeok can easily run away with the game because her Ai is weighted to build seowons everywhere. AI Korea is actually an amazing case study because it shows that if the computer was programmed to follow a more "meta" strategy then they would be much more fearsome.
Consider looking up that mod that bans the AI from engaging in holy sites and see how much better they do.
As for why they get extra units instead of
even higher boosts, that is to make these high difficulty AI extra fearsome at the start. You literally start far behind them. This is part of the point of high difficulty - to be unfair. The extra settlers, though, is what almost always makes players who aren't used to it either quit a game or like you, mod it out - it only takes a short amount of time to overtake them even with that advantage, but the psychological factor is definitely there and
works by virtue of that mod being used. So that matches up with your #1 and #2 quite well. But one risk of having really high boosts is that you can accidentally create runaway conditions where an AI player begins engaging in some action that makes the game unwinnable - imagine a 200% production boost combined with a neighbor constantly throwing warriors at you from turn 10. And that's not a good thing.
Of course, everyone would love a more competent AI opponent, but they simply haven't made one. When they release access to the core files once they are done with content for the game (which they traditionally do anyways) I am pretty confident it won't take long for very scary AI overhaul mods to pop up. Because it's literally just weighting that keeps every AI from being as scary as korea on tech and Pericles on culture. And if they ever figure out how to make it so they keep producing units during wars, then it'll be really freaking hard.
