My experience is that granaries become less important on higher difficulty levels, combined with small maps. It's possible that you'll have to compete with your neighbours for space already after your first 5-7 cities, and in that case, you'll be able to grab more land if you build settlers early, not only from your capital city, but from any city that has enough population to build a settler. That means that it will take a lot longer for your cities to become powerful, but on a small map on Demi god or Deity level, you might not get space enough for more than 5 cities if you take the time to build a lot of city improvements and let your cities grow, before you build your settlers.
This depends very much on how crowded your part of the world is from the beginning, of course. You'll notice how far away your neighbours are, as you scout around, and make your decisions based on that.
(It was a very long time since I played a game like that, so I might be mistaken when I say 5-7 cities, I don't remember exactly how many how many you normally can build with those settings. But I'm sure you understand what I mean anyway.)
I think standard size maps on Emperor difficulty is the kind of game that I've played most, and with those settings, I usually build at least one settler, maybe even two, before starting on the granary. I'm more likely to build only one if I have a nice food bonus in my capital, and I'm more likely to build two settlers before any improvement, in case I find a couple of nice spots with cows for my second and third cities.
And then there are those games that I like to play nowadays, of course. 20k culture on Warlord level, on a small map. In those games I never build a granary, nor a settler in my capital.

If I don't find a free settler from a goody hut early, and get a scientific leader (to build the Pyramids) within the five first techs that I research, I just feel that it's a lousy start, and I start a new game.
