I don't think it's unreasonable to work a granary into the queue before your settler as well, especially if you're skipping the shrine, and certainly if you're working wheat / deer / bananas. That extra bit of early growth is nice, and then the extra food actually become hammers when you're building the settler.
It's almost never worth it to build a Granary before building your Settlers. The conversion from food to production is not linear.
Someone please correct me if I get this wrong, but iirc this is how it works:
1 excess food will convert into 1 hammer, 2 excess food will convert into 2 hammers, 3 excess food will convert into 2.5 hammers and 4 excess food will convert into 3 hammers. after that investing into food really is not worth it at all: 8 excess food converts into 4 hammers.
so what does this mean? assuming we have a pop 3 city with 2 hills and 1 wheat/deer, a quite realistic scenario:
a settler is 106 hammers. in order to get your 4 city tradition going you will need 3 settlers. in total that makes 318 hammers.
you get 1 hammer from the palace, you get 1 hammer from settling on a hill, you get 4 hammers from working 2 hill tiles and you get 3 hammers (2 food, 1 hammer) from working your wheat for a total of 9 hammers.
this means you will: get the first settler out after 12 turns, the second after 24, the last after 36. assuming you start somewhere around turn 20 to turn 30 that means all your cities will be settled at around turn 65 to turn 70.
now we do the same calculation with the granary: you get 9 hammers baseline, but with the granary you get an additional 3 food (2 for the building, 1 for the wheat/deer). those 3 food will convert into 1.25 hammers.
(once again, please someone correct me if I'm wrong)
so with the granary your city will be at 10.25 hammers, but will need to invest an additional 60 hammers, meaning the first settler will be significantly delayed.
this means you will: get your granary out after 7 turns, your first settler after 17 turns, your second settler after 27 turns and your third settler after 37 turns.
so even if those +3 food accumulate for almost 30 turns (!) your last settler will still come later.
now of course building a granary is not bad if you only need a few more turns to go from pop 3 to pop 4, but most of the time growing from pop 3 to pop 4 takes quite a while since 3 food tiles aren't abundant. even then another calculation has to be made factoring in the fact that you start settler production much later and I will not get into that. I could've taken improving mines with a worker or chopping forests into account but I'm terrible at maths and this was already too much, so please go easy on me.
In any case though I would much, much rather make an Archer or even a Warrior in order to have an additional unit to
a) escort me Settlers or stolen Workers
b) protect my Workers that are improving mines/luxuries
c) protect my tiles from being pillaged
d) do early city state quests
e) clear barbarian camps near my city asap