Even without whipping, the granary is a no brainer, especially on the slower speeds. All your cities will grow twice as fast, meaning you can work more cottages or whatever faster.
I don't think so. Lets math this out.
Let's take a typical city with a couple of floodplains plus a minable plains hill and a river to spam grassland/river cottages on. Assume a happy cap of 5 (one luxury resource on immortal difficulty)
option 1: Ignore the granary.
option 2: grow to size 2 then whip the granary
option 3: slow build the granary while working the plains hill at pop one.
I can't chop the granary cause I'm too busy building cottages at this point in the game (when i first get pottery). Plus, I probably need to save my forests for catapults, who knows if i have any to spare. I really have better things to do with a worker than chopping a granary right now.
Option #1
turn 22: grow to size 2. (+3/turn for 22 turns for 66 food total)
turn 40: grow to size 3. (+4/turn for 18 turns for 72 food total)
turn 60: grow to size 4. (+4/turn for 20 turns for 80 food total)
turn 81: grow to size 5. (+4/turn for 21 turns for 84 food total)
Option #2:
turn 22: size 2. 22/180 granary.
turn 34: can whip granary now (after working plains hill plus a grassland forest for a few turns).
turn 56: size 2 again.
turn 65: size 3
turn 75: size 4
turn 86: size 5
Option #3:
Turn 34: work plains hill mine until you're at 170/180 granary
Turn 44: granary complete while working flood plain, food bank nearly half full.
Turn 55: size 2
Turn 64: size 3
Turn 74: size 4.
Turn 85: size 5.
So building the granary first is slightly slower to size 5. But that's not the biggest problem. The biggest issue is the lack of commerce while growing!
Like, take example 3... here i'm not working the first cottage until turn 33! So i lose out on 33 turns of cottage working... it would turn into a hamlet in that time. So i lose effectively 2-4 commerce per turn depending on if i'm financial or if the plains hill has a river... for 33 turns! And that's just one of the cottages... for the second cottage, i'm not working it until turn 55, compared to turn 22... so another 33 turns of cottage working is lost. etc, etc, for the other cottages, depending on if they're built yet.
So overall I'm probably losing around 200 or so commerce from this city over 80 turns, just to get this silly granary that doesn't even save any time while growing to size 5.
edit: i suppose we ought to go ahead and have a debate now about whether its worth it to chop a granary early... how many forests do we have, really?
edit2: some bad math fixed... not sure if i found it all... math is hard... >.<