If you're playing on Tiny or small, then yeah I usually push out a settler or two first, then make a tactical decision whether I have time for a granary or not. Your early warrior scout should tell you where the expansion is coming from, and if they're going to take any of your good nearby city sites. On a bigger map I would do the granary.