Worker. Do the math. Or just try it.
You don't need defenses early on. AI's simply don't declare right off the bat anymore. Barbs won't cross your cultural boundaries for a while, and animals can't.
You need two things at the start. A way to up your production (a worker does this best), and a place for a second city, which your starting unit will scout for you.
If you could grow into improved tiles, that's one thing. But you can't, so it's a waste. You need improved tiles to make growing worthwhile.
Without going into the formulas, assuming you have 3 food tiles to work, and didn't settle on top of a food bonus, it takes you 14 turns to grow to size 3. You improve your build time on a worker by a mere 5 turns. This is because each new citizen has no tile better than 3 production (hammers + food) to work, but eats 2 food).
The only significant way to speed up your civ is to get a worker out early, or chop trees, which also requires a worker.
Worker all the way. But you need to research the appropriate techs to make him useful. This means skipping early religions, but I do that regardless.
