Workers are indeed vital. But so are the techs which allow them to improve tiles. If you're building granaries after one warrior you must be researching pottery pretty early. Don't do that until you have both the workers and the early 'worker techs' you need to improve the resource tiles near your capital. Depending on what those resources are, you're looking at any of: agriculture, animal husbandry, fishing, hunting, masonry, mining, and the wheel if your resources aren't connected by a river. All civs start with at least 1 of those, and most start with 2, so you just need to fill in the gaps. Pottery can follow, but for cottages as well as granaries. Calendar - which gives you sugar, silk, bananas, spices etc. - comes later but should also be a priority if these resources are nearby.
Overall, there's a reason why many the best players focus on getting the minimum number of cities needed to build some of the more important national wonders (6 on normal speed). Fewer good cities (well-placed with improved tiles) are better than more bad cities. They give you a stronger economy and a much better base for expansion, conquest and cultural / tech advantage later in the game. So don't worry too much about numbers of cities in the early game - or score, for that matter, which is a pretty crude measure too. Quality should eventually beat quantity.