It depends. In crowded games I won't build many workers but instead build an army and take the workers from my first victim. I've had game where I built 2 workers for 3 cities and then gone to war.
Another time I spent a lot of my early research on early techs and didn't have many new buildings to keep my city occupied. Wasn't that interested in the wonders either, so I built more workers than usually. I also had lots of forests that I prechopped waiting for something to use them on. When I started building my army I got all the workers to finish the chops and build roads toward my target.
The idea is, if I can build enough improvements to have good tiles for my city to work I don't need any more workers. Sometimes I'm a little behind with the tiles, so I still need to work on my strategy. Of course there are times when I switch from mine to cottage to farm depending on the current needs. In those cases I actually need more workers since I need more tile improvement.