It becomes apparent pretty quickly. As it stands you might have enough workers that you're not working unimproved tiles, but your empire would almost certainly benefit from having more even if individual cities don't. Laying out roads to city sites in advance will help you settle faster. Laying out roads to enemy civs will help your armies get there faster (especially important if you'll be sending reinforcements after the first stack). Pre-chopping forests will help you get your important infrastructure/units out faster. Chopping forests/jungles and laying down forts around your border cities will make your cities safer from enemy SoDs.
The most important time to have lots of workers in my mind is when you start settling jungle in a serious way. Adding 6 turns onto any improvement (plus the 7th to move onto it in the first place) slows down improving land a lot, so having 2.5-3 workers on your jungle cities is huge. Of course, you want to have them built before you settle the jungle so those cities can take off right away, rather than building them as you settle jungle.
I have tried replaying the first 100 turns of the current Emperor Cookbook game - http://forums.civfanatics.com/showthread.php?t=314555. In these games I established my third city just before the 100th turn. Replays with 1 worker appear (to me) to be better than replays with 2 workers. I have not yet tried 3 or 4 workers, but I suspect I would need a dramatic change in my approach to see any benefit from either of these. These results are not directly comparable with the posted saves in the cookbook game since I made no attempt to explore.
There are a number of factors that may affect this result.
1) I have not chopped very much - this may be the main reason I'm not seeing a dramatic benefit from more workers.
2) There is no jungle to deal with, but in any case, I have not yet discovered iron working.
3) My research path misses AH and thus there was no possibility of building pastures.
4) I have not been involved in any wars.
5) I would have benefited from connecting my third city earlier, but only by a couple of turns.
6) I would have benefited from building hamlets on the floodplains near my capital, but this probably only cost a few commerce.
It is clear that after building the third city, there would be a significant benefit from additional workers, but by this I mean a total of 2 or 3 workers rather than 4.
I suspect my play style fails to exploit extra workers effectively, perhaps not just chopping but other things as well.
RJM