There are times you have to control your workers manually.
In the game opening when every decision is made around worker/settler, worker techs and worker actions. Chop, improvements, basically minimize the turns that your worker is not working. Thats why i almost always get BW and wheel first so the working can chop and build roads. These early worker actions can give you a very strong openning (i.e early cities, troops, commerce/techs and etc), the effects of a good opening will grow exponantially and translate into very good mid-end game.
Also after you get some techs that allows new worker actions, you definitly want to micro-manage the workers, i.e. after you res railroad, make many 2 worker groups and build railroads on all mines, lumber mills so you get instant production boost. Same with Replacable Parts and etc.
Also during war time, you need groups of 2 workers to build roads leading into enemy terriotry so you save much time moving your troops.
The only time I use auto worker is in the late games when most of the important improvements are already built. I actually devotes lots of time (maybe 10%) micro-manage workers through out my game, it is just such an important aspect of the game.