Is it really a waste to have too many workers?
I was actually really surprised that people have so few workers. I guess I'm still in civ3 mode, in this respect.
The more workers, the faster the tiles are upgraded...which means the faster the cities can get productive and growing. It also means that towns will get placed sooner, rather than later. When I expand or take over cities, the horde of workers get the city networked and improved really quickly.
It's true that eventually, I'll have a stack of workers doing nothing. This seems like a waste at that point, but they were hugely busy until that point.
In my next game, I'll try having fewer workers and see how it works out.