i keep one per city, and about 3 cities early. if i build up to 5 cities then my first 3 workers suffice unless i have a lot of improvements i need to build.
if i don't get around to workers until later, then i try to get 4-5 or so, but i always intend for any worker number over 3 to be sacrificed as a combat pawn. the AI likes attacking workers so i send in a few when im at war. if im defending, many AI will chase a worker all over the map instead of attacking me and if im attacking i can pull defenders with workers.
when i capture enemy workers, i IMMEDIATELY delete them asap so that the unit is absolutely gone. i do the same in mp.
late game, with a one-track road system you can convert 5 cities worth roads into railroads pretty quickly with 3 workers or so. if i stagnate at my third city which usually happens, then i get my 3 workers and every time i capture an AI city, i puppet it then i send in 3 workers to re-develop their plots and REMOVE ROADS I DO NOT NEED OR WANT which saves me money. then, if i want to annex it, i already have a correctly improved city even if no buildings.
those same 3 workers will go to every city, one at a time with my army to build roads ahead of or behind my troops so that when i capture a city i can have it connected pretty quickly. if i find my empire has nothing real for 3 workers to do then i go to war - if i find my empire has too much for 3 workers to do, i slow down everything and focus on defensive building.
if you monitor the pace of how much work is left in your territory for your workers to do on a basis of 1 worker per city, then you can estimate if you are advancing at a viable pace.
Hammer - the button does not say "build smart improvements in places where they operate on a min/max basis and appropriately consider how many farms/trade posts/specialists a city will WANT to work given its location, size and room for growth. a city might be intended to be size 12 with 4 farms, 6 posts, 2 scientists for example. but the AI doesn't know how to think like that, it doesn't look at a city and have the worker specialize it. now, if you could tell a worker to specialize one type of improvement -
or tell a worker "automate luxeries/strategic resources only" or "build one-lane roads to connect cities only" that would fix it well.