I agree with Olleus, every civ needs Stoneworks, so it really only delays workers by up to 7 turns. Plus workers usually take longer than 7 turns to build so this doesn't give the first civ to get this tech, much of a bonus. Also with there being only a few bottom level tech's, it is researched quite soon by any civ. So the effect is minimal.
If you want a flavour limit on workers, place a limit on workers: population size of city, and institute captured slaves from barbarian attacks - so you have to actively go out and get slaves (apart from maybe certain civs with certain traits allow the worker:city size limit to be less - but not slave capture)
Or maybe a life span for workers, so after x turns (or x amount of actions performed) they *die* and disband, forcing more to be built (certain civs could have longer life spans)