Then make them count
For example, limit them to 2-3 chains, but each more powerful. After all, workers usually don't level past lvl 4. For example:
1) Mobility=>Mobility II=>Pathfinder (new promo proposed by someone in different place, double move in forest and hills but no extra defence)
2) Hardy I=>Hardy II=>Hardy III (as is, but with more withdrawal, something like +1def and +20% withdraw each level. I expect my workers to run away from enemy, not kill them. Might add +% vs Animals and Beasts at II&III)
3) Skilled I=>Skilled II=>Skilled III (buffed.Right now after four promos you get +60% work rate, not worth it really. With +33% level you 4 lvl worker is twice as good as fresh one. Fight even try 50% but it might be too powerful early on)
With this, you can specialize you workers to area they are working. Lots of animals and barbarians and not enough warriors to protect them all ? Choose Hardy line. Frontier with hills and forest, but you already have enough military to keep them safe ? Mobility line. Improving flatland/roaded tiles in heart of your empire ? Skilled would be best