Agree fully. The idea that a shambling horde of nomadic workers construct farms etc. for the farmers to use always seemed to me like one of the sillier concepts of civ. I'd be quite happy with cheaper and non-captureable workers as a quick fix though. Another possibility would be to add a rather big chance for workers to flee when attacked similar to the Inca slingers. I want to remember that something like that worked pretty well in Fall from Heaven II/Fall Further.