There should be a mod to turn off governor-added specialists, or some greater option of the computer not assigning specialists to cities. Growth is slowed and I reassign every damn turn. This aggravates my bad habit of micromanaging everything, everywhere.