When you use a specialist, your growth will slow down or even come to a halt.
It depends a bit on how much gold you can afford, but basically i try to avoid any specialsists in early game. If however only 1 or 2 cities cause problems, i might make an exception. If you need to move up the luxury slider 10% to avoid x specialists, look at the amount of money you lose by moving the slider (after setting those specialists back to work) if the difference is less than 6 gold per specialist, use the tax slider. If the difference is more than 10 gold per specialist, use the specialists. Between 6 and 10 gold depends on your economic situation.
Later in the game, you need to look better at what you lose by using specialists. I assume your cities are now maxed or almost maxed in size.
if for example you are producing knights (70) and using a specialist makes you go from 20 to 18 shields by using a specialist, that doesnt matter, it is still 4 turns. If however a specialist brings you from 18 to 17 shields, the loss is pretty big.
Scientists or taxmen i often use whenever that is enough to solve happiness problems. Also in conquered total corrupt cities, since that is the only thing they can really provide me.