These are still preliminary thoughts, but:
It took me a while to realize that you can always switch back - so if growth is slow for your town at the moment, might as well support your cities for a while until better food bonuses come online. (You can still buy districts if needed.)
Complicated math, for sure, probably also depends on where you want to grow your empire? (Since towns may be future cities or even capitals, too)
How to specialize is a completely different question, though, but mostly depending on terrain, probably? (If there's not an obvious situational option like Frontier town)
What can be self-defeating: locking them in early in Modern, the factory mode - forgot how it's called - can save you a lot of money later (if going for the economic victory)