In general farming, unless there's not many farms. I sometimes used the trading one just for the happiness bonus on towns that won't send the food to cities or I'm having some kind of happiness crisis. Fort town feels like a waste. TBH, I don't think the game should lock you into one specialization. Maybe there was some kind of exploit that I'm not seeing, or maybe they wanted thing to be more "set and forget" (instead of a micro "There are enemies here, make it a fort town now!", then switch back, etc), but it makes some of the situational specializations something you never pick.