What I like to do is whip a Granary in my Cottage Cities, since generally you will see payoff in population by doing so.
Other buildings, I like to slow-build using Organized Religion and working 1 Hills square alongside my Cottages, giving me the one bonus Hammer per turn. That way, I am not missing out on the Organized Religion bonus, but I can also focus on growing my population and improving my Cottages.
If your economy isn't crashing, it can help A LOT to have a few Farms in your Cottage Cities and to work a balance of Farms and Cottages while growing your City... more Farms initially if your economy can handle doing so, and less Farms as you approach your Happiness (or Healthiness) caps. I would think that this detail is far more important than whether or not you work a Mine or two--generally, a poor Cottage City is one that has less than 5 Food surplus per turn, while a better one will have 5 or more Food surplus per turn, which often means having to work some Farms alongside the Cottages and the occasional Mine. As I said above, as you approach your Happiness (or Healthiness) caps, you can lay off the Farms a bit, but until that time, you'll generally see more gains overall if you get a reasonable amount of Food surplus for City growth.
Ideally, you will also aim to be bringing in 4+ Hammers per turn or 8+ Hammers per turn if you have your State Religion in that City and are running Organized Religion... perhaps a Grassland Copper or Plains Horse square exists in that City, which could mean that you wouldn't need to work any Mines at all in order to get your base 4 Hammers or base 8 Hammers.