Rather than going fully into specialization directly, I'd "bump" Pepper's idea first where he wanted all buildings obsolete around two eras after they were discovered.
In my experience, the building lists really adds up when you reach the Industrial Era. So a good first start would it to review all buildings up to (and including) the Renaissance Era and try to figure out, how and when they can be obsolete and replaced.
For example, whe have tons of different Mining /Farming buildings that require two resources in your vicinity. They add a nice flavour early on, but later it can be tedious to go through all cities and build every new combination building just to get some cheap extra hammers / food. A building at around Agricultural Engineering could replace ALL of them with a building that gives plain +5
. For some cities this will reduce food production, while for others it will boost it a bit. And all in all, 5
(or even 20) isn't really a GREAT deal in the industrial era.
Same goes for mining buildings. The main reason we had them was to simulate that it is very difficult to transport ore. You need a mine and then smelt it directly in a local smelter. But this changes with railroads. So at railroads, there could be a single mining building that gives the boni from mining camps, along with the ability to build every smelter that exists (if you have the required ore, of course).
To reduce the smiths / smelters and Factories that every city can build (but obviously not want), there could be a prereq building for all of them, that you don't want to build normally. The Factory, for example, is a prereq to all production factories (IIRC). But it is inself so good that you can't skip it. So there should be something like an industrial park (we have one, I know), that is very cheap, but not having a bonus in any form. It is purely there to enable all the factories. Or, if someone prefers to split their factories to different cities: The Industrial Park could unlock a handful different Industrial Parks that specialize in certain things: Food, Military, Electronics, every-day-stuff etc.