Buildings acting as a modifier for specialists would be kind of neat too, and of course tweaking would have to be done to somehow provide more food (conditionally, like with extra buildings or imps or techs). But part of what I had in mind was a way to simulate depressions and dark ages and to dispose of unwanted buildings. Right now, there is only more and more. Once you build a University and a Hospital and all these temples and cathedrals, having full effect, even if your population goes to 1. The emergence of Civilization was largely ABOUT specializing in cities, made possible by technologies that allowed more food production, which urbanization further made possible boostrapping to more specialization and food production.
What you're proposing is more specialist centric and what I have in mind is more building centric. As a game I think it would work better since deciding what to build and allow to go to ruin, ie the design of your city, is a lot more involving than just designating a specialist, though it involves designating a specialist.
I guess this building idea goes hand in hand with my idea for having an extra commerce Education, so if you don't keep up the education technologies are actually lost. Just generally, I feel maintenance of a civilization should be more than just a tax, and entropy should be a constant drag. Civilizations have risen and fallen, and more often because the economy couldn't maintain the skills that built it or because the fields salted up from too much irrigation rather than from conquest, though military demands are an additional factor to the entropic force.
That education idea could probably be modded too, but I won't get to it right away. I have a feeling I'll be finally getting to making these great masterpieces with Civ 4 when the current thing is Civ 6, like these people that are still modding Civ 3 hot and heavy. I know that these things are possible to do, but am only now starting to learn beyond XML.
I've never played colonization, though I mean to try get it and try it one of these times, and I did try FreeCol once and didn't really understand it. I'd like to try and mod colonization to be about a space colony whenever I get around to it. I did watch someone play it and he was very frustrated because he couldn't get enough liberty bells before the British came and crushed him. I told him about civ fanatics, recommended he look up some strategy suggestions.