You're just punishing late settles and conquests by removing their ability to build certain buildings (so they'll always be behind). I don't see any specialization there.
Hm. Maybe.
My thinking is that, if every city now has to skip certain buildings in order to prioritize others (assuming higher production costs), then every city will be weaker, not just late ones. And so there would have to be decent buffs to a lot of stuff. I would like to think that the strength of later buildings would make the lack of early ones less significant. Will you really miss having a monument in your modern city? Is it worth spending hammers for those yields when you could jump straight to later stuff?
Also, capture chance would probably need to be massively increased or set to 100% in many cases since you couldn't rebuild old stuff.
...
My memory of vanilla is fuzzy, but I feel like there were more often cities that struggled with certain things—either difficulty growing, poor profuction, slow border growth, etc. With the VP happiness system, we're incentivizing building cities which are strong in every aspect, but that's also kind of why they're so samey. If you don't build buildings for every type of yield, you end up with huge unhappiness problems.
It makes sense to a degree. Real cities don't just ignore entire aspects of society. Even an industrial city is going to have some culture, some food, commerce.
Perhaps if the happiness system weren't as strict—if instead of median global values, the targets were lower—it would be more feasible to focus on one or two yields and let the others fall behind a little. You would still be punished for ingoring something civ-wide (e.g. you still need science to avoid falling behind), but not every city would need a library, university, etc.