That restriction doesn't seem to kick in nearly as often as you would think. More often than not, you are struggling to find the time/hammers or the right spots/tiles to build them.
That still leaves the opportunity cost of not getting something else first, but it feels like you get a lot of good stuff and could make up for it in saved production for units, fewer units needed to defend, great generals, active research, or housing.
read post #53, explains my reasoning to why the restriction is important, around that 10 pop range where you've got a industrial/commercial/campus/holy(got a religion) and are really struggling with housing to get more pop to build anything else as your still in the mid game and no where near neighborhood districts. I've seen this numerous times already. You probably want an entertainment district at some point and looking at 16 pop which is a late game city to get a potential encampment and that would mean no theater district so culture victory out the window. Opportunity cost is very high for encampment.