I guess they're just as situational as all the others mentioned, so as usual unless you play one type of game on one type of map every time there probably isn't a definitive answer.
I think this is the best answer. It would be poor development if you
had to build everything. Civ is not meant to be sim city.
Having situational buildings is great is gives you more choice to pursue certain strategies etc.
That said I think the worst buildings are
Caravansary - It's a lot of hammers at a time in the game when you really need to be building more important things. It's not bad for what it does, but there are always better things I could think of to build. Later in the game when you can afford the time to build this, you probably won't notice the extra gold that much or even be using caravans for external gold.
You'd only ever build it early if you had a reason to send a lot of land routes (i.e. a landlocked capital on a river). That said it would be good for Arabia to get even longer distance trade routes.
Police Station - Very expensive but I guess if you were leading in certain technologies (i.e. space victory, culture victory or military victory) you might want one of these in your top 1 or 2 science cities.
Medical Lab - Great population bonuses but really too late to make that much of a difference to the outcome of a game - as usually a games outcome is already decided. It's something you tend to build if you're ahead of the crowd and don't need to build anything else.
Harbor - Necessary for city connections and provides a small gold boost to trade routes from your main trading port and longer trade routes. However most of the time you'll build this to get access to the seaport as you have limited numbers of cargo ships. Generally speaking the Harbor is pretty lacklustre after the BNW changes - it has a much more focused role. Usually you only send external cargo ships from 1 city (mostly the capital) so you want a harbor there and on any cities on different continents.
Forge - It has a bonus to unit production and a small bonus to iron but iron very rarely clumps so usually it's only 1 extra production per forge. It takes a few turns to build this and it doesn't really feel like it makes an obvious difference. I wouldn't really notice it if it wasn't there. For instance if I was playing Rome to do a Legion rush the 120 hammer cost probably wouldn't be covered until I could build more than 10 legions. Again it feels like something that doesn't really add much value.
Bomb Shelter - Yeah its pretty useless but I guess if information era games lasted longer than it might be useful in certain circumstances. That said if an AI is able to nuke your cities by turn 300 then you've probably done something wrong to start with.
Stadium - a lot of hammers for a small amount of happiness. Generally you shouldn't need this
Arsenal/Military Base - While these are maintenance free usually they don't add any real value unless you find yourself neighbors with a much more powerful civ that wants to kill you. For 500 hammers I could build a rocket artillery and have 75 hammers left over. The rocket artillery would have much more defensive value.