<- New to the forums.
Castles are the one non-wonder building that I have never ever ever ever built, even since my starting days with Civ4 vanilla. I saw it on the tech tree in my first game, diagnosed it as useless, and that was that.
Hospitals and Recycling Centers are buildings that I used to build early in my civ playing, but don't build anymore. I generally beeline to space-race techs before getting ecology or medicine, and generally the end benefit from Genetics or some of the other buildings provided by those techs are enough.
Other than that, I've found a use for pretty much everything, given the situation. I've built Walls in particularly nasty early wars in newly captured cities since they're cheap and make holding the city alot easier. I even managed to use Malls (America UB) to good use despite my assessment that America got the short end of the stick in terms of UB - getting 1 smiley off the top plus 2 smiley per media-based luxury meant keeping my key, large cities in permanent happiness.
Honestly, some of the other responses here baffle me. Like the Monument. How the heck do you have an early game at all without Monument (unless you're Creative)? Even back in Civ4vanilla (before Charismatic started wtfpwning early games), it seemed like you absolutely needed to use Monuments (then Obelisks) to secure borders, secure resources, keep more of the map revealed (stupid barbarian cities), and get an early edge in square-by-square cultural wars with bordering enemy cities.