I only play at Regent level and I rarely build CH's, doing so only toward the latter stages of the game if/when I have cities far away from my capital.
Sometimes I might build one earlier if it's for a city that's on an island and a considerable distance away from my capital, ie, a high corruption city. In a case like that, I will try to rush the CH, if my gov't type allows it.
A couple of other improvements I rarely build are the Coal Plant (too much pollution) and the Coastal Fortress. The CF doesn't seem to have too much of an effect, although I will build it if the city is next to a narrow strait, where enemy ships have to pass close.
Actually, I never build the Coal Plant.
I sometimes forego Walls as well, especially if there are no threatening Civ's nearby early in the game, but this frequently comes back to haunt me.