I thought at first that I would eliminate the "library, market, bank." but then I remembered how many important things they are prerequisites to. So I'd say eliminating the "courthouse, forge, barracks" is the lesser evil.
If you whip a forge, you get the extra production. Assuming your city has even 4 hammers, you get a second bonus hammer on top of the "benefit" hammer + increased happiness. If you whip a library, you get the extra research and culture that much sooner. If you whip a settler you get all the benefits of an extra city x turns sooner + your own city will grow faster (since a settler halts growth). Etc.
Maybe 1 hammer per turn is a benefit, but it is not the only one. Don't forget you get also the benefit of whatever you whipped!Whipping with a granary isn't as amazing as people think, and my focus is on those those heavy micromanagement, heavy calculations benefits for a minor benefit (often it works about to be <2 hammers per turn).
If you whip a forge, you get the extra production. Assuming your city has even 4 hammers, you get a second bonus hammer on top of the "benefit" hammer + increased happiness. If you whip a library, you get the extra research and culture that much sooner. If you whip a settler you get all the benefits of an extra city x turns sooner + your own city will grow faster (since a settler halts growth). Etc.