The simple fact is, a granary is an investment that will pay itself back over time.
The earlier you build it, the more time it has to pay itself back and the better it gets.
There are 2 ways you can use your resources, you can spend it or invest it. Invest being something that will make you grow and will more than pay itself back over time.
Settlers, workers and granaries are investments. Units are only investments if you conquer cities with them. Culture hardly is an investment. It can be a necessity to survive, and it can win you the game (if you choose to go for that)
Unless you are planning to make those other things investments by going for a very early attack (and not just without a reason, but a usefull early attack, that will pay you some rewards) or going for a cultural victory, in early game you should try to maximize your investments and minimize your spendings. Therefore, build the bare minimum of units and culture needed to survive.
Try building less and less of these and see with how little you can get away. This mostly depends on barbarian settings and AI agression.
buildings for happiness could be counted as an investment, but it is an investment that costs shields and pays back gold (in needing lower tax slider) Espescially in early game, gold is by far not worth as much as shields and therefore, it is better not to invest in this early.
leaves the choise between workers, settlers and granaries. Must find a good balance between those.