In the early game, it's much better to prioritise units. Most buildings have relatively high shield-costs (relative to what a small Despotic town will be bringing in), and also cost maintenance (gold) on every subsequent turn; whereas units are cheaper and don't cost maintenance until you exceed your 'free-unit allowance'. Under Despotism, you get 4 free units per town, so the more towns you have, the larger a military you can support cost-free (although your free-unit allowance can/will change abruptly when you switch governments, so be careful). Also, the AI rates your military according to total strength (valuing attack 1.5* more than defence), so if you're concentrating on buildings rather than units, it will be more likely to attack you.
My newly planted Ancient-Age towns usually get a Warrior to guard them, a Worker to improve them (in a 'core' town making at least 2 food and 2 shields per turn, you can usually build one of each during the first growth cycle), and
then maybe start on a building -- but more likely more Warriors, or Archers, or Chariots -- or another Settler (or Worker) if the town is about to grow unhappy from 'overpopulation'. While there is still land to grab, a Settler is likely a
much better investment of shields than a Granary, which itself would be a
much better choice than a Temple.
Speaking of Granaries... well, if you've managed to find a town site(s) with 2-3 high food-bonus tiles (and freshwater), it might be worth putting up a Gran in there, and using that town(s) exclusively as a "Settler-/Worker-pump", peeling off the 'excess' population and then regrowing fast. But putting a Gran in every town is a poor use of limited shields, especially in the early game -- after all, you can build 2 Settlers (i.e. plant 2 more towns) for the shield-cost of 1 Granary.
Useful buildings? Barracks, once you have the shield-output to get them up quickly: build them especially in shield-rich towns, where you'll later be building your more expensive units, so that they start out as Veterans. Courthouses are also important, in towns that are further away from your Capital than your 'first ring'. And of course, Aqueducts will (eventually) be needed in any towns that have no freshwater-access.
As for Temples... well, there's a great quote from CFC-member
@Bede that always gets pulled out for questions like this:
Temples...temples...priests are prevaricating parasites who pillage the body politic.
You want culture, build libraries. You get something back from the investment.
You want content citizens, build marketplaces, trade for luxuries, build towns for luxuries, build colonies for luxuries.
If happiness is a problem in a settler or worker farm, it is a self-limiting problem. Raise the luxury tax, hire an MP, you only need to make the expenditure for a couple of turns. Temples are with you forever and are a permanent drag on the economy.
Unless you're going for a Cultural victory, you
really don't need Temples during the early game (or even the mid-game, for the most part), and especially not at Warlord level. Cultural expansion from the Capital is taken care of by the Palace, and if you plant your towns with no more than 2-3 tiles between them, so-called Cx(x)xC placement, then those towns' borders will join up without needing to put any Culture-buildings in them, anyway. Planting a Temple on a coastal town, just to get access to a Fish or Whale? Almost always not worth it. But a Lib, if the extra coastal commerce will be converted (using the SCI%-slider) to boost the beaker-output? Yes, that's worth it.
At Warlord, you shouldn't need Temples for Happiness, either: the first 3 citizens per town are born content, and (under Despotism) a couple of military units stationed in a town will keep 2 more citizens content, so your towns
can't riot until they hit Pop6 -- by which time hopefully you have also found some Luxes, and/or are close to switching to a less corrupt government form (e.g. learning Republic and then switching to it will bring in enough income to make a few people people happy using LUX%-slider spending instead).