I find that the game is much easier with a decent military, so it's worth putting off those buildings. I usually forget to do this, and then I get wiped out early. But if I play right, I try to queue my production to include a new unit for every 1-2 buildings. Then I have a nice army just in case. Last game I was sandwiched between Alex and Zulu. I had two early war declarations and had just enough bowmen to survive. Now I have a healthy garrison and can go on the offensive against one of them.