Do NOT let your military fall too far behind. If you are consistently at #6 in the soldiers demographics and you're significantly below the average, then people WILL target you and try and invade your lands. Always build at least Walls in all of your important cities in case one of your neighbours decides to be sneaky, and if your capital city is on the cost, then get defensive buildings as soon as possible to negate the possibility of a devestating frigate rush on coastal maps.
Also, frequently check to see who's if the #1 soldiers stat is rising fast. If it is, then someone is preparing to mount an attack. ALWAYS check the demographics.