Pretty much. Your main focus Is your tech or culture but if your neighbors see you as weak, they'll come a-knockin anyway. Focus on your main goal and let your cities buid the occasional unit for defence. Then, when they decide they don't like you, they will be forced to think twice before...