Towns can always be invaded. These are the ones with the large flag overhead.
Those pesky tiny things cannot. I like to call them villages, much smaller flags overhead.
If it has a governor, you can smackey it. For those things that only have a Mayor, they made some kind of programming decision to make it not-possible.