In a few past games (about four or five, IIRC) I have shared a border with a rival that was easily defended: isthmus, mountain chain, etc. I have built forts/fortresses along the border and manned them in an effort to keep the other civ out of my area. I will usually sign an ROP with the other civ - blocking them out of my territory with my units and allowing my own to pass freely. However , soon after I get Cavs, I find those forts far behind the front lines as my hapless victim gasps its last breath.
I do use them on the odd invasion, too. If I'm about to expand to a new continent, I will often send a couple galleons/transports over loaded with rifles or infantry and a few workers. They land somewhere, preferably on a mountain near the core, and construct a fortress. I then pull my workers out and put a good number of cannons or artillery there, along with a few cavs if I have them. The enemy eagerly throws himself against my impregnable bastion. What units are left over are bombarded and attacked by cavalry. As the enemy uses up all of its units on my little expeditionary force, I invade with the real army and wipe them out in short order. The now-elite infantry is used as garrisons for my newly-captured cites.
That's about it, though. Rarely build them for anything else.
EDIT: I don't want to be off topic, so to answer your question, yeah, but you have to be able to control it first. In other words, you can't destroy it while enemy units are on that tile. If no enemy units are there, you can pillage, bomb, or bombard them. Takes a little patience, though.
