I agree - get them while they are in their ships.
One option, if you know where the ships are coming from, is to send your ships towards them and harass them.
If you have the Great Lighthouse, you might even be able to form some type of blockade so he will have to attack your ships before coming through.
For defense, make sure you have walls or size 7+ cities. Catapults aren't very popular, but would be effective in this case. If you can drop a berserk down by one HP it makes it that much more likely that your unit will survice the onslaught.
Another option is to hit them after the initial assault. If you have catapults, they can be used in the initial assault, and to hit the ships before you attack with your own ships. Just make sure you have enough units in the city to defend the initial assault.
IMO berserks are the absolute worst enemy you could have in this game, because they can't be controlled with the conventional land defense tactics that I regularly use. Of course once you get riflemen they lose their advantage but they definitel have their time to shine in this game.