Most easy would be to place the frigate on top of the whaling boats (without grouping them).
Unfortunately, the frigate will not notice when the whaling boat returns to the harbor. That is something which you would have to keep track of by yourself.
Yeah, I was wondering how useful posting a combat ship on the whale square would be but most of the time the privateer catches the whaling boat in transit between the city and the whale square. And I usually play on Ozzy's TAC America map which has a huge open Atlantic which is hard to police against an AI which (apparently) isn't hampered by the Fog of War.