It's only caravels and subs that can enter closed borders. not galleons. It has been confirmed that there is a bug where other types of ship occasionally sail into closed borders. The check at the end of the turn still teleports them out, but only to the nearest neutral tile, which may now be in an area which should be inaccessible. Very occasionally ships also manage to pass through closed borders as long as they neither start or end the turn in enemy territory. I think this is what has happened here, assuming that your borders haven't moved recently.