I think they will move a boat with a settler through your territory to settle neutral territory beyond.They can never settle cities inside your culture. Nor can you in theirs. But technically they can move through it and settle on the other side. They don't though. So even if you have open borders with an AI and there is free land 'behind' you, they won't move a settler party through your territory and settle on the other side. However, if there is a path of neutral tiles, even just one tile, they can and probably will move through it and settle, if you wait too long.