Are you sure that is actually happening, or just getting the "blockade" message. The message is buggy, but I do not think that the actual mathmatics is affected.
I think that after you build the seaport, you can sale the harbor to save gold and having to look at the buggy message.