Actually, you do have access to it. I remember once, I had no access to horses in or near my main territory, but there was one on the other side of the continent that no one else had taken. I then took a galley with a settler over to it, built the town right on the horses (that were on the shoreline), and my horses were then connected to my network by sea.