As for enemy ships attacking the weakest target in a stack, I don't believe I've seen that. It shouldn't happen, as far as I know. Can you provide additional information about the circumstances surrounding this? (Patch version, ships involved, civs involved, promotions, wonders, in whose borders, and any other details that might be relevant.)
I guess the Noc Noctis explains that, but it does not explain why the automated option to have your boats defend your waters does not work.
As for details on the losing of ships, I have had it happen even playing as the Lanun, and taking 5-8 pirates with The Black Wind at 100xp and fully powered (13 str) and sailing as a "stack", as it were. Within a few turns The Black Wind had not been attacked while in enemy waters all but my pirates had been picked off one by one (and yes, they were all, including the BLack, on undeclared nationality). It was on Patch Y, and the last 2 before that. But I rarely played as a seafaring civ before then so I can't say if it was always a problem or not. All I know is I can automate my boats as such but they do nothing when privateers enter my waters.