Actually, it wasn't quite everyone- Just Antikythera, Argos, Portugal, Gibraltar, Espanola, Echajinia, Genoa, Sohi, and Maroc. Oh, and that revolt in Corsica and Sardinia. A couple nations were neutral, and the Fatamids actually helped you.
The second time was just because thomas, Bris, and Nerdking decided to be backstabbing bastards, and caused that whole Mediterranean War- Gibraltar, Maroc, Espanola, and Fatamids vs. Echajinia, Argos, CECA, Arabia, and Adriatica, IIRC, and the Seljuks fighting in Anatolia adding to the mess.