^^ If he manages to get frigates with range and logistics, that makes things much tougher! But like Acken said, if you have castle+frigate+xb that should be enough to take out one frigate per turn. You are in a bay, so I am thinking you can have a couple more XBs for support. So build roads all around the city and have 2+ XBs hanging back, not next to city (so a spy cannot see them) and not on the coast (so they cannot be targeted by the frigates as they sail in). If you have a privateer in your bay, that gets you one more turn.
Plan for him to bring 3x frigate with range and logistics. If you get lucky, he brings only one privateer, and you sink that first, and he will be stuck. But you cannot count on that. I think 3x frigates with range and logistics can bring a city down to zero health in just two turns. You need last three turns, sinking one frigate each turn. But I would guess he stops training the frigates against the CS once he has range. XP to get both range and logistics takes almost twice as long, and your opponent does not want to give you that much extra time!
Keep in mind that ships exert zone of control on land units. Your XBs will have range 1 while his frigates have range 2. That is why you need roads and the element of surprise.
Stack a melee unit in the water under the privateer, since that take will take your opponent at least one shot to clear out. Pre-build a second privateer down to one turn, but don't finish it, so you can pop it out after your first privateer dies (and it will).
Can you post a screen shot of your city?