I agree, but I think there's a couple of things that make anti-Semitism different.
The first is that the figure of "the Jew" is something more than just the Other. Most Others are imagined as having bad culture: primitive, simplistic, decadent. But it's at least a culture, and the barbarian form of the Other is at least part of a Nation, however contemptible his nation is. "The Jew", on the other hand, is imagined to be defined by his lack of culture: the "rootless cosmopolitanism" which allows him to move between nations, taking up and slewing off languages and nationalities like a cloak. "The barbarian" is only as much of a menace as his numbers allow him to be, and is no fundamental threat to the nation so long as he is kept firmly Over There. But "the Jew", by his nature, is a permanent and ever-present threat to the Nation. "The Jew" is almost unique as a racial caricature in their consistent portrayal as powerful and sophisticated. And in that sense, I think the figure of "the Jew" supplies something crucial to the European political imagination that sets him apart from "the barbarian", which requires "the Jew" to remain a figure distinct from the Irishman, the Mexican or the Muslim.
The second point is that the Jews themselves have, by historical accident, provided an incomparable vehicle on which to project "the Jew". There's no other ethnic group which is so widely spread as to be recognised across Europe, so thinly spread as to be vulnerable, yet also so well-established as to be a credible as a shadowy menace eating at the heart of society. The only comparable diaspora are the Roma, and they're too obviously impoverished and marginal to bear that kind of scape-goating. If the figure of "the Jew" has to be identified with a particular ethnic group, it's almost natural that it should fall on Jews.
The question there, I suppose, is whether it does have to fall on a particular ethnic group. Most contemporary hostility to "the Jew" doesn't invoke his ethnicity explicitly: it's enough that he's cosmopolitan and educated and un-nation-like. But is that really enough for the latent anti-Semite? It's hard to convince people out of convinced nationalist circle that, for example, Jeremy Corbyn is "not really English" just because he doesn't share their particular chauvinism. He speaks English, he looks English, he's of English descent: even casual racists are just going to see him as "an Englishman I don't like". And if all of England's woes can be traced to Bad Englishman, what does that say about England? It's much more convenient for the source of the contagion to be identified outside of the national body. And, historically, the obvious scapegoat has been Jews. Perhaps in future it won't be: perhaps there'll be a large enough population of mixed descendant and language and identity in enough positions of power to provide the newscape. For the "rootless cosmopolitanism" to become an actual describe and not just a crude euphemism for "Jew". That remains to be seen.
So I suppose I think that anti-Semitism is and is not inevitable. Conceptual anti-Semitsm, hostility towards "the Jew" as an abstract figure, is inevitable so long as European political thought does not undergo some drastic shift away from its ethnic and quasi-ethnic preoccupations. Actual anti-Semitism, hostility towards Jews and those of Jewish descendant, is not inevitable, because it's not necessary to identify "the Jew" with actual Jews- but I still think it's hugely probable that the identification will be made, and even where it isn't made explicitly, it will still float under the surface. Taking another Labour Party figure, it was even easier for the right to portray Ed Miliband as "the Jew": cosmopolitan, urban, educated, the son of immigrants, and not at all coincidentally, an actual Jew.