A 300 year old account of Christ, put forth by a fallible human being, said that he said he was the only way.
The account was written during the lifetime of eyewitnesses.
And clearly the example isn't exact. Most Baptists aren't bloodthirsty zealots, but you will find plenty among their ranks. Not every Wahabbist is a suicide bomber either. But to use Islamic extremism as a reason to stop a Sufiist community center from being built is like using the Klan as justification to stop a Quaker from opening a day spa.
Maybe in the logical realm it isn't, but in the practical world it is (NOTE: I am not supporting banning the mosque. I think banning the mosque would be silly and would not be a fair use of zoning law at all, therefore it would be a mild form of religious persecution.)
In the real world, Terrorists are Muslim for the most part. If I reversed the statement, it wouldn't be true anymore, because most Muslims aren't terrorists, but most terrorists are Muslim. And somehow, though unfairly, people get the feeling that Islam is symbolic of terrorism. In the Middle East, they will see it as a victory monument to radical Islam (Though Taillesskangaru's quote: "If we allow the mosque, the terrorists will think they won, if we don't, the terrorists will win," is valid here.)
In short, life isn't being fair to the Sufi, however, life isn't fair to anyone. Zoning laws have gotten so out of hand that the New York Government is actually able to ban the mosque. THEY are the problem, not random protesters who believe its a bad idea and would like them to voluntarily change the location, not even those who think they should be forced to move, but the people that can and may very well actually make them move, they are the problem. The common public is not the problem. Their rights to freedom of speech, like the mosque builder's rights to freedom of religion, are both equally protected. And, while not totally fair, it is offensive to a lot of people that the mosque is being built there. It doesn't offend me personally, but something like 60% of those polled think it is. If I were the Sufi, I would move it due to the offense it is causing, though he is within his rights not to if he doesn't see fit to.