There's no in principle reason to deny polyamorous relationships the legal recognition of marriage.
I wonder at two things. The first is the practicalities of the legal arrangements involved, the second is whether it enables religious/misogynist/cultist abuse in situations were consent is very compromised.
I don't think either of those are conclusive arguments against, however.
In western societies it's a fairly niche concern. The poly community's concerns at marginalisation are entirely valid and there's an argument that same sex marriage advocates are throwing poly interests under a bus in order to claim greater social acceptability for the cause.
And mere legalisation wouldn't promote more multi-person marriages. So the "break down of whatever" arguments are silly. I can see an argument that widespread polygamy with one male and many females is socially destabilising, but it's just not a realistic concern in a relatively liberated western context.
Neither do I accept the argument about exploitability. We don't question why people get married currently - I can get married for literally any reason I choose (money, tax benefits, property transfer, love, citizenship, mocking your anachronistic views of the institution by getting divorced a month later) or none at all, no questions asked. That shouldn't change... the state querying people's motivations to marry is a dangerous path indeed.