Mugging your customers is doing something active that harms your customer. Not letting "X group" into the business is refusing to provide services to certain people. As ugly as that is, I can't see any moral reason to tell them they are not allowed to do this on their property.
Discrimination is harmful. It harms society and it psychologically affects the individuals that are discriminated against. It robs people of their dignity and its decline over the past century is one of the main reasons the present day is so awesome.
It is pretty clear that there is a distinction in America between businesses and private residences, for example you are not allowed to shoot people that come into the former. As such you are going to need to present a more specific argument than 'private property rights' to justify allowing businesses to discriminate... what is it about discrimination that makes it permissible for a business open to the public to engage in it?
While I do have a somewhat low opinion of humanity, most people aren't bad enough to see "No black people allowed" and just walk in, not caring. I think the market NATURALLY punishes people who are bigoted like that, and far better than big brother could do it.
It was less than a hundred years ago that this was the norm.