It's hilarious how some things are taboo "just because."
This is how societies end up with a breast fixation in the first place.
By saying men can go open-chested but women can't.
Thereby adding an element of forbidden mystique that otherwise wouldn't be there.
Nothing wrong with a sense of modesty and decency.
Besides, most men shouldn't be going around topless, anyway.