A large part of it, I think, had to do with the systemic racism toward African Americans and immigrants in general. It kept the poorest sections of society from uniting, or even considering uniting (outside of an idealistic few). Also, the spectre of African Americans, Irish, Poles, Chinese, et al being used as scabs or cheap labour kept the labour movement from becoming anything substantial. In Sweden, Britain, France, Germany, Russia, etc. these were not significantly large concerns.
So, eh, racism did it. Interesting point of view. So, ummm, how did US labour unions grow so large?