Actually, no. It wasn't that. It was "We don't like being taxed by England without representation. Absolutely none of the colonists could vote (not even the land-owning white men), so they had no influence in parliament whatsoever.
That's also a difference between the American Revolution and when the south betrayed the country a century later. The south had representation.
Almost all of the British couldn't vote either.
The conservative argument of the time was that your interests (just like those of Manchester for example) were represented by those who were investing in or trading with the Colonies.