I remember a thing about Poles and generally being angry that the East Europeans (that the UK itself lobbied to include in the EU in 2003, ironically) were migrating to the UK. East Europeans being probably the single most white people on Earth, I admit I don't really see why people angry at dark-skinned people would vote for Brexit on this subject.
Especially as, since Brexit has been voted, net immigration in the UK has barely went down, BUT it's now much less Europeans and much more non-Europeans (and as such, dark-skinned people), so it'd have got the exact opposite effect. Which is funny.