It's a semi-vowel like W. English R can also be argued to be a semi-vowel for some pronounciations, and we also pretty much use L as a syllable by itself too (like in "table" - I'd argue there's no schwa in there).
The vowel-consonant dichotomy is good enough for most layperson purposes but it breaks down when you start looking at phonetics in any detail.
LOLWUT, do you live in a cwm or something?