As you said, no one would go around saying "I am white" or "I am black". It is inferred that he had no difference of race to the other competitors in the slums of London, when he mentioned his and his family's history as Bean asked him how he liked his new position.
Anyway, let's just agree (I don't mean you) that if my summation that he was playing a white person is correct, then it is weird. I never claimed it would be weird if he was playing a black person in the Victorian era. British tv has quotas for minority actors, so such things (the weird part) can happen.
What you seem to be saying is that if a character isn't clearly identified as black you assume they are white.