I think the concept of white supremacy comes in part from the idea of the supremacy of anything white. The last one is an archetype as old as humanity which is closely related to the concept of sin, or lack of it. White is the purest color, white things in nature are clean, snow, milk, moon, stars, flowers, sea foam... therefore "white" people must be clean, without stain, sin macula, sinless, ence must be superiors. I think writers as Tolkien or Martin were thinking in the archetypical idea of superiority of the white color when created the elves or the Targaryen, whitout any direct racial implication, which, in the case of George R. R. Martin at least, was a dangerous game to play.