At least in Brazil, many of the Indians displaced by other Indians still exist. But rather ironically, on some cases they had to be told by "white" researchers that they were displaced by other Indians, because their lack of writing means all they were left with were some vague legends. But the archeological and linguistic evidence is clear: the Tupis who inhabited the eastern coast when the Portuguese arrived came from the Amazon, while several tribes who then inhabited the backlands were originally from the coast, and were kicked out by the more advanced Tupi peoples. Some of the last remnants of pre-Tupi Indians were indeed still being kicked out when the Portuguese arrived. So from the POV of Indians the portuguese were just yet another warring tribe, tough bizarre looking and magic-wielding.There is obviously no difference between whites, Chinese or Philippinos in this regard. I probably should have made that clearer. I just said whites because the argument was originally about them (and Valka, who is also white).
Either they're first and original inhabitants, or, as I said, there's no-one left to resent them for displacing them, so they have the best existing claim.
However, that's exactly my point - there is not much point to it. It can be used as a neutral, descriptive term that applies to almost everyone.
If someone uses this in a way that implies one is responsible for the actions of their distant forefathers, it becomes a slur.