History and society, obviously. The same things deciding that the laws of your country stops at the border and if you walk past it and kill someone, you'll be tried in this country and not yours.
Indeed, but that's a question of power; what it really boils down to is 'which country will enforce its law in that area?'. That's not a question of right: the fact that I will be tried in Britain does not imply that I should be tried in Britain.
You're too smart to seriously pretend that what prevents people from grabbing lands for themselves or another country is only the "other country accepting it".
You perfectly now that it's rather because the home country has a claim on its land that is far stronger than the one of one individual.
While that is the case due to the balance of power - the home country can enforce its claim to the land better than the individual - I maintain that inasfar as we are discussing 'right' (which of course is only in question between equals in power), 'the country' has no automatic ownership over the land. I said earlier that government is a social contract - you pay taxes in exchange for a share in what the government offers you - and so, if the occupant of that land will inevitably benefit from the government's services, there is a case to be made that the government can insist that he pays taxes, and therefore remains under their jurisdiction - there's no such thing as a free lunch, after all. However, in an ideal society, I think that that would be entirely true.