I don't think the border expansion in Civ5/6 is perfect, but it makes at least some sense and it works, so I wouldn't make huge changes to the system. The problem is - can this be made more realistic? How were borders formed in the real life in the past? When conquering an "empty" land (like Australia, America etc. - the areas that didn't belong to native people), how did this work?
The most typical way of border shaping is by war or diplomacy (+ economy). In the game you cannot buy a land from other player and you cannot directly acquire tiles by war (only all tiles around one city). Maybe this ability would be nice, but I cannot really imagine a good implementation.
For claiming truly empty lands the game mechanics is settlers. Because as a nation you can claim some lands all you want on paper - but without some human presence at least in the form of patrols it is just an empty claim. I would even say it would be a strange option to claim land tiles far away from your cities without serious restrictions... because how... and why?
When I think of means of border shaping, there is war, diplomacy, police/enforcement presence, ability to give some benefits to the people (protection, social welfare) and also culture and religion.
In game terms apart from war only economy (producing settlers), culture (in direct form and I think a policy card that accelerates growth) and religion (religious settlements) contribute to border growth. Theoreticly you also can buy cities in trade deals but apart from peace deals that isn't really possible.
I would like some later, stronger policy cards that help border growth or even tile flipping. Policies that symbolise police/patrols or state welfare.
Also encampments and maybe forts claiming unclaimed tiles would be nice.
If far off tiles can be claimed easier then I think the claim on unprotected tiles should also be able to weaken. So if I somehow gain tiles far away from my cities they are only mine until some other civ establishes a stronger presence there e.g. by settling a city.
I see "culture" more as a general "things people do", which is unaligned with a specific civilization; otherwise, you should give each civ it's unique culture tree with their special cultural developments. If you just think about the civics you have, like Political Philosophy, Feudalism, Opera And Ballet, Social Media, etc, they're advancements in culture, but have nothing to do with which nation someone feels like he belongs to.
Okay. I won't derail the thread with a philosophical discussion about what national identity is. XD