The scout example is out the window with No Trespassing. Its unclaimed territory, you have no right to say no trespassing.
Missionaries is still valid. Would you be ok with diplomatic penalties for killing civilians or do you want no repercussions to killing civilian units in your borders ?
Prior to @18th century when Surveying was perfected enough to create accurate maps, borders were generally quite "porous", being described as being "From geographic benchmark A to geographic benchmark B". It's why rivers were so popular as border demarcations: they were a hard line in the landscape that was hard to misinterpret. Absent rivers, borders tended to be rather vague -- especially to people on the ground with no clear benchmarks in sight. It's
why there were so many frontier border clashes in the first place. But nations going to war over a border clash?
Maybe 1 in 10, but very possibly only 1 in 50.
As for killing lone civilians in Civ V, it rarely happens. Usually the civilians are just
captured by marching a Military unit onto them. If a Border Clash mechanic were instituted, just like you have the option to return civilian units to their civ of origin, just send them back to where they came from. Just have them appear at their civ's nearest city three or five turns later. (Like donating units to a CS.) The
killing of another civ's units would apply almost entirely to military action against other Military units.
Border clashes are almost entirely about securing _your_ borders. Attacking someone else's city is obviously outside of your borders and inside theirs. That's an invasion/act of war in and of itself. Even if it's a city was established by sneaking a Settler through your frontier and setting up shop in your backyard. There, the usual diplomatic promise is to Demand that the other civ give you ownership of the trespassing city. (Don't hold your breath.) Once he says "No" then attacking the city is a
defacto DoW. That is, the other civ WILL be attacking your units and cities, whether you say anything or not.
Basically, Border Clashes are about unit versus unit actions, while Wars involve attacking cities. [Has anyone _ever_ concluded a war without a city being attacked at some point? Honest question: I have no info on that particular demographic.]
Naturally, any kind of border clash would have some kind of negative diplomatic relationship hit for that civ and any of it's allies. (Including CS Allies.) That's the tradeoff: you maintain the integrity of your borders (out to as far as what _you_ think is reasonable [but most likely limited to some tile distance defined by the rules]), but at the cost of diplomatic tension with that civ. Build up enough tension and sooner or later somebody WILL declare war. Both sides arguing that their cause is Just and Righteous: "
They declared war on
us!" versus "
They provoked us into declaring war!" More than likely Denouncements were exchanged some time prior to then, just adding more fuel on the fire.
But not always and not immediately. Which I think makes for a more accurate model than "It's declare war or nothing!"
Note: In regards to "natural" or "reasonable" frontiers, You start with the city center, surrounded by six hexes. Eventually, the city will grow to a radius of three workable hexes. However, secondarily, with enough years of Culture expansions, a _complete_ city encompasses a SIX hex radius. (You can see this best if you build an entirely land-locked Great Wall city with lots of elbow room.) To me, that 3-hex radius is a "reasonable" frontier. For the 6-hex radius, you better be playing on a Huge map to argue that it represents your "natural" frontier territory. Sort of equivalent to America's Manifest Destiny and "Fifty-four forty or fight!"