That's 20 turns from whenever you move your forces back, right? So if you keep them there forever, the promise is never fulfilled?
From what I remember, in order to keep the promise all you need to do is hold off on attacking them for 20 (or turns). Doesn't matter where your troops are. It's pretty straightforward.
The promise = I am only keeping these troops by by border as a defensive measure and will not attack you (for 20 turns)
Keep this promise and there's no penalty. (In fact, I think you get a bonus for keeping you word.)
Again, it does not matter where you keep your troops. They're only asking for an assurance that these troops are for defensive reasons only.
This happens in real life all the time by the way. It's 1940. Stalin is worried that Hitler might attack. Hitler gives his assurances that he won't by signing a non-aggression pact. Stalin lets his guard down, Hitler attacks. Hitler takes yet another diplomatic hit. If Hitler kept his word, and never attacked the USSR, he'd get a diplomatic bonus with Stalin.
[I thought it was 20 turns. If it's 50, I think that might be too many and the rule should be changed. I'd be okay with anything up to 30 turns since I usually have some idea whether or not I'm going to attack someone within that timeframe. Even so, I still agree with the concept.]
No, I want to attack him in like 5 turns so all my units except 2 random ones are 4+ tile away from their border. And then it's just plain annoying if it pops up. As said, the rules need to be clearer.
But I think this is totally fair and realistic. By you're own admission, you want to launch a sneak attack in five turns. Perhaps you weren't subtle or artful enough to do it without alerting your neighbor to the obvious threat of an attack. Or maybe you just have an excessively paranoid neighbor (plenty of leaders, especially dictators are known for their paranoia). He asks you for assurances that you don't intend to use those units in an offensive capacity. Makes sense to me - you were planning a sneak attack and somehow the AI sensed it. Sometimes I get "false alarms" but usually when they call me out, it's because I really was planning to attack. I don't see why, when caught red-handed, players should be able to lie without taking a diplomatic hit. I think that would make the game significantly less realistic.
I've launched countless sneak attacks that the enemy has never detected. I've also been called out before attacking plenty of times. I think ambiguity is more realistic than knowing a firm rule, because that's the case in real life - some leaders are more suspicious than other leaders in other circumstances. My best advice for attacking in five turns without getting called out on it? Keep your units as far aways as possible, in a way that will still enable you to carry out a successful ambush. This might require careful use of roads, artful positioning of units, and other tactical elements that might be difficult to orchestrate. That's war. In the real world, you can't always successfully deceive your enemy.