I've not tested repeating any of these cases, but here's what I've seen so far:
-If you open the diplomacy screen and declare, your units will be moved out, same as civ4.
-If you are in their territory and declare by attacking with a unit, the attack proceeds as normal, but all your other units are moved out. Kind of cool idea (if intentional), allows a limited sneak attack.
-If you are in their territory and they ask if you're massing to declare on them and say "Yes" you declare and you troops stay where they are and they think you're honest. Nice when the AIs screw themselves for you, but a very silly mechanic.
As to the original topic, no you can't cancel. Don't give open borders if you think they will attack you in < 30 turns.