Hmm, maybe this is a bug with my version. Haven't been able to find anything about a fix in the patch-readmes though. Under Foreign advisor, there are three options; "Treaties", "Trades" and "Details". Under "Details" there are the current agreements, but there is no way to cancel it. Either it's a bug or I have missed something, although scrutinizing the menus.