You are correct. This is the way to do it. When looking at the active agreements, if you hover the mouse over it, it will turn blue if you are allowed to cancel it peacefully.
By the way, this is also the way to declare war before actually going to war. Just cancel the peace treaty.
Andy S