Can't declare war on Charadon

Just having my second complete play through of the game and trying to do things differently. This time I accepted Charadon's surrender and he has stayed alive. But despite having built all the culture buildings I can I am constantly being asked to give give my cities to him and I have suffered one rebellion. And my % of the culture of the cites is growing only very slowly. So I thought I'd declare war on him and finish him off but there is no option! Is this correct?

Once you accept, you can't break it.
(Outside of worldbuilder)