I can think of only one possible way to do this, but it comes at a cost (more on that later).
Open your game, then open WorldBuider (WB). Save off a WB file and close the game. Find the WB file in the saves folder and open this file in Notepad. The WB file contains a lot of info about the game settings and map, but it is actually fairly easy to read/modify.
If I'm correct, the Permanent Alliance should be in an "option" line in the very top section of code. Just delete that line to remove the option.
Next, in the following sections will be the code related to each individual player. Each player is represented by a team id with the human being Team 0 (if I recall correctly). My assumption here is PAs work similar to when you start a game with Teams, so once a PA is created the two players (AIs and/or human) will share the same Team Id. So to remove it, I simply think you need to unsync the Team IDs by giving one the appropriate number - in sequence - that would exist without the PA. (Note: I've not done this particular thing before but I think this could work based on what I have experienced messing with the file before)
Save the file and then play it as a scenario. (You can also play as a custom scenario and add/remove settings here as well) Note: doing this adjust barb techs but since I assume your game is quite late this is probably irrelevant.
Now, here's the kicker. The cost of doing this is that all accrued stuff like research, great people points, and gold are set back to 0. For instance, if you had X amount of research into Y Tech, that research has been wiped away, so you must start fresh on that tech. Your bank - gone. GPPs - see ya. Basically anything that accrues turn by turn. This can hurt quite a bit a later stages in the game when you have a lot of stuff going on and generate so much per turn. One idea though is to check what is going on this turn, and try to wait until you finish the next tech and/or great person [don't run any specialists in other cities for that turn(s), before saving off] Like spend your bank as much as you can if it is large. So if you can wait for this a bit, it might limit the damage. (Oh, and note that when finishing a tech you get tech overflow so do the math and adjust sliders so you don't over-tech before saving off WB file)
You might consider doing this on an older autosave as well just to go back in time a bit.
