Desperate times call for desperate measures. There's no point in making normal war check, continue research and production as usual if looming defeat is obvious and ultimate. If I've lauched my spaceship while AI just begun to construct it and there's no possibility to win culturally/domination and such for him then AI is obviously doomed. In such case there's no point to care about anything except to hurt me as hard as possible no matter what the cost. Currently AI would continue behave as usual after spaceship launch as if nothing happend. Some kind of "dead hand" policy would be appropriate for such cases.
This is pretty much the only case where a AI should think on dogpiling a player , but....
I have noticed that the discussions about inteligent behaviour of the AI commonly fall in 2 fallacies:
- A game with a AI more chalenging for the human is a game with a more inteligent AI
- The AI should use the strategies human use beating the AI
The first is not completely wrong ...
if you have only one AI oponent, a thing that is rare in Civ IV. Normally you are facing more than one AI player and all of them ( ideally ) are trying to win most likely at the expense of the other AI in game. And to say the truth all this discussion of "AI should go burn" is filled with this "one AI" fallacy .... think on it: if a AI player is coded to go burn the cap of a player ( human or not... ) that is close of winning without thinking twice, who profits with it? Most likely other AI that stayed in home preparing it's army to invade the guy that decided to go "bomb Iran" losing a good part of it's army, right? What I'm saying is not that the AI player should not go burn period, but that the AI player should not go burn if it will not win anyway or if it will be in a perilous situation even if it sucessfully thwarts the win and that it should first try to find someone to do it's dirty job if it can ( that suposing that we teach the AI to bribe with a minimum of efficiency

) ... if a AI player wants a city burned, it doesn't matter who burns it, right?

Other thing linked with that is that you can have more than one player closing to a win at the same time.... what should the AI do if there are 2 SS close of being finished, one guy with two legendary cities and close of the third, one guy close of winning via UN and other close of winning via AP

( nothing forbids this scenario .... ) ? Nothing? Try to burn them all? Who to prioritize in case of trying to stop more than one?
The second fallacy is pretty widespread unfortunately

People consistently forget that a lot of techniques or habits the humans have in game are the direct result of how the AI plays.... the extreme example is the liberalism beeline: it only looks so shiny in SP because the AI ( sensibly, I must add ) chooses normally to also tech the lower side of the tech tree, and because of that, the human can buy/steal/extort the techs of the lower part of the tech tree with ease.... It also applies in here: saying to the AI to burn something because a certain player is close of a win implies that you have some kind of alarm linked to some kind of game event, like getting the 2nd legendary city, a certain SS part, a certain % of the votes in UN/AP ( or something more rebuscate ) ... Humans can do it well with stock AI ( and even with Better AI ) because the AI players normally play in a gradual way, stockpiling it's culture in a predictable way, building the SS parts as soon as they get the tech, fighting one war at the time, buinding up it's diplo links slowly... Because of that, humans can learn easily to time their game in a fashion that allows them to go burn something and still have their chances of winning intact. The issue is that, as any competent player knows, nothing forces the aproach to the victory conditions to be gradual: you can pass from 40% of land mass to 60% in one turn, you can only secure the necessary votes for a diplo win in the turn where the vote is cast, you can build all the SS parts in one turn, you can even ( theoretically atleast ... never saw it done , and in fact I'm trying to do that in a SG as we speak ) raise 3 legendary cities from 0 culture in one turn. No matter how sophisticated the trigger mechanism we put in the AI to go burn ( and even suposing that we somehow resolve the issues I talked before ) , due to nature of the victory conditions in civ IV, there will be always the risk of a player ( AI or human ) being caught by a sudden aproach to a VC by other player.
To add , as any player knows, a military operation needs time to be prepared, especially if the player is a rather inflexible computer algorithym

That is why the Civ IV AI has the "hands full" period, right?

Saying to the AI , like you said in the part I quoted, to forget about normal war preparations , is pretty much condemn the AI to do a uncoordinated ( thus, with little chances of suceeeding ) attack , especially if the target player adopts a non-gradual aproach to the win, even by acident ( a unexpected event spread the AI religion to a recalcitrant theo civ, one AI changes religion and sudently will vote of one of the candiadates for the UN win, one civ vassals other and grabs it's votes .... this things can even happen to the AI today ).... even humans, that are normally far more inteligent than the AI, can be easily surprised by issues like this and be unable to mount a sucessful offensive in due time ( I've seen that happening more than once ), so the danger of the AI suffering from that is quite high.
To resume, I'm not saying to tell the AI players to not go burn something to stop a enemy win ... I'm just saying that it might be hard to pull a mechanism that works better than the current "do nothing" aproach, atleast in terms of increasing the odds of a certain AI player winning the game by using that mechanism.