I have seen enough examples of the AI dispersing independent powers that I can confidently state that it has been programmed to do it. It just has recently become more hesitant to actually do it. I suspect that they tend to not disperse when someone is trying to befriend it and they like that player and don't want the relationship to deteriorate. Or it might just be my rationalization and it is just a bug. Problem is, the AI cannot explain itself, so sometimes it is hard to say whether this is working as intended.
You are quite right, and I overstated the case.
On the other hand, you are also quite right that the AI's behavior on many occasions and in many game instances defies logic or strategic analysis.
While I have also seen numerous IPs 'disappear' from the map, especially in Antiquity and Exploration, which IPs disappear is a mystery.
Hostile IPs right next to an AI Civ are still harassing that Civ all through the Age, while other hostile IPs some distance away are neutralized by an expedition or made friendly with Influence.
A hostile IP that wipes out a settlement is not counterattacked, but instead that entire direction is abandoned - possibly a strategic decision, but not consistent because on other occasions the AI Civ throws armies at the IP despite having a major war with another Civ going on another front.
And in its use of Leaders and deployment of armies the AI is, at best, mediocre. Very, very rarely does it use Leaders to bring up entire armies at once (I never saw that before the last patch, have seen it exactly 3 times in a dozen games since) while usually the Leader is wandering around alone like some very expensive Scout, and with about the same life expectancy. The AI does not consistently use ranged units - although it does build plenty of them, unless it has a Unique unit of another type, in which case everything seems to be subordinate to spamming the Unique. Rome and Greece are grand examples, fielding waves of Legions or Hoplites with almost no ranged units at all. Regardless of how good the individual unit is, this kind of army is easy meat for a mixed force of adequate melee infantry backed up by good ranged firepower.
The AI does prioritize cavalry in Exploration, even Leaders without any cavalry bonuses attached field armies almost entirely on horseback - but the AI does not seem to understand combined arms at all, even as defined by the game, and this makes all of its forces relatively easy to defeat by any human player who is paying attention.
The result, unfortunately, is and has been for over a decade a game in which combat is, after you learn the basics, pretty much mindless and not much fun at all. Refighting Little Big Horn as the Lakota is only fun a couple of times before you look at the starting situation and wonder "why bother?"