OK, but 3/5 of your games require mods to have "good" AI.
I was a bit in a rush and not detailed enough with my words. It is absolutely the case with SMAC and HOMM5, but Pandora's AI is even without the mod giving it the final touch already quite decent (as said, the modder equals the AI programmer here - the mod is more of a final AI patch never having released as official patch by the publisher because of...well, probably the typical publisher reason of ending official development at some point

)
And usually, that just means harder AI rather than more fun AI.
I have trouble to get what you mean by "hard" (ruthless and designed to win at all cost and/or loaded with "cheats"/boni to be competitive?) vs "fun" (just weak so that everyone can win? Or rather behaving human like in a sense of some variance or caring fro diplomatic relations?) AI.
For the games I listed, I can say that good AI here does not mean "good by many cheats/boni" (not to create another misunderstanding: All those games do offer those "impossible" difficulties, where the AI gets boni left and right. However my impression is solely based on games on the "equal foot levels").
For "being ruthless and having desire to win" - yes, that is greatly increased in all cases. The games vary in their topic and the overall peace/war leaning they have, so in Gladius and HoMM5 there is little "building" anyway. Pandora and especially I:GW maintain a variety in AI behaviour though - if you alone have a look on the modifiers affecting diplomatic relations in I:GW you will be surprised, if you e.g. think that "a lot of things affect diplomatic stance in Civ6" - I think I:GW has more:
https://steamcommunity.com/app/1183470/discussions/0/2791621875954477767/ (54, if I counted correctly)...and some of them a pretty "human". Pavel, the dev, said about that system that the intention is to guide the player to treat the AI like a human and not an AI. It's not perfect, but is closer than comparable systems in e.g. the Civ series or Paradox Titles.
Probably the case of SMAC is closest to what you describe - with the mod all AIs expand likey crazy and get strength...and all tend to relentlessy harass you, if you just try to peacefully buider. So yes - here the AI has become more uniform
I think the pattern is clear: 4X games pretty much never have great AI. Sometimes they have artificially hard AI and sometimes they have AI that's fun to play against, even if it never wins, but I've yet to see a game that has both. I guess there are probably a few out there, but I suspect that those games are much simpler, too.
Hard to answer that, as it depends on how you define complex. Also my examples vary in terms of how close the come to Civ. The range goes from SMAC (which is "Civ in space", but closer to Civ2/3 in its mechanics), over Pandora (which has great similarity with SMAC, though lacking some civil aspects like Social Engineering) and Gladius (graphically similar to Pandora, but with a greatly reduced civil part - however, with a deeper combat system), to I:GW (which indeed follows the principle "less features and numbers of different units/buildings/whatever", but what is in is interwoven with everything else). HoMM5 stands hybrid between a roleplay/startegic game a bit aside, as it is almost a stretch to label it as classic 4X.