Civ 4 represents the result of 15 years of refinement of a genius concept by people who really love the game. Civ 5 represents a total from-the-ground-up rebuild initially guided by a vision that boils down to "I really like Panzer General, and I want diplomacy to be as vague and frustrating as possible." From there through Civ 7 this idea wasn't so much refined as halfheartedly fixed up with other ideas of varying quality kind of randomly pasted onto it. They're effectively two completely different series, and the first one ended with 4.
More random ramblings on this...
Just played my annual couple of games of Civ 4 and have to agree. Following after a run through of Civ 5 and a Civ 6 game, after not playing any Civ for over a year, it's like seeing the series with fresh eyes (after playing them all since Civ 1). And I noticed more strongly than ever, how different Civ 5 is to what went before. From Civs 1 to 4, the basic idea is refined, mechanics improved, the GUI improved. Each starts it's design where the previous left off and refines further, changing what were generally perceived as weak points. Also taking advantage of improving computer technology. But it feels to me that Civ 5 dumped this approach. As if Civ 5 threw Civ 4 away and started again. I think Civ 5 is "OK", not terrible, but still, at times it feels like Civilization for people who disliked Civs 1- 4. So it's no suprised there is this divide between many Civ Fanatics. Those who liked the series up to Civ 4 and those who live Civ 5 and 6 (and will lap up 7), but have no interest in the first 4. It's like instead of getting a focus group of Civ 4 fans and asking them what they liked and didn't like in Civ 4, they sat down with a bunch of gamers who all hated the Civ series up till then and asked them "what can we do to perhaps get you interested in a Civ game?" Civ 6 does this even more: "make it look like a mobile phone game".
Why this happened, to what level it was deliberate and what level was incompetence, I have no idea. Probably many factors. It really struck me though, just how different Civ 5 is. As if Civ 4 was a failure! It clearly wasn't, so I'd guess it was more that the designers of 5 didnt' have a deep enough understanding of the series till then. Yet despite that, they stumbled upon a formula that worked and brought millions of new Civ fans in ... while turning thousands of existing fans away.
The funny thing - I've played loads of Panzer General and Panzer Corps over the years. Played PC2 a lot recently. Also games like Order of Battle and Unity of Command. I have no problem with 1UPT. I really like it in those tightly focussed tactical wargames. 1UPT is an abstraction that works well in tactical wargames on maps of small areas. I hate it on the global scale of large scale, global strategy games like Civ, though. Thought I'd add this just to show it's not that I have some irrational hatred of 1UPT! I like it. I just don't believe it is the right solution for every map in every game. And definitely not the one for a Civ style 4X game.
Back to my recent play throughs of Civ4. I do believe Civ 4 has issues that needed improving. I have no problem with stacks, but when the AI turns up with stacks of something like 100+ units, the game does drag! There is a middle ground betweem 100 units slugging it out and 1 unit, though. I much prefer the Civ 4 AI to both 5 and 6. It's not perfect, but it does behave mostly coherently. The characters like Montezuma and Tokugawa, the different ways they play, are still fun to this day. Again, the diplomacy of Civ 4 needed refining, not throwing away and made bafflingly absurd and frustrating. Similar the road and rail building spam. It gets a bit monotonous by the end, but needs refinfing, not rejecting. Apparently Civ 7 will not have builders at all. Since Civ 4, this is how anything that casual players find a little awkward is dealt with. Take it out.
BTW On Stream achievements, it says 34% of players of Civ 6 have declared a war. Perhaps this is a clue as to who the game is now aimed at? Players who want to lay down 5 or 6 cities, then click end turn for a few hours until winning. This is how Civ 6 feels to me - though I am one of the minority of naughty warmongers who has actually declared a war!
Anyway, I totally agree with your comment that Civ 5 is a completely different series. Playing the games now, after even more years to let change settlle, made this clearer than ever. It's not just "old people" either. My children loved Civ 4 and both still occasionally play it to this day, preferring it to Civ 5. And neither has any interest in Civ 6 at all.
I agree with those that say a Civ 4 remaster would be excellent. Well, I'd be delighted with a remaster of any of Civs 1- 4! Honestly, any of them remastered would be better, deeper, more interesting games than Civ 5 & 6 .And 7 most likely. because if they reintroduce any of the things that made Civ 4 great, it is likely that it will annoy the new fans of Civ 5 & 6.