Good analysis.
Here's an idea out of left field:
One way to make the end game more challenging could be creating Stellaris-type end-game crises, i.e., emergence of very powerful entities that overwhelm the unprepared, and may even need collaboration to withstand.
It would need some imagination to get it to fit the Civ environment, and perhaps try to fashion different scenarios targeting different game styles.
Some ideas:
- (the obvious) extraterrestrial invaders, even Mars-men ;-)
- a powerful previously undetected civ with large technological/military advantage (would need a limitation on how much of Earth can be explored), but to could also be an undetected "marine" or subterranean civ
- AI uprising creating armageddon
- Emergence of a Financial Giant/Cartel "secret" organization that messes up economies across nations
- Drastically increase impact of the current end of era crises, like a Bioweapon running out of control, enough to scramble existing balances of power
etc.
Actually, "very powerful entities" just has to be tweaked to include 'very powerful Events and Forces' to apply to a historicalish 4X like Civ:
The Industrial Revolution introduced an entirely new social/population class, the Industrial Worker, which took a century or more for governments and societies to come to terms with (if then), and it up-ended politics and society and led to the 'Ideology' crisis that wracked the early 20th century with warfare.
Ideologies, already used as an excuse for disliking your neighbors in Civ VII, could be expanded to show their real impact, as they warped governments and societies both. While the impact of Fascism and Communism were obvious on Germany and Russia, FDR's New Deal of government intervention in the USA was almost equally radical, and a new American civil war or revolution was avoided by a smaller margin than is usually suspected.
The growth of and interdependance of Governments and NGOs (Non-Government Organizations) is nothing new, but getting more intrusive recently: international bankers and traders started warping government decisions by at least the 15th century CE, and kings and governments borrowing from international bankers and then not paying their debts brought massive instability to both governments and the banks. And the series of financial 'crisis' events occurring almost like clockwork since the early 19th century that disrupt society show that no government controls them, no matter how they 'tweak' their control over their economies - there are too many Outside The Government factors at work
The impact of humans on the planet expanded and accelerated in the past century, and even if the overall planetary impact is ignored (as many are attempting to do today) local impacts have been massive in things like the Dust Bowl in the USA or the on-going drought and starvation in north-central Africa. Climate Change, even temporary, has been a factor in human settlement since the beginning of the Neolithic, with major disruptions or contributions to making the disruptions worse for societies and governments from the 5th to the 17th centuries CE.
So there are a lot of ways to provide an End Game Crisis or set of calamities to challenge the gamer - some on-going throughout the game, like the planetary or semi-planetary climate/ecological variations, and some Unique to the late game as individual Civs and governments become more susceptible to Outside Factors they cannot ignore.
)
]
