We all know the eternal endgame curse of 4X games (and some other strategy games but not not all of them). When a typical 4X game begins, it is so interesting; but we abandon so many games we begin, because they often (not always) become significantly less exciting the longer game session goes on. What's the problem? War system, victory conditions, diplomacy, yield balance, pacing, eras, tech tree? [well it's visible part of the problem is "I know I have won already, game is set halfway through, but I have devoted separate post below just to that aspect of the same problem]
The problem is deep: growth of complexity across every 4X game session is exponential and those games design fails to keep up with that inherent growth in terms of mechanics and interface, leaving the player more and more overwhelmed and burned out as the game goes on. Player has to use interface and mechanics built around early game very low complexity, to deal with the scope orders of magnitude greater by the endgame. And every game session, in every 4X game, trying to somewhat model human history over time, it simply has to contain rapid, exponential growth of complexity. In the early game you deal with two tile improvements and two barbarian units in one city, over the course of several turns. By the late game you can juggle dozens of cities and units with hundreds of tile improvements, and the more game goes on, the more happens in shorter amounts of time. This is unavoidable.
Not only the more game goes on the more and more happens in shorter and shorter amounts of time, here systems sciencr gives us the idea of emergence - at some threshold critical points, rising complexity creates entirely new qualities which cannot be reduced to their precedessors. In the early game you think of individual tiles and your little city neighborhood. Over the course of game you fluently start thinking in terms of local diplomacy, larger diplomacy, finally "globalisation" and victory conditions. In 4X games thus in every game session you dont just do more and more per turn; your ingsme activities and their menral framework and scope and scale of complexity fundamentally and dramatically changes as the game goes on.
This is not typical for most other video games. In most of racing, sport, action, rpg games you progress, unlock new toys and obstacles, but you think fundamentally about the same things from the beginning to end of your campaign. In Doom Eternal you begin with few toys and enemy types and end with dozens of toys and enemy types but you essentially have the same mindset as you begin and end the campaign: kill monsters by shooting gun at them. Even in complex tactical rpg games, even though you unlock tons of more items and skills and enemies with time, you do fundamentally the same things with your 4 - 6 team members: use skills and items to kill enemies. Relatively few game genres have this rapidly expanding and deepening cognitive load which 4X games have.
This is the problem, because interface and mechanics of 4X games being built and tested and designed around early game, collapse under weight of huge end game numbers. 1UPT district system feels great in civ6 early game, and it is a tedious terror of the late game - by the very nature of things, 100th tile improvement (+1% imperial yield) matters orders of magnitude less than 2nd tile improvement (+100% imperial yield), but you still have to spend the same amount of time to manage it. It becomes more and ml ore meaningless and therefore more and more boring!
It doesnt matter what shallow another mechanics, or number balance, or fancy fireworks you throw at that problem; you cannot solve the above issue of rapidly increasing cognitive load and rapidly decreasing meaning of individual player's actions by throwing even mire garbage on top of that, or tweaking some numbers.
The only solution I can imagine which could work would be to rise to the level of challenge and make game's mechanics and interface evolve as the game goes on. So for example player moves from having to manage every damn mine in an empire of 100 mines (tedious, boring, meaningless) to now having to manage imperial mining policy on a strategic level, or mineral trade in the global context. Less tedious, more impactful.
"Do you have any examples of 4X games dealing well with this problem?"
I think civ5 had one mechanic which was good in this regard, and the fact it was removed is miserable. Ideology system emerging in the industrial era. It was building upon your previous achievements and imperial backbone (so it doesnt "come out of thin air", it emerges from previous complexity), didn't require tedious micro on previous levels of interface, and introduced an entire new layer of complex interactions between players just in time to shakeup lategame.
However, significant offensive against this peoblem woukd need more than one such mechanic dealing well with the endgame/rising complexity problem, the core game design should recognize this issue and attempt to figure out how to keep "not feeling overwhelmed and bored" ratio to "making meaningful decisions" less asymmetric over a game session. Over the entire game design and pacing and balance and all mechanics.
Time will show at what point will 4X realize the nature of the problem (if I am right ofc; I was inspired by my recent months intensely enlightened by systems science in my academic fields)
EDIT
_hero_ and Kan Boztepe have rightfully pointed at the huge role played by "I know I have won already" aspect of the endgame problem. I devote a separate post to it, few posts below, but I still think it ties to the general theory of everything; "there are clear champions and losers 2/3 through the game" is because of exponentially rising power of early dominators, and to change that 4X games need some bottlenecks which shakeup things and make new rules of the game emerge by the late game, allowing underdogs to potentially win. There is actually such historical thing which can be adapted into great mechanic like that in civ series, Industrial Revolution; I have written more about this idea in that post below. So I think this caveat is still within my theory of What's Wrong.