As people have said, the problem is that late game, the winner is already, more or less decided. This is because the advantages of all the previous ages have been accumulated and the difference between the strongest and weakest opponent is so vast. Playing on a harder difficulty level does not really help, as chances are you'll be crushed, either completely or to a point where a comeback is almost impossible.
The solution is to make ways in which the player order can be drastically shifted and where in the score/power/economic spectrum civs can move up or down 10 places. For this to work and actually make it possible to stage a comeback, it needs to be based on something different to the rest of the game. For example, corporations in BtS could have been used as a way for weaker, losing civs (less power, behind in the tech race, less land, less population) to overtake stronger rival civs. However, because acquiring corporations was based partly on technology and partly on GPP, it just gave another advantage to strong civs who would snatch up all the good corporations before their weaker neighbours had a chance.
What is needed is to give a bonus of some sort to civs who are losing in the later ages. The real trouble consists of devising a method in which the bonus is not so great that doing badly in ancient ages is, effectively rewarded, but still strong enough to give you the capability to at least challenge the stronger civs. This would be interesting for the human player for the following reason. If he is behind at this stage, it gives him a reason to not give up hope and the game, but carry on trying; and if he is ahead at this stage, gives him a reason to stay on his feet and fight off contenders.
I must admit, I have no idea what mechanism could achieve this. My first thought is to give civs which are losing badly a diplomatic bonus with each other, so that they can group together to try and stage a comeback. But this is effectively the same thing as giving a diplomatic penalty to civs for wining and brings back the Civ3 scenario where everyone would declare war on you in the modern age if you were wining, which felt very gamey.
Perhaps if you are in the bottom 25% of the civs when the civ who is leading technologically enters a new era you should recieve a free golden age or a free great person? That also feels gamey, and would not be satisfying if this happened to you as a human player.
What might just work is something similar to the quests in BtS. If they were only offered to civs who were behind, and could be completed realistically (unlike, say, the holy mountain one. Did anyone ever actually find that mountain?) and gave a large bonus, as well as unlocking another quest with the potential for another reward.
Example (set in the Civ4 world), you are 2nd to last in the tech race and generally doing badly. You have only just researched Assembly line while the leading civs recently acquired Industrialism. You, get the following quest offered to you:
"The world's workshop. As a source of cheap labour, companies from the developed world are wanting to use your population to produce goods to ship them back to their home cities. Build a factory in 33% of your cities to receive one of the two following bonuses:
+100% to all trade routes for 30 turns
or
Receive a Great Engineer and a golden age"
You then do the following, pick the trade route bonus and now have a massive cash surplus and use it to raise your slider by 20%. At the same point you get the following quest:
"Rise to Power. Happy of the services you are providing to them, foreign companies want to invest even more in your cities. Have a factory in 66% of your cities to receive the following bonus:
Learn all techs which are (at this point) known by at least two over civs"
And boom, you've caught up with the rest of the world, and have lots of factories giving you the ability to build a decent army and get the land/population you need to make sure you dont lose this advantage again.
This is still quite gamey, but I hope more fun. One could say its even realistic by saying it is how China/India/Brazil quickly grew out of poverty at the turn of this century.
tl:dr
You could imagine a whole range of quests similar to this for different eras, each with the aim of making it possible for a civ who is behind at the start of one of the later eras to take over from the leaders. That will add the unpredictability which is sorely lacking in the end game of Civ4/Civ5. I'm not saying that quests are the right way to do that, something more passive may be better, but there needs to be some mechanism to shuffle around the world order, and it needs to only be very slightly biased in favour of the civs currently winning.
i'd be pretty dissapointed if this was leading nowhere, as it usually does!
I love playing in ancient & medieval era. The problem in the Modern Era onwards is that most of the game is done. You're in a very good position unlike in start when you're struggling to establish yourself as a super power.
The game can be made more challenging by few ways. After every era computer checks the success rate of all countries. The AI of weaker civilizations will play much better like it does on higher difficulties. Though it won't get any bonuses but it will play more intelligently. Secondly tech transfer could be introduced within countries which have open borders like in Civ IV mod Total Realism. This would reduce the technology gap between different countries.
