Civ has never been good at making the games stay competitive by the late game, so what's going to change it now?
The civ AI was never competitive during lategame due to frontloaded bonuses - overhaul mods improved that repeatedly, but Firaxis doesn't learn.
If most of the difficulty bonus consists of free settlers, units, and even techs, it's just natural that the AI is extremely dangerous at game start and quickly falls behind once you've caught up.
IMHO, the AI should only get enough starting bonus that they can' be easily rushed.
Their ongoing bonuses should scale by era, however (e.g. start with 20% research bonus, get 10% more each era).
Example (from
https://civ6.gamepedia.com/Game_difficulty):
On Deity, the AI gets:
Starting bonus: 4 techs, 4 civics, 2 extra settlers, 4 extra warriors, 2 workers
Ongoing bonus: +32% tech/culture/faith, +80% gold/production
So I'd take away at least one of the free settlers and some of the free techs/civics (which also prevent peaceful players from expanding, founding a religion and getting an early wonder, resulting in warmongering dominance on higher difficulties).
Instead, the tech bonus could be significantly higher than 32% later on (as the AI will never optimize their cities as we do). An increasing bonus would also allow AI's to catch up, even if you overtook them once.
Also, I'd give free workers when they found a city instead of 2 free workers at game start. Lack of proper infrastructure was always a weakness of the Civ AI, and an immersion-breaker on top of it.
As far as I'm concerned, a newly founded AI city on deity could get up to 2 free workers, 1 free military unit and a free granary or monument without breaking anything - it would help them more than a zillion % of production bonus they invest in stuff they'll never need.
Regarding conquest, we could give the AI a chance to spawn a free unit at their capital once they lose a city ("refugees regrouped and formed a resistance"). This would not only be immersive but also help against steamrolling the AI once you've won against their main army.
_______________________________
TLDR: The AI really needs bonuses that help them stay competitive past the point you've caught up with their headstart bonus (to avoid the "moment of inevitable success" for the player)