Anyone who says they are not disappointed in finding out AI doesn't scale past Chiefdom is lying to themselves. The scaling bonuses and boosts have always been one of the most annoying aspects of CIV. Yes it makes the game more difficult, so it achieves the end goal, but it seems cheap and grows old.
I was really hoping with CIV 6 and all the complex systems and changes that they made, that they would finally really scale AI. Hopefully this is a big system upgrade in the first expansion, because it would be really fun to see how well a deity level AI would use all the systems at its disposal.
Only a tiny fraction of the player pool will play on higher levels. It doesn't make sense to develop a superb AI and then dumb it down for a vast majority of players. This is why an AI scaling won't happen soon.