Unfortunately this correlates a lot with the game being interesting. AI is really bad at handling complex logic over large maps, while human brain is extremely effective at managing terrains (and that's why we love such games). Going from 8x8 chess board to 19x19 go board made a difference of 18 years in AI development (AI beaten chess world champion in 1997, but for go it was only 2015). Managing even small map in civ the way AI manages those games is way above modern supercomputers, not to mention home device.
The only reason why Civ4 AI looked good is because the game lacked tactical component of Civ5+ and huge AI bonuses were outside of player view. The second part is true for Civ5 Vox Populi as well - it takes a lot of effort for AI to appear competent, but the real challenge it provides comes from the amount of bonuses even bigger than the original, just going behind the scenes.