If there's no challenge in their flagship product they will lose a large chunk of their customer base. That isn't what any company wants. Which leads me to the logical conclusion that the fixes needed to AI isn't as straight forward as many would like. Which makes sense given that districts (like 1UPT before it) are harder to get the AI to understand as a human does.
I disagree.
I don't think they will lose a large chunk. Most games have an average playtime of what? 100 hours is a lot for most games. If you play 100 hours of most games, it usually means you're happy with it. For civ, you have barely scratched the surface. But most casual players will be more than happy to play 100 hours, buy a new DLC and do something else. The time of fun per buck is good.
However, Civ has the potential to hook you for thousands of hours. Civ IV certainly did for me. Civ V and VI definitely not. But why would Firaxis cater to someone who wants to play 1000 hours of a ame they buy once when they can have a game that is played by 9 other persons for 100 hours only, for the same resource cost?
They simply don't have any incentive to make a challenging game. OVer 100 hours, players explore and discover, they are not in challenge mode. The first time you meet an opponent, you're afraid and happy you managed to push back his army. Then you move on to something different, and you don't realise that you were not actually ever in danger. Unless you play a lot. And players who play a lot don't bring money, so the company will not lose a large chunk. They will lose a small, vocal minority, and even then this minority can still realise that the game isn't as good as the previous versions, but it's still a decent use of their money.
Firaxis has never been able to provide a good AI. I'd love to be proven wrong, but considering that a simple "if you're 6 to 1, go and suicide some units against the city" would make the AI way mroe challenging, I don't think it's technical limitations that stop them. Tactical AI is low on their priority because the AI appears to be threatening when it has a lot of units, and that's enough for most players. It doesn't have to actually be a chalenge for them to sell the game.