Would you say the same about chess?
That's a good question. I think no, there is a different layer there in civ.
In chess, the AI plays exactly the same game as the player, and the only thing both players play for is getting the opponents knight or a decisive advantage. Interestingly, if you are really into chess, you learn the moves, and historical games. The amount of possibilities is huge but nonetheless limited - it's possible to foresee all possible moves for the next few turns for humans as well as for the AI. So, in a way, humans play chess like an AI? Or at least very "mechanically." Also, chess is a game that lives from reactions, you don't move your pieces much without taking every opponent piece into account.
The same isn't really true for civ (regardless which one):
- the AI doesn't really play the same game (some rules change, and actions have different outcomes than for human players)
- the AI doesn't really play to win (but it might win by "accident" when it's weighted actions gets them close enough)
- possibilities are endless, there's much more freedom, and everybody plays for themselves for the most part. Tactical moves can be somewhat foreseen by humans and AI, but the strategy is much harder. It's also not as based on reacting to what others do.
- humans don't really play to win in the same way as in chess as well: it's neither about one winning move, nor about getting a decisive advantages. I mean, if you play chess and you lose two pawns early with not getting anything in return, you'd probably resign. In civ, we always accepted that the AI starts with free techs, extra settlers, or "better pawns."
Could we get better AI? For sure! Especially the tactical AI and the basic city building AI (e.g., where buildings and cities are plotted down) could be much better imho. The basic mechanics of the game, if you will, that follow rather strict rules. There are other games which show this in some, but not necessarily all parts, e.g., Old World, Age of Wonders 4 or Humankind (although the latter two "trick" the tactical AI with a limited battlefield). What's more difficult for AI is probably the strategy, e.g., useful long term decision which buildings to prioritize, when to build which wonder, when to build a unit instead of a building, etc. Apparently, there are some weights active that pretend that there is a long term goal, e.g., when the AIs build lots of missionaries to get relics or explorers to get artifacts. Or how they seem to know where a good place for treasure resources is. But then they seem incapable of capitalizing on their achievements and just keep going in the same way.
Very different perspective: last year, I had a few months in which my board game partners weren't available, and so I started to play a bit with board game bots. If the bots are designed in a clever way, this can be surprisingly engaging for gameplay. But the interesting takeaway is that you learn how exactly the bots work, as you perform their actions. And you see how they operate with different rules than the player, have no agency, get random bonuses or perform triggered or randomly decided actions. All with the goal to provide an opponent for the player that's engaging but simple. And they succeed! They hardly model another player, but they function as a game mechanic, as a great obstacle. I think if civ AI or general rules would focus on the same, it could work better as well. But somehow, strategy gamers seem to prefer if the AI plays more or less along the same rules than they do themselves. At least in games on random maps - I mean, the old RTS campaigns of AoE or StarCraft are also examples for opponents that don't play the same game, and that's largely what made these fun. But why do we prefer an AI that plays the same game as we do in civ? Because we can then be more proud of us that we have won against the AI "on even terms"? I'm really not sure why we think it is better, but it somehow feels like it is.