Why can't it be both? Why is there a distinction drawn between "Trying to win" and "Having an enjoyable experience"?
Well ... weren't you actually the one who drew that strong distinction, by labeling in-character (but strategically suboptimal) AI decisions as "stupid" and to be done away with?
Personally, I believe Civ
should pursue both goals (immersing the player into a rich alternative world, and providing strategically challenging gameplay) as often as possible, and strike a balance between them in the cases where it's not possible to pursue both. I'll give an example.
In Civ3, the AI was programmed to rapidly expand at the beginning of the game. This was strategically sound, since rapid expansion (REX) was the best early game strategy in Civ3. But it was also a very unnatural and historically implausible way of developing a civilization, and it felt awkward to many players to have their leaders forego everything else in favor of spreading one's civilization very thin over a very large area (you basically spammed settlers and built little else as long as there was room for cities). Unfortunately, these players were forced to adopt this style, otherwise they'd end up with a pretty huge handicap of being boxed in by the rexing AIs. This led to many complaints by the players who focus more on immersion and roleplay.
Fortunately, in this case there were solutions to that conflict which catered to both tastes. In the example above, the problem was not that the AI was "too strong" - the problem was that the game's design strongly favored a non-immersive early strategy. The AI's adaptation of this strategy was then just a consequence of this (flawed) game design. In Civ4, the designers succesfully reconciled immersion and challenge in the early game by making settlers more costly (and cities with unimproved terrain less efficient). This way, the AI's early strategy could be smart
and immersive, and players weren't forced to adopt a non-immersive strategy any more.
So, in this case, and many others, the conflict between immersion and challenge can be resolved rather easily.
On other occasions though, such a solution is not possible. The reason for this is quite simply that real historical leaders didn't behave like gamers trying to win a game. Ideologically driven, strategically suboptimal decisions are quite common in history. You cannot create a believable, plausible, immersive, rich history if you make all leaders behave like "real" leaders wouldn't.
In these cases, I believe that a balance between immersion and challenge has to be struck. I also believe that Civ4 has achieved this balance quite well. In Civ4 diplomacy, you have "gaming" modifiers (e.g. friends refusing to trade technologies with you any more if you're becoming too advanced) as well as "roleplaying" modifiers (e.g. a friendly AI helping you out although it weakens their position).
What you seemed to suggest, though, was to
forego the simulation approach in favor of a "pure" gaming approach. That would be a bad decision imho, because I believe that most Civ players value both approaches to some degree. (I'm not a pure roleplayer by any means. I, too, like to be challenged by a competent AI, just not in a way that
destroys immersion.) So if someone (like PieceOfMind) says that he'd prefer a nudge into a specific direction, then I'm perfectly okay with that. Totally
selling out to a pure gaming approach by foregoing the simulation elements won't find my appreciation though.