The job of the dungeon master is to design dangerous encounters that CAN kill the player, but it's not much fun if the Arquebus-wielding kobold scores 3 critical hits and kills the player. As a DM and having designed every form of encounter you could imagine and far far more -
Very different style of game, and also one in which you aren't going to have players taking the role of the kobold; if they were you'd want to imagine that the kobold would do everything it can to survive the encounter and beat its aggressor. A strategy game is by its nature a game where the enjoyment is derived from having the superior strategy; it relies for enjoyment on the player feeling they have outperformed/outwitted their opponent. No, it's probably not a style of game designed to appeal to roleplayers - and indeed when I play computer RPGs like WoW I find I get bored and/or frustrated very quickly with the spoonfeeding these games' AI likes to do, where difficulties are set so that it's hard to get killed in the first place, but even if you are this is essentially without consequences. It ruins immersion and any sense that the challenges or events are real if you can't sense that the NPCs are looking out for their own interests and instead will fight to the death just because that helps the players gain experience. As for paper RPGs, I grew up on Warhammer Fantasy Roleplay (the original) - in which combat was brutal and a single critical hit can readily kill a player.
Consider that Civ has had both single- and multi-player options through most of its incarnations. If you're playing against humans, I doubt you'd come on here and throw a tantrum because you're frustrated that the other guy is trying to win. You expect him to. It makes no sense to expect the AI to play a game using the same game engine any differently; and if it did the single-player experience would probably pale in comparison to the multiplayer experience.
the AI should not play to win, it should play for entertainment. Each AI should have "it's goals" and some of those goals should overlap one another at different moments. Conflicts should be dangerous - and winnable at the same time. Reminiscent of RFC UHV's.
This seems to conflate two different issues. The AI does have goals; those goals include win conditions. And in my experience there is no more satisfying Civ experience than a late-game scramble to deny Denmark the diplomatic votes it needs, racing to complete the Utopia Project before the Russian spaceship takes off, or completing your own spaceship just as the Ottomans are beating at the gates of your capital desperate to prevent your victory (all examples from my Civ V games - and in one of them, the Utopia Project one, a loss to the AI). This is playing to win on the AI's part.
What you're talking about is difficulty beating the AI in combat. An AI that wipes you out, even if it's not going for domination, is not playing to win, it's playing to beat the human specifically. This is frustrating and bad for enjoyment as well as for the game. It can be good in some circumstances - in that game where the Ottomans were a turn away from taking my capital when the spaceship took off, neither they nor their Songhai allies could have won, and in geopolitical terms they would basically be handing victory to the Inca, who they didn't even like much.
EDIT: In terms of the city attack issue, I have experienced this recently too, and yes it can be remarkably frustrating! I would agree that this is something that could be looked at - it's good if it's hard to take a city per se, if the AI is in a position to defend it effectively and it's well-placed. But I had one game in which Attila was to all intents and purposes out of the game, having lost his entire army and, with Attila's Court under siege and every unit he brought out being killed, with no way of rebuilding to resume the attack. If the war has already been lost, it should not then take a millennium to capture the city itself (it was in a well-defended position and with one garrison unit).
But when it becomes commonplace and done just for the sake of it, and especially when you invest the time to half-complete a game to lose to a sneak attack rather than meaningfully losing the game because you were outcompeted, then it's boring and ultimately annoying. Not because the AI plays to win, but because it plays to cheese - in the general scheme of things anyone can accept losing games of Starcraft. But no one likes losing to Zergling rushes. It's irrelevant whether the player on the other end is a human or an AI.
Gods & Kings has helped to an extent - in my last full game the Germans hated me for much of the late game, but they wouldn't have gained anything by going to war with me since they were aiming for a science victory, were in the event able to achieve it before I was, and were on another continent at war with Greece anyway. So they never declared war, just rushed to win the game.
Though I've had similar experiences in vanilla, as well as random backstabs in G&K (quit one game after a random backstab from Isabella, and in my current game Bismarck decided he didn't like me even while we had a declaration of friendship), at the very least G&K diplomacy gives you more options to at least give the illusion of control over events. Mongolia has a giant fleet and wants to take Kuala Lumpur? Then if I boast that I'm going to protect it from them, I deserve what happens next. I used to be friends with the Siamese and they take a dislike to me? Well, I did try converting their holy city so they probably can't be blamed for getting annoyed. Even Bismarck in this game - I did at one point have a DoF with his Swedish enemies, and he then took a dislike to my Siamese friends - given that he's a bit paranoid generally, he might start to think there's a pattern... (He's not at war with me yet - at the end of my last session I signed defensive pacts with Ramkhanhaeg and Hiawatha, so it will be interesting to see what happens if he does declare war).
A game should make you thrilled - not mad. The AI should play to entertain and cater to the player. Based on the players actions, the AI should try to "intervene" to prevent success, in a way that the player can respond to.
Unfortunately Civ is badly-designed to allow this generally, a limitation of the core game design rather than Civ V specifically. It's not a particularly interactive game, and the major way to actively interfere with your opponent's strategy is by going to war with them; if you can't just get ahead by rushing to your win condition, that's usually your only option. So an AI aiming to prevent success is one that's necessarily more belligerent - ironically an AI that's not trying to win the game but just to present challenges for the player is more likely to result in these kinds of frustrating situation than one that is playing to win.
An AI that's playing to win won't have any need to destroy or attack other civs that don't represent a threat, any more than a player will - I don't usually go for domination victories, and rarely play to completely wipe out other civs otherwise, not least because doing so will make aggression from remaining civs more likely. An AI playing to win has no motive to behave any differently.
Really nice post, now read the OP and tell me how what you wrote relates to it please. He's playing on Warlord and can't take a freaking city, while we are whining that it's in fact too easy to conquer the AI even on Immortal.
What exactly is the AI in this case supposed to do? Hand the city over for free because the player can't take it? I mean, seriously?! He's playing on Warlord and he can't take a city, so he's blaming the game for it? And you are defending him?!
The way I put this in another thread is that the Civ V AI scales badly. In Civ IV and earlier Civ titles, you could get by at low levels with sandboxing, not specialising, not expanding, etc. etc., all strategies you'd get punished for at high levels. That made the game accessible to new players and difficult for experienced ones.
Civ V has much more of a blanket approach - you need to play as though you're experienced at pretty much any level of play. You don't specialise, you lose whether you're on Prince or Immortal. You don't build enough units - same applies. Which makes the game less accessible to begin with, but once you've mastered the basic strategies that work at lower levels, the same strategies work in much the same way at the higher levels - with the result that after you reach a certain threshold in the quality of your play, the game becomes "too easy". Emperor doesn't really force you to adjust the way you play to win once you've learned how to beat King.
Once the city is capped, 1-2 warriors should be built by the AI who immediately tries to retake the city - a battle the player should be able to win. This level is hardly about competitive gameplay, and more about learning the game.
So, this achieves that - he "cap's a city" - then has to "protect his gains", learning more about the game. Now, if he were on prince, the city would have walls up quickly and a spearman in the borders - The difficulty level should consider the needs of the player and use an AI that is responsive to those needs.
This is all very true, but I think that the AI does attempt this to some degree. I've noticed when playing low-level games with less experienced friends that the low-level AI will not fire with garrisoned forces, which it does by default at higher levels, and it does employ fewer units (simply as a consequence of getting fewer bonuses if nothing else). Indeed this is something I don't think was as true of older Civ AIs, when all that ever changed was the bonuses but the AI was essentially identical at all levels of play.