I would highly recommend playing an alternative, purely war-focused, sim, to build up your war skills. I also would advise against leaping into the higher difficulties (for multiple reasons) , but specific to what you're saying, I think this is a matter of keeping your eyes on the prize, understanding the "eXploitation" phase of the 4X genre. Just making the game punish you more dearly isn't going to elucidate the systems that are apparently opaque to you. Also, Civ is the sort of game where many decisions are hidden, and even remain hidden by endgame. Contrast this with an Arcade Fighting Game, where, if you get beat up by an expert gamer, or by the hardest AI setting, you can see right in front of you what moves the enemy is doing, and you can adapt by mimicry (to an extent). Anyway, there's a ton of reasons why I think increased difficulty is not for you right now.
Back to war for a moment, with all the other stuff in Civ to worry about, it's easy to oversimplify your war strategy and not grow in that dimension. As well, when you are pampered with technological or industrial superiority, you can easily let that take the place of tactical knowhow.
I learned everything I know about strategy from
Intelligent Systems games, like Advance Wars.
Advance Wars: Days of Ruin (also called
Dark Conflict in Europe) is a non-cartoony iteration of the series that you can get for the Nintendo DS , has lots of scenarios aside from the main quest line. If you're not put off by the cartoony, there's Advance Wars II, in which the individual powers of each "faction" are much more pronounced and are part of strategy the whole of any mission. However, only Days of Ruin has an experience/promotion system.
----
So with war aside, I think you need to read up on the formulae of the game, where all the numbers come from, so you know what you're actually maximizing. What you're trying to do is get to that finish line, which means having all this Tourism, or getting far-far down the tech tree and having productive cities with a safe capital for the space launch, or having lots and lots of money , and maintaining your competitive edge despite sacrificing your spies to CS duty. In short, you're trying to reap lots of resources. The entire game is just getting more resources.
In a nutshell, you are expanding your influence over land, deriving power from that land, and making tradeoffs of some resource for other resources so as to derive even more power and control even more land. Every decision is about getting the most, whatever the most is. Ideally, you can foresee your final result, your "grand strategy", which allows you to value either tourism or science or whatever the utmost, but practically speaking, you want to be as strong as you can on every dimension, when many things about the game are still uncertain.
When you understand that in isolation, then comes the diplomacy. If you don't make nice with the other parties, they make trouble for you. But you are ultimately in competition, so that's why you have frenemies. If your opponents are just the AIs, not Humans in multiplayer, then this isn't really a big concern as the AI really, really, really, pretty much, cannot threaten you. (I mean if you actually roll over and let the AI have its way with your cities, it will scarcely manage to even plunder one of them, suiciding its units on a cities basic defenses.) But for me, this diplomatic aspect, with frenemies, is the whole draw of the game. I just scarcely find value in "diplomatic" games that don't have the grit of an underlying 4X economy.
----
So maybe I've said a bunch of things you already knew (except about the war experience part, I really think that helps you). But you said you go into Unhappiness, which, if you know what all the formulae are, that can never happen because you will manage it. You describe a sort of aimlessness, and I am telling you what you're missing is the genre of the game, the 4X's. Exploit! And to exploit you need to know what the mathematics are. You can't trade off 1
beaker for 1
gold without actually knowing how hard each is to come by. So read up the War Academy, learn what the buildings do, learn the happiness function and learn where science comes from and how city growth works. Learn the typical timeline of things, like how many social policies you'll have by the time your population is at such-and-such. All these things.
And keep in mind, the best strategy was never reached by working out formulae on paper. People tested things. They played a game with one strategy, they made notes, tracked how many turns until they reached Industrial Era, how many turns until they won, how many turns until they settled all their pre-medeival cities. Then they check another approach. A game with tradeoffs (you see on these forums people bandying the phrase "opportunity cost" all the time) is going to be very complicated, because any sort of advantage might be turned into another advantage, you have to check to know which option is really the most valuable in all the situations.
You could simply choose to take other strategist's advice at face value (for instance: Tech pottery first, focus food, get archers, upgrade to Composite Bows as your first priority and defend yourself in your first war; build a National College literally as soon as possible) and that can get you moving (and will have instant results in how easy things will seem and what difficulty setting you can cope with), but I think in your case you need to take some ownership of seeing the pieces in this game yourself.
