Yeah, I have to say that Civ is not so much geared to providing a challenge as much as it is to providing an experience. Of course it's not like an RTS or MOBA, but it's also not really like solving a Rubik's cube either where time is all that matters and there's just a rote set of rules to the solution. There's a feel to Civ, and certain people will like certain aspects more than others.
That said, beating the higher difficulties definitely requires a transition of mindset from the game as an experience to the game as a challenge. What I find is that the goal of beating the same isn't as difficult, but rather that transition is the difficult part.
It's not a novel idea to build 6-8 Archers, research Construction, and kick other civs around. The concepts of prioritizing the National College and Universities aren't difficult to discover either once you find yourself falling slightly behind in each game. What's hard is to tell yourself, hey, maybe I won't be able to get the Great Library this game. Or if I don't get the religion I want, I'm not going to reload. Speaking for myself at least, I found that it took something out of the experience when I'd play a civ like Persia and not get the Chichen Itza, or otherwise get sidelined out of something that I wanted to try out to see how thoroughly I could pain the map red. But then I took a look at some other players whose LP's I really enjoyed, and how they were willing to play any difficulty at any start. I mean, they would roll with horrible Tundra starts, and turn 40 DoW's from multiple civs. There's a certain feel to that that I liked.
Anyway, that's the best piece of advice I have. Realize that it's going to be impossible to continue to beat up the AI, doing basically anything you want like you used to on King and lower. Be willing to forget about a Wonder if you're beat to it, and just hold out and survive with as much territory as you can until you get to the later game where the AI just starts to lose to itself. Do that and you'll realize that all along you were probably playing to a lot higher level than you thought. I think most people after maybe half a dozen games can give Emperor a whirl and beat it with the correct attitude.
As for specific advice:
1) Prioritize tech. Emperor is not as bad, but the tech bonuses the AI gets on Immortal and Diety are such that you have to prioritize NC and Universities every game or you're toast.
2) Early military. No one ever really lost a game by getting half a dozen early CB's.
3) Expand onto luxes, then sell the luxes to finance further expand. You can match the AI on Emperor without any crazy stuff, but just be familiar with the standard openings.
4) Plan a Victory Condition. Don't be surprised if you're beaten to a Sci victory in the event you didn't beeline the right techs, allocate specialists, sign RA's or build Hubble. Likewise with culture. You can't just warmonger an immense empire and win by default. Even domination victories need to be well engineered, getting key techs at key times.
