In my experience, every difficulty seems impossible at first. I started out on Warlord (in vanilla) and won my first game with a spaceship victory fairly easily, going to war with hardly anyone. Maybe it's gotten harder in BtS.
When I tried Noble, I was all like "hey, the AI actually declares war on me and I can't expand my empire to infinite." It was quite a step up from Warlord. I also couldn't get
every single wonder if I played an industrious civ. Whoa.
Then finally I got the courage to try Prince. I remember what happened my first game- I was busy conquering some Barbarian settlements and Genghis Khan Gengh-raped (sorry) my capital with Keshiks. Taught me a valuable lesson. Eventually, I learned to build a military early on and Prince became my new official level. Over-expansion also became a major issue on this level.
When BtS came out, I actually found it much easier in single-player than Warlords, which I never quite got the hang of. This got me to try Monarch. At first I was like, "this sucks, the AI cheats and I can only build 3 cities before my economy implodes." But I kept at it, and now I can win about half the time if you don't count all the games I quit early on due to bad starts.
As much as I'd like to say I eventually moved on to emperor, the truth is I haven't tried it yet
too scared. Maybe someday.
But I'm comfortable having ascended to the monarch level, so I continue to play it. It's still quite challenging for me, so I feel no need to go higher.
In answer to the OP, I have always tended to assume that everyone plays at the same level I play at
I just took that test! (-2.75,-2.46)! Wow!