Alright, I play BtS, so keep that in mind.
But, I had a similar problem when I got sick of kicking everyone's butts on Easy difficulties with peaceful builder philosophy.
The main things I've discovered is that you've got to get militaristic, if even just a little bit. Even as a peaceful builder, you need a strong army to defend what you've created from the uncultured hoards. And, if you can take out the guy next to you early, why not? A lot of things can happen in the course of a game and, even if he's looking like a loser now, that could change. Kill them and take their women. That gives you, not only clout and gives the other civs a reason to give you a break, but it gives you some breathing room on your borders. A strong military is not optional.
As a peaceful builder, the tendancy is to just try to build everything that's available and then maybe some military units later once you've run out of stuff or at least stuff you care about. But look more closely at the city and figure out what the city really *needs*. In my mind, there is no given formula for this. But don't go building a bunch of libraries on the borders of hostile AI's. At least, not until you have Walls and Barracks and a staunch defense. AI's tend to suck at culture, so you probably won't have to worry about their borders overtaking yours.
Another thing is, and this is critical, use smaller maps. Use them as stepping stones. The game is a lot more transparent with 2-4 AI's playing, instead of 10. You can use this to analyze your playing more critically and streamline your play style, once you've masterd Tiny, move up to Small. Use them as stepping stones. Not only did I find this very helpful, but the game runs better.
I had the exact same problem you are having in transfering into higher difficulties a few months ago. After scaling it down and analyzing my mistakes, I've got a win in Price difficulty.
I'm sorry if that wasn't very specific, but that's more or less exactly how I addressed my problems. Basically, there's a hitch in your swing and you need to do some batting practice to work it out.