The worst civ to be beside IMHO is the Russians, because they always seem to expand around you, and if you're in a tight position often they can cut you off early on from settling - also, their Scientific trait means that they can become the leader in the tech race, and soon you could become one of their quasi-vassal states if you're not careful!
Another annoying civ to be near (I agree with bloodofages) is the Americans, because often they expand (just like Russia) faster than other civs and generally end up quite powerful. I find the Iroquois to be similar, and this really gets me - they end up being one of the most powerful states (above Rome, for instance, in a recent game), and this is completely unfaithful to history! I know it's about gameplay and not history, but it's still annoying!