I haven't seen wandering barbs cause damage in a few months. It's easy to kill them before they have an opportunity to pillage because they prioritize our units and resource-improved tiles... instead of beelining to whichever improvement is closest and least-defended. It's very often a barb will slowly move around a city+archer to get at a resource on the other side, bypassing all the farms on the way, only to die.
Barbarians in camps capture settlers/workers. This creates a 2-tile zone around each camp that requires escorts until the camp is cleared. Since the camp barb has high defensive , mstrengthaking it harder to kill, it can significantly hamper early expansion. Once chariots appear it's even more challenging since it creates 4-tile capture zones.
When they come at your city en masse, they are not easy to kill and do often destroy improvements. Every game where I have several different luxury resources available near my capital, I am always tempted to expand as fast as possible to snatch the spots before the AI. And almost always, I regret it, since the Barbs make me pay for neglecting security.
What makes Barbs a challenge (to me), is having them disrupt your infrastructure and this threat makes you always think security first, especially in the early game. This only comes from the wandering barbs and wandering barbs in quantity. If you can count on only one Barb showing up at your city at a time, then you only need one unit to police the area. He, along with the city sniper, can maintain security (and with Oligarchy, he is a free unit so the only cost was the production time and with Honor he's boosting happiness, so you'd want him there even without the Barbs). When you have to expect two or three Barbs showing up at a time, you have to provide more security and this, more than anything, inhibits your early growth as a player since you must build units and helps give the AI a chance at claiming the prime real estate in the player's vicinity.
The Camp sitters are relatively harmless and and only inhibit settling of cities if they happen to sit on the exact hex you want your settler to build. Most of the Workers and Settlers you find in camps have been captured by wandering Barbs and sent back to the camps. But most of all, the Camp sitters are predictable. As you say, they have a zone around the camp that they will attack occasionally and that's it. The player realizes this and it is easy to avoid confrontations that you don't want; the AI on the other hand.....
I mean think about it: Which is more frustrating to your schemes? 6 Barbs running amok on your improvements and chasing your workers to hiding or 6 Camps sitting out there in the grayed out area?
One more point and I'll wrap up this too long post: When a camp is spawning Barbs and sending them into your territory at a steady rate, it means that if you don't do something about that camp (and the others in the area that are joining with him to make your life miserable), you'll never get to improve your plots efficiently. Especially after you have three or four hexes improved, you just don't have enough warriors to keep the Brutes off all of them. This spurns you to go take on the tougher Brutes in the camps. Without the camps pumping out the wanderers quickly, you can safely ignore them until you want to go hunting for fun and profit.
I really enjoy my early games when the Barbs are making life tough and a rough start makes the rest of the game much more challenging.
Edit: it is also more of a challenge when you go to clear out a camp and find three other Barbs sitting around the camp fire....
.
.
