Yeah, raging barbs certainly does slow down everything. I've noticed comparing a Pangaea with raging barbs on to one with barbs off, the land mass fills up much faster without raging barbs. You figure that every unit that gets killed by the barbarians is one more that has to be rebuilt, slowing down worker and settler production, as well as economic losses. Barbarians are constantly pillaging, meaning the workers that ARE produced are much less effective, constantly having to rebuilt improvements.
This ends up giving a key player advantage though, as you are more able to adapt to the raging barbs, and run your defenses out a little more aggressively, preventing the barbs from getting into your improved territory.