Attack barbarians, you'll have a attack bonus vs barbarians on all levels except deity. Besides, you should be building offensive units over defensive ones anyway, so your attack value will be bigger than defense value, giving a bigger chance of victory with attacking.
Also, if you attack them, you don't have to worry about them pillaging improvements, and your workers will be free to roam again.
By attacking their camps you earn some gold that you can use in trade or to upgrade units.