The lower the difficulty setting, the later barbarians show up, and the fewer spawn. At higher settings, they appear sooner, and have an unnerving habit of popping out of nowhere to create cities. It's those cities that are almost certainly creating those frequent barbarian "waves" you note.
Your best solution is to try an easier setting, or to prepare for barbarians. Try to get two cities going before they arrive, and give that second city a barracks to make it a military unit creator. (Let the first build other things, notably strong, early wonders.) Fortify units whenever possible, since this gradually adds over time to their defensive strength, and put them on top of developed resources. Have military units travel with workers. Grab Bronze Working as soon as possible, and develop any copper resources within your zones of control. Be sure to research hunting, so that even without copper (or, later, iron) you can still build archers. They're far better than warriors.