Units alone are not enough to take a city if you have made a wall. I usually build an Earth Wall in every city immediately after I research Megalith Construction. Helps with crime, too.
EDIT: Earth Wall adds 40 defense, city becomes invulnerable when defense is over 80. So a few culture levels are needed, too.