You can also right click the unit in your city screen, then choose from the options presented. The best way to do a mass upgrade is in your f3 screen. Select display by units. Right-click on each unit, and you will get the same result as the city screen. As you upgrade, the next one in line moves itself into the old one's place. If it can't be upgraded at this time, move on to the next icon.