I think it was an attempt to reduce 1UPT unit clutter, but I think it has done more harm than good. Units become too expensive, and in the late game the AI loses its units and can't replace them.
I think it was better in Civ III where armies were limited in number; you could only have one per Great General.