Depends on your style of play. In the early game, two units are good for the Despot military police effect. When I shift to attack mode, I often move some of these units to the front line to bolster the defense there. Once I am to Republic (Monarchy isn't worth it) and have destroyed a couple of opponents, I often leave cities empty. I have a couple of mobile units that can quickly respond to any threats. Barbarians are gone and any ship landing usually only involves two to four enemies.
Border cities get all the units and the best defense is a strong offense, so again actually garrison units are usually one or two. This is important if there danger of culture flipping the city. You lose the entire garrison, so what I do is place troops outside the city to retake it in case of flips.
So where do all the units go? To attack formations near the enemy. Huge stacks of ten, twenty or more. Attack hard, attack fast, attack with overwhelming force. When defending, make them pay a heavy price. A catapult or artillery inside the city is good for softening up units that venture near a city. A damaged unit has little chance attacking a fortified defender. Station mobile units as pickets so they retreat into the city when hit. The computer rarely attacks with mobile units in high numbers so the pickets minimize casualties.
I keep a mobile reserve to respond to emergencies, but large garrisons are not needed for the aggressive style that I play. I am playing Emperor difficulty, random civ, standard size map. Think about this, if you garrison three units in every city and you have twenty cities that is 60 units. Think of the damage 40 units massed into huge stacks can do on offensive and you see where I am coming from.