2 things, healing needs a full turn of inactivity to kick in, and healing isn't instantly going to 100%:
Outside your borders it's 10 HP per turn.
Inside enemy borders, it's 5 per turn.
Inside your borders but outside cities, it's 15.
And inside any of your cities, 20.
Extra: naval units can only heal within your borders unless they get the appropriate promotion.
Extra 2: medic units give extra healing but I don't remember off hand how much.
Extra 3: units requiring a strategic resource and/or a tech require you to have these in order to heal, so if you lose iron for example, your knights/swordsmen won't heal. If you get that Great Admiral who gives you an Ironclad before you get Steam Power, you can't heal it either.