Horseman are kinda lame by themselves, but they're still useful.
The retreat capability isn't much use when the horseman is attacked in the open. Most often, a one-hitpoint fleeing hm will get killed by a second enemy pest in the same turn. But it's very handy as an offensive feature. You can use hm to weaken the strong defenders in a city, without losing all the early attackers.
So moving up on a city with both horsemen and swordsmen, pillaging or what have you with the hm while the swordsmen get into position, can be very effective, because you can take down the city in a turn without losing many units at all. Use the horsemen first, and then the sm - once there are a couple of points taken off a spearman, a swordsman will usually take it down in one attack, even in a city.
At the end you have some red horsemen needing healing, but you have some solid swordsmen in place to quell resistance and defend the city, and the hm's can scoot back to your nearest safe city to recover.
Of course it's way better with 3-movement attackers (ansar warriors, cav, etc) because they can pillage each turn and still come to rest under cover of the advancing swordsman/med-inf stuff.
Upgrade paths are different; it's handy to build up horsemen around a growing civ as peace-through-strength troops (to discourage predators) because you can turn them into knights and cavalry in the hot fighting of those later times.
PTW's med inf / guerrilla upgrade path at least means you don't get stuck with swordsmen chasing around after modern armor, but it's a bit less sexy than the cavalry path - a guerrilla is a bit of a compromise; the defence is nice but weak and by cavalry/tank times, one-movement attackers are pretty useless.
I usually accumulate horsemen in the central cities and swordsmen out in the periphery; because of production rates, you get about the same amount of each, with the horsemen ready to move where they're needed, and the sm contributing to defence.