In general, critiquing units depends on difficulty level, as Diety AI through sheer numbers and ease of replaceablity alters the debate.
I find mounted units Immensely useful because they can move after attacking. You can cycle these guys if you've built invasion roads, and you can of course attack from many tiles away.
In general, you're fighting one of 2 wars against the AI: 1) Smash and grab invasion, or 2) Kill Zone defense. If it's on Diety, then there's the heightened scenario of war of survival.
In option 2, a Kill Zone defense, Mounted units are fantastic. Again, a well-roaded defensive spot means the AI moves in and promptly gets assaulted by range (Composites, XBows) and also horses. You just want to be sure the horses are not finishing off the unit, so it doesn't end a turn in the wrong spot. You want a holding melee as terrain depends, but roads, range, and mounted are the ideal tactical units to kill enemies at speed once they've entered the area you want them to enter.
In option 1, AI smash invasion, the key is to move hard and fast and take the AI cities you want. In that scenario, you generally overwhelm with range, put a few cheap melee as shields to take city shots and die, and then have horses fly and take a city or take out an AI blocking unit. Knights and xbows are the invasion combo of choice for speed and ferocity of attack. Now, if you can't take a city quickly, then the units required are different... but frankly, if you're invading the AI rather than defending, and you can't take the cities, then you're probably in the wrong tactics.
Some specific scenarios:
Byzantine Cataphracts are stunning city killers. Less city penalty, higher strength, and again, they move after attack. You can do serious work with these guys as Byzantines. Songhai Knight upgrade, the Mandalaku, is similar but less era-dominant. Still, quite helpful. And a UU with 5 movement helps cycling more.
I've said this before, but the critiques of Byzantines tends to focus on religion, but rarely do you see it mentioned as a very smooth expansion civ. Ideal for Tall 3-city opening and then swallowing up a neighboring AI.