I think Pikemen's worst crimes are really just that: (1) they're a bit overpriced, and (2) they're a defensive unit.
On the defence, 51 combat strength versus 48 is significant, because Pikemen are healing faster in their own territory, whereas Knights are healing slower in enemy territory (and probably getting hammered by city ranged attacks as well, and also possibly cross bows). Pikes may also well be fortified, giving them extra strength. And then you can throw in Oligarchy. And you're also probably building more units while under attack, which unlike you're opponent don't need to travel to the front.
The speed difference doesn't really matter
on defence. You're enemy is coming to you, so you don't need to be faster than them. Did my Pikes kill all the Knights that attacked me when Germany Knight rushed me? No, but I killed a lot of them, and the rest ran away. Germany then surrendered. Job done.
The melee thing is really irrelevant. Yes, Pikes have a negative. But you're fighting a unit one era behind you (Swordsmen), and your first promotion cancels out the negative. And the negative is also irrelevant when your Pike is garrisoned in a city.
And the tech isn't out of the way in some senses. Yes, it's a leaf tech, but it's a leaf tech on the way to campuses, commercial hubs and industrial zones. Knights are just on the way to more things which punch.
The real problem with Pikes is that they are
defensive. First, that means you can't use them to take enemy cities - they just don't work like that way. Second, and this is the killer, you don't need them to defend from the AI. The AI is so bad at combat, that "a good offence is the best defence" could not be more true. I suspect they are also fairly useless in MP, because I'm guessing that's all about the Zerg rush.
Bottom line: if Pikes were cheaper, and the AI much better at war, Pikes wouldn't be half bad.
Melee units clearly beat anti-cav units.
They really don't. A Pikemen, in his own territory, will hold off a swordsmen unit. As soon as your Pike gets one promotion, they are on the same footing. If swordsmen turn into muskets, yep, your Pikey has a problem. But so do Knights - Muskets are nasty.
Here's the killer: if you send Pikemen over the border to wreck havok ... they will just get wrecked. They are terrible on defence, because they don't have all those fab promotions melee units have.