The feel of knights differs depending on the difficulty. If Emperor or below, you're trying to get them before the other side. Then use them to pillage roads, resources, etc. They can't take out pikemen / longswords in mass but its really hard to kill them unless the other side misplays. (Using your scout with extra vision is important for this.)
On Immortal+, you have to have a plan for dealing with knights with Swordsman and Composite Bows only. It can be done but requires good city placement.
Part of the strength of knights is that they arrive early enough in the game that all civs might not have a mass of units yet - depends on the difficulty.