I have to chip in on this too - I agree. I've been playing the GEM map (maybe a version or two back - 2.62, perhaps?) as India gradually swallowing China, but there's just no defense against Mailed Knights. I put multiple promoted Heavy Pikemen in my stacks, but the Mailed Knights just cut right through them, and I'm pretty sure that Heavy Pikemen are supposed to be the counter to exactly this unit! Then there's the flanking thing and my well-balanced stacks end up shredded by just a few units.
As for buildings, don't make all units like foot soldiers require buildings. You need to trot out units even from freshly made cities, and there would be quite a few more early losses if you had to build infrastructure before you could harness gangs of youths to defend your borders. Having building requirements is only justified for specialist units - siege units I would just about accept, or perhaps those Mailed Knights I'm complaining about. The likes of standard Archer, Melee etc units are too crucial to restrict to cities with the luxury of building peaceably for a while first.