I think there was this theory that a tall civ would gain enough from population that it could still field a decent army. Buildings such as walls and castles buff the population benefit partially for this reason. Its not really panning out in my experience though. I'm alright until around industrial era (before that I'm forced to build armories and stuff, which I'm fine with).
However, now techs are coming quickly and new supply sources are not. Population growth doesn't offset the losses to tech at all, even with many boosts. And the next supply building available, the arsenal, is a lot of hammers to pay for something that gives 0 yields. This is when the problems start for me. There is usually another wave of issues when aircraft arrive, because now the idea of a small elite army protecting choke-points is much harder to pull off.