Historically, which should be the First Argument for changing something that doesn't seem to work in-game (after all, several dozens of millions of people working with the problem for centuries are more likely to come up with 'solutions' than a single game designer or even a group of game designers, no matter how brilliant they think they are - or how much input they get from a few dozen Fanatics) here's what Limited Units , Armies, and onquests.
Early Game (pre-Industrial)
Armies were limited by the Upkeep - the 'Maintenance Costs", because governments simply did not have the administrative structure to extract large amounts of Gold/Cash from their population and apply it to supplying armies - and the technical means of supply were limited. Effectively, no large body of people and animals could be supplied away from a coast or river, so most armies ended up 'living off the land' - and starving to some degree.
Raising Units was limited by the cost in People. Armies required, generally, young men in good physical condition, precisely the prime workforce for everything else in fhe Civ: take too many of them away to drag a spear, and everything else suffered. That meant, while Civs like Egypt and China could (and did) 'conscript' an army or workforce in the 100,000s, they could not keep them away from More Useful Things for very long. A Professional, permanent army, on the other hand, required an even greater numer of people working to support them, or they were limited to that part of the population (Warrior Class, militant Aristocracy, comitatus, etc) that could support themselves while they went away to fight. A large percentage of the population doing that resulted in an extremely 'skewed' society - see classical Sparta, in which every adult male Spartan was a professional warrior, but it required a large slave (Helot) workforce that the adult male Spartans spent most of their time watching closely, with no time left over for things like art, trade or culture of any kind.
Conquering Others was limited both by the above problems of raising and maintaining armies away from home, but also by the fact that there were very few mechanisms - social, cultural, civic - for integrating Others into your own Civ. Persia, Rome, China managed it, but the Chinese in fact took centuries to integrate non-Chinese cultures like Northern Barbarian pastorals (in fact, they are still working at it a thousand or more years later with the Uighers). That meant, with those exceptions, Empires tended to be ephemeral: they rose and fell, came and went within generations or years, not centuries - Mayflies in the game's timescale.
Late game (Industrial and later)
Armies are limited by population. Since States can draft or conscript virtually anyone, the problem is that taking workers and farmers out of industry and the civil economy cripples all support and supply for the army you formed. Germany in WWII had a Manpower Crisis as early as 1940, because they had stripped out too many men into the army and their industrial output was falling for lack of workers. They 'solved' the problem by using slave labor, a solution that was ridiculously inefficient (1/3 the productivity per worker on average) and added to the implacable nature of their enemies - and got several prominent Germans hanged after tthey lost the war. A small extremely skilled army can offset this by keeping most of the population working while the 'professionals' do the fighting, but that only works for Short Wars: no professional force stays professional in a long war, because wartime casualties cannot be replaced with equally skilled and trained men, so the Quality inevitably declines. Historical examples are the German Army in WWII, and also Napoleon's armies from 1805 to 1815 and the British Army 1914 - 1918, which started out extremely Professional and ended up almost completely Amateur due to extreme casualty rates.
Units are limited by Expense of both raising and equipping and maintaining the equipment. Modern weapons are not only relatively expensive and complex to produce (requiring, among other things, massive quantities of Resources and major Industrial Plant to manufacture) but they also require continuous streams of ammunition, fuel, spare parts, etc to keep fighting, and all the infrastructure of supply, transportation, repair facilities, etc. On average, a modern combat force has 3 - 10 people 'behind the lines' supporting for every man up front fighting.
Conquering Others is now limited by Ingrained Culture. Andalusians to Zulus, Chinese, French, Germans all Identify keenly with their own, and will not willingly change in less than generations or centuries, if at all. That means, while you can conquer an area and its people, keeping that area and those people requires constant Maintenance in the form of civil and military administration and occupation - all of it drained from your Civ. Again, using WWII as an example (because I've written books on the subject so I'll stay with what I know) in 1941 the German military massed about 150 divisions to attack the Soviet Union, but had to leave another 60 divisions out of the campaign to hold down their conquests in the rest of Europe - in addition to large numbers of 'nonmilitary' occupation forces like police, SS, Gestapo, and civil administrators. And despite in some cases 6 years or more of occupation, at the end of the war the conquered and occupied people in Scandinavia, Poland, Austria, Slovakia, Belgium, France, etc in no way identified themselves as German - quite the opposite. After over 300 years, the Scots are neither English or "Great British", but distinctly Scots and increasingly restive about being associated with the other two. This is fairly typical World Wide.
So basically, Initial Cost in Gold and Population, Maintenance Cost in Gold, Manufactures and Population are the constraints on Military Units, armies, conquest throughout the game, but the importance and emphasis of individual effects changes from start to finish. And a lot of the changes are in the techniques and mechanics available to the Civ: the ability to 'mobilize' Gold, people, and manufacturing for the Leader's purposes is different and changes massively as the game progresses - or should, in a well-designed Game.
, this is just another view, and IMO closer to reality, but not crazier than Abe in the 4000 BCs...
