The point is that you won't need to build chariots in every city, all the time. You just don't need and probably can't afford that much units. You might prioritize units in early game when you have few cities, if you want to go for early conquest but there's no reason to avoid building that T1 building eventually, mostly considering that Civ VI focus on city specialization. Building units everywhere is overkill and eventually become a waste of production, you just don't need that many units. Meanwhile, you could use more science to upgrade your units, culture, gold to keep that huge army of yours...
That's exactly the point! The military AI is so incompetent that it's almost always better to get units than buildings.
I mean Is there even a point in building districts other than campuses?
Yeah, there's that thing called a trade route, know in some cultures as OP route. You get one from a commercial hub or harbor so you might want to have a few of those. Government District can give you a nice production boost, a well positioned IZ will let you pump out those units faster, same for encampments and harbors. Faith can be used to build units if you're going for that strategy. Campus isn't even a priority, commercial hub/harbor take that post. The only district that is really low priority unless you're going for cultural is the Theater Square but it still worth building some eventually. Edit: I forgot to mention entertainment complex and water park, though that is mid/late game. It still relevant if you conquering everything in your path.
You also need to consider that you can play Civ peacefully. Yeah, conquering from the AI is good but far from the only strategy that works. If you're going for peaceful, all you need is a decent stand army to show force and defend yourself.
I'm repeating myself in multiple threads now but Civ VI focus on city specialization. You can and should do both units and buildings, each on the cities that are better suited to work towards that specific thing. Focusing on units and ignoring the rest make zero sense. Instead you prioritize your strategy, in this case being aggressive but still develop your empire on other areas.
