Really strange logic you have there. Assuming you have iron, why not build at least a couple of swords? No matter what the AI loves to spam archers/longbows, and swords are much more efficient at taking them out. Do you really want to sacrifice so many catapults to deal enough collateral damage for axes to mop up?
It is the mechanics of defense in Civ4. If you have a mix of attackers, each one ends up fighting the unit best suited to defend it. If you have a concentration of attackers, you overwealm the "best suited" opposing units, then you get to chew on the soft underbelly units that are bad at defending against your unit.
Archers are weak -- so much so, that an axe with combat II or CR II is more than enough to take them out.
Swords have serious pain problems against axemen. A single axeman can require your stack to be much larger, and cause lots of casualties.
Once you reach catapults, using they become seriously optimal for taking cities. Promoted with city raider, they are as strong as axes directly (which is almost as strong as swordsmen), and they do boatloads of collateral damage.
Only in the mopping up phase, after every enemy unit has been collateraled down to roughly half health, do swords start becoming better than catapults.
And it doesn't take many swords at this point for the mopping up. Many other units work perfectly well for this as a bonus: elephants, horse archers, knights.
The point is, the nearly-all-axe-army with spearmen to defend against chariots is an efficient way to conquor in the pre-catapult era. Play for the now, not for the then -- because after you conquor territory and double your land area, your poduction just went through the roof, and replacing your army for the next era's wars got much easier.