Txurce
Deity
Back to my Paratrooper reasoning then...
The need for a Vanguard class is already greatly diminished in the Industrial era (and even Renaissance to a point). You don't have to worry about being unable to build your bread and butter unit due to lacking Iron. At this point they exist for low cost, recon and mobility if you don't have horses. Once you hit Modern era, all of these things become significantly less valuable. If you're a conqueror you mostly have your recon and mobility roles filled out with other specialty units (tanks, planes, helis) and low cost isn't as much of a priority as battle effectiveness per unit. If you're tall/peaceful you don't particularly need recon or mobility either, and while low cost is useful in a pinch, I would still favor Mech Inf for bread and butter defense, especially since you become unit soft capped at around 15 units.
Making the last link Special Forces (replacing Paratrooper) with all those promotions available does several things. It provides a sensible upgrade path for your existing Vanguards. It creates a rapid deploy assault unit for conquerors that isn't overpowered (tweaking strength obviously). It still allows for a modern recon unit that, while I stated isn't as important, still has its uses. I think the cost on these should rival Mech Inf because even though they will be weaker in strength, you're paying for versatility and mobility. It fits a theme. Special Forces are a first in unit to wreak havoc on supply lines, harass the enemy, create subversion, etc. In Civ they wouldn't be able to do full blown assaults on main units but they will be useful for holding ground, pillaging, killing weakened units and knocking out AT guns, AA/SAM, and Artillery.
Your first paragraph effectively illustrates the natural end of the need for a Vanguard class - just as Rifles signal the natural end of range units. There's nothing wrong with the demise of either.
Your second paragraph strikes me as conflicted ("a rapid deploy assault unit for conquerors that isn't overpowered", "In Civ they wouldn't be able to do full blown assaults on main units but they will be useful for holding ground, pillaging, killing weakened units and knocking out AT guns, AA/SAM, and Artillery"). The word "assault" isn't in the Vanguard job description, and knocking out artillery, etc, is definitely significant offensive damage.
Again, my point is not to drop the paratrooper as the last link (although I don't think it's needed). It's that it should be just as weak relative to its time as the rest of the Vanguard line.