Skirmishers and Rangers are completely useless. If you need to scout in the Medieval or later, your old Scout can do it (if it survived), or you can use an actual military unit. I'll often send a Horseman, which is a fast unit anyway.
Spec Ops is very niche but at least occasionally useful. You can paradrop them behind enemy lines to pillage some oil, and the spec ops is even tough enough to possibly make it back out. If the AI ever used Balloons / Drones, taking them out would be another use for spec ops. But Skirmishers and Rangers, those just have no valid use. I can build a Skirmisher or, for 20% more production, I can go for a Crossbowman, which has +1 range, +10 melee and +10 ranged strength, and a promotion tree that's useful for actual combat scenarios. It's a no-brainer.