I think technology basically regresses *during* Leto II. Enforced stagnation, little travel between worlds, etc.
Its been a while since I read them, but I don't remember seeing anythnig I liked in Heretics or the others.
Reverend Mothers having weirding is one thing; being able to teach it en masse to combat troops is another.
But I think the original Dune (and sequel) and then the House Atreides/Harkonnen/Corrino books are probably the best and most accessible places to mine usable stuff for this kind of setting/timeframe.
Somethnig else that could be intersting to see; the Conclave of Lords (I forget if thats the right name) as something a la UN. A Conclave ban on atomics is incredibly canon.....
Part of what I love about the setting is that there are all these different power groups who all think they're in charge; and they're both right, and wrong. The Guild, the Conclave, the Emperor, the Bene Gesserit.... all have a piece of the pie, but are reliant on the others.
Also:
So the key Civ idea of several generations of units, obsoleted by technology, has no basis in the books.
Yes, true, but: it works. Its a good gameplay mechanic that is fun to use. A tech tree with new stuff and multiple generations of units is pretty hard to avoid using the civ engine.
The AI understands it and uses it well; a system based on promotions and unit preservation may not get used well by the AI.
I'd leave it alone.
Infantry/melee/thoptor sounds good, but still a little shallow; I think you need to add in some more types.
The basic unit roles from civ work pretty well;
City defender (archer)
City attacker (swordsmen)
Anti-city attacker and anti-anti-mobile unit (axemen, crossbow)
Mobile unit (horsemen, later splits into armor and aircraft)
Anti-mobile unit (spearman, SAM/AT infantry)
Artillery (catapult)
Why are melee units 2 moves but ranged infantry aren't? Doesn't quite make sense.
So I'd consider:
1 move:
Infantry (city defense)
Grenadiers, Bursegs/Saudakar (city attack shock troops)
Thoptor (aircraft, anti-bladesmen)
Sand rover/quad/scorpion (mobile, anti-artillery)
Bladesmen (anti city attack, anti-missile use)
Missile trooper (anti-mobile, anti-air)
Artillery.