FWIW, check out the succession game going on (Epic gamespeed); nearly turn 200 and we have almost no spice.
So the AI's lack of spice by turn 100 or so is not an indication of weak AI; as humans we have other priorities too.
I think its fine that spice doesn't become important until the midgame.
I'm okay with that, since I know the background story (or let's say, the possible backgrounds

) of DW.
...but also Slvynn is right: The spice IS the ressource on Dune and should be the most important thing, at the latest, when "offworld-trade" is researched (just following your argument with no offplanet-contact in beginning)... but as the succession game shows (and many other games, including most terraforming victories, as they often have only few spice), it doesn't matter that much, if you have much spice, few or even no spice at all... maybe raising (late?) tech-costs could solve the problem. Doing that in combination with increasing spice-output, the commerce output AND the "offworld-troop-recruitment" would profit.
Since david has suggested focusing on Wonders next perhaps he can make the SDK change to allow some varied Spice powered wonders.
Also the implementation of regular buildings (maybe a market-orientated building giving 1 gold/spice or something - compare it with tribunal, giving mostly 3-4 gold savings) wouldn't be that bad IMO.
All in all, spice still doesn't mean too much in game, as there are just some "spice-tile-upgrades", but no spice-"trading", exporting, storing... being honest, also a (spice-) refinery should not give +25% hammers, but +25% commerce/gold... or what else if not spice is refinated with the refinery building? The refinery could be the first spice-building, giving +5% (or +10% if too weak) commerce/gold for each spice ressource in stock.
Edit: Having ONE game, where 1/3 of economy (let's say commerce

) is powered by spice, is not a strong enough arguement against spice improvement IMO... I'd even say, it's an argument FOR spice improvement. I read it like: "in most games, spice makes up at most a third of economy. Most economy is granted via cottages or non-spice ressources."
IMO there should be games (not all, but it should be possible), where at least 2/3 of economy is running via spice... especially, as soon as the guild has "returned" to the planet.
Greetz, Hived.