THIS explains EVERYTHING. I am serious. You just need to read Soren's epilogue of civ4's design vision (at the end of civ4 manual), and compare it to Shafer's vision. I am serious, not whining or anything.
You can pretty much understand what happened after comparing both visions. The first is a vision of grandeur, of a boy dreaming of building the ultimate empire building game... the latter is the vision of a boy building a wargame with some cities in between that serve as tank obstacles (if you reach that era with AI's left, that is).
The difference is immense, and is reflected in the game design.
That, and the fact that Soren graduated from Stanford... the other guy, well, as far as I know, is one of the top 10 of UoL (University of Life).
'Nough said.