You're right, this isn't a problem with the drivers, it is civ 4 and civ alone. The only reason I recommend updating drivers is that often the gfx makers fix the various problems that the games themselves cause.
And after doing some research, I think the cure is to skip all the way to my last...