Cody,
The speeds also change the rules for culture expansion, food/growth requirements, hammers gained through chopping forests, and maybe a few other things.
To the OP, I've never had a game under 6 hours. My average is 15 or so, even if peaceful. My epic/micromanagement/warmonger games will take 25 hours or more.