I have noticed this as well!
For me though the middle patch was the worse and the FINAL patch made it sometimes better...I'm guess there must be tons of computer calculation going on in the background, but I bet if some good programmers sat down and fleshed it all out they would see that a lot of the decisions the computer is making over and over don't have to be made as much and processing time could be reduced.
Guess that'll be for civ V
or....Sid Meiers ALPHA CENTARI!!! CMON SID!!! Plllllzzzzzzzzzzzzz!!!!