The computer has to do a number of calculations for unit moves and builds for each civ in the game. The more civs that you use, the more processor time has to be devoted to servicing all of those builds/unit moves, etc. This overhead, if you will, is what causes the "Turn Lag" in the late game for those of us who don't have very powerful machines, especially when using really big maps. The amount of memory also has an effect on turn lag.
When I had my creaky old Athlon machine, I couldn't even play on a map bigger than standard because of this, the lag was so bad. My new, very fast quad core is much better, but I still get a tiny bit of lag when using a huge map with more than 25 civs. An example would be sort of like opening five or six applications at once on an older machine. They all slow down because you're asking the computer to do many things at the same time, and it must divide its' time accordingly.
Now, I don't want to hear any smartie pants comments from the i7 peanut gallery about how your computer doesn't slow down.

My Phenom doesn't either. I'm using a single core as an example here to illustrate my point.
Btw, kensington888, what are your machine specs?