More units existant = More delay between turns
Larger map = More possible units
More Civs = Faster approach of the unit 'limit' (delay between turns increases at a greater rate)
Fewer Civs (below map default) = more space for barbarians = more barbarians units = possible early onset of delay (worst on larger maps with fewer civs)
Barbarian World option = barbarian cities are on the map at the start = many extra units in play from the beginning of the game (increases the starting turn delay, worse on larger maps)
Raging Barbarians option = many more barbarians can spawn = increase in delay once barbarians start spawning (worse with larger empty spaces and on larger maps)
No Barbarians option = eliminates delay from barbarians (but allows faster expansion of civs and so may speed the onset of non-barbarian-related delays)
To reduce the eventual delay between turns, play on a smaller map. To delay the onset of turn delays, reduce the number of civs in the game, but not below the recommended number unless you are playing with the No Barbarians option. If you find the delay between turns to be a problem in the early game then you should definitely play on a smaller map, and should turn off Barbarian World or Raging Barbarians if you have them on.
You may also see a slight performance increase (over the course of the game) by decreasing the amount of land on your map. Every additional tile of land is another place the AI can stick a cottage, and that commerce translates to many additional units it can support. Most mapscripts have a Sea Level option (or the equivalent) that controls how much water there is on the map; more water means less land. Just note that the AI can't handle dispersed maps, like Islands, Archipelago, or even Continents, because of its current refusal to build and use naval units.