I disagree that the processor is more likely to be the cause of slowness from my own personal experience.
Everything influence the performances in a game like CIV4... I can tell the difference because between work and home computers (all my machines mount CIV by default) I have tested CIV on at least 6 different configurations (4 laptops and 2 desktop)
Graphic card mostly influences the smoothness of play during your turn.
CPU influence mostly the time between turns
RAM memory influences both
1. Graphic card
a better graphic card with large dedicated memory makes the difference when you run CIV at higer resolution.
Obviously with same resolution and graphical detail a better graphic card will give you a smoother experience.
With larger number of units in play (can happen at several points in the game) you'll need a better graphic card to handle all the animations (especially when zooming out).
With a larger map, the dedicated memory buil-in the card will ensure smoother scrolling (more buffering around the visible area of the map).
Some video cards have problems with CIV.
On my travelling work laptop (an horrible Lenovo X200s) I get "holes" in the UI!
very annoying but working. On my Vaio, it's a smooth as silk (even when I play on a 1920x1080 resolution on a HD TV).
If you have one of those cards using the computer memory as graphic memory (typical in cheaper laptops) you are gone and you'll have a hellish experience.
2. Computer memory
I noticed that 1GB of RAM is the very minimum you must have to have a decent game play, especially in later ages when you get a lot of units, cities, and civs in play.
2GB of RAM are a much better deal if you want to play long and prosper.

Over this thrashold there is no big gain in putting in more RAM
3. CPU
This is a huge difference especially in the time to be spent between turns, but also influence the smoothness of play during the turn (but the graphic card have a larger influence in that).
Here again there is a thrashold.
under the trashold you can be sure you will have huge problems.
over the trashold you can rest a bit safer.
For my experience moving from 1core to dual core makes the trick.
Any dual core seems enough runs smooth.
In conclusion if you want to play well (or better have a smooth experience with limited frustration of waiting)
you need to have all three components over a certain trashold.
For example there is no point of having a super dupper CPU when you have only half a GB of ram.