I no longer have sympathy for people who complain about people taking too long. It used to drive me crazy too, but I've had a few games lately where people constantly complain into the mic about me taking too long when I am busy with a lot of units to move. I am so sick of hearing high school kids on Xbox Live crying into their mic because they have to wait for someone else to finish their turn.
Just because you don't have any units to keep track of doesn't mean that goes for everyone. I rush warriors at the beginning of the game so I have them moving around and fighting barbarians and collecting the goody huts while everyone else just pushes B and waits for the turn to end and their city to grow. I rarely use up the entire timer, but I am so sick of people whining if I use a quarter or half of the timer. I usually have three warriors running around when most people have one, so obviously I will have more to do than the rest of you, and that's not my fault.
I'm sorry I can't make the combat go any faster! If I could skip the combat animations, I would, but I can't, so I have to wait for them, and then so do you! When people take a while, it does not mean they're just sitting there staring at the screen debating if they should end their turn yet or not. They usually have things going on, and just because you don't have anything going on yet doesn't mean they don't. So stop being so impatient and if it bothers you that much, rush a warrior yourself, read a book, or watch TV for the 10-15 seconds you have to wait between turns.