I always found it funny how people compared Civ 4 to Civ 5, not because they were comparing the two games, but because they really weren't. Rather than comparing Civ 5 Vanilla to Civ 4 Vanilla, or Civ 5 GnK to Civ 4 Warlords, many took Civ 4 in it's most advanced state (BtS and patches) and compared it to Civ 5 from it's infancy. It really doesn't make much sense to compare a mansion to a construction site, or a house with ongoing renovations.
Civ 4, and it's expansions, were very good, and as gps stated, for those four Civ games the developers looked back on the previous games, and added improvements, without any radical changes. While that is a good approach there is only so much you can add before you're out of ideas for the current structure. At that point it is a much better idea to take the series in a new direction, and that's what Civ 5 did, and it's worked out amazingly well.
There are many who think that Civ 5 should have just stuck to the formula of the previous games, but would anyone really have been happy if the developers just gave us a shot-for-shot remake of Civ 4 BtS, just with a box that had a '5' instead of a '4'? I know I would have been very upset.