32 people tends to be very disorganised and, depending on the map, very spammy. admittedly, bigger maps would reduce the problem of spam, but with the way in which tf2 spawns work, this could well reduce the effectiveness of all but scouts, medics, soldiers, demos (i.e. the faster classes) to close to zero. already, heavies spend half their lives walking to the front line anyway.
but my main problem with 32-man is the lack of communication. talking to your soldier when you're a medic is absolutely essential... you have to be able to say stuff like "I'm being gangbanged by two scouts" or "i have to go find some health" or "kritz/uber ready to go". on a 32-man server it becomes nigh-on impossible to communicate in that kinda way - a 16 man team is very loud and inherently quite uncontrollable. it is very difficult to make sure that you have good class balance, and arranging a defense or offense becomes a matter of luck and chance rather than skill and planning. for me, 32-man takes away the essence of team fortress - the "team".