I thought I'd post my own thoughts here.
Just finishing up an upgrade from a seriously hacked up copy of VB 2.2.x. Our stats:
* 35,000 members
* 80 forums
* 2M posts
Backend database server is dual Athlon, 3GB RAM, and the same for the frontend web server.
It took some time (obviously) to convert the posts, etc. We had over 4GB of attachments in a table (lol) and now those are off in the filesystem with VB3. The search engine rebuild took about 3 days.
But, the good news is that it was pretty much a flawless migration. With the exception of my users *****ing because all of the old hacks aren't in yet. I took advantage of the downtime (rebuilding indexes, etc) and started incorporating all of my custom hacks and applications into VB3. By the time I went live, much was already done.
I have noticed some speed issues, but I haven't isolated it to anything yet, so it could be the network today.