Windows 7 releases memory as needed. Zelig already talked about this a long time ago: You're not using the memory, so why not let the OS use it for something. And when you do need it, its released to you. Windows 7 improved a lot over Vista in that sense. Windows 7 could be using almost 2 GB of RAM when my pc is idle, but as soon as I run an app that needs 7.5GB, bam, 1.5GB of the memory is released to me, on top of the 6GB that were already idle.
In other words, your reason for going back to XP is bull. If it seems that programs slow down, check other factors too.