I actually had a very good experience running XP 64 for a short time. I downloaded the free trial for my last PC build (because I didn't have a spare Windows OS at the time) and it seemed to work very good. It just seemed to be a very fast and efficient OS.
In fact, I was so impressed with XP64 that I wrestled with the idea of purchasing Vista over XP64 for quite some time because I liked the OS that much.
Driver-wise, I found everything I needed for XP64 (even drivers for an obscure Tascam audio interface/controller surface I use for recording). And everything I found that had a driver for Vista64 had one for XP64 too (there may be exceptions to this, but I found none for what I needed).
In the end I went with Vista mostly because of the enhanced security, DX10, and better "future-proofness". They can keep the eye candy (I actually prefer the look and feel of XP still).
All that aside though, Vista has been working equally as well for me as the XP64 trial did so overall I'm pretty happy with the decision to get Vista.
C'mon in... the water's fine!