The question to ask is what sort of performance drop you suffer when populating all 4 DIMM slots. It varies according to what sort of Athlon 64 CPU you're using, but here's the short and sweet:
Using 4 DIMMs of PC3200 (double-sided)
Winchester and older cores:
DDR333, 2T command rate
Venice and newer cores:
DDR400, 2T command rate
* if using single sided RAM with a Venice core or newer then all 4 DIMMs may be used without a problem (but no one uses single-sided RAM)
---
edit
---
It should go without saying, but using only 2 DIMM slots will not hinder performance in any way. Always try and stick to only using two sticks of RAM whenever possible.