Building a Computational Server

[biochemist]

I work in an academic biochemistry lab and have been charged with building a new computational server. We are interested in a machine that will serve two major purposes. First will be data processing, which involves opening 5-10 large images (20+ MB) at a time, so hard disk access speed can be important. Also, large disc capacity can be useful. The other purpose of the machine will be structure refinements, which are basically number crunching exercises. While the programs we use do not take advantage of multiple processors, a dual processor machine allows us to run 2 jobs at once. In addition, I would like to set up a RAID array so that a hard disk crash will not result in any loss of data. I would gladly appreciate any recommendations you could offer. We are looking to spend about $3-5,000. I feel comfortable installing hardware but am not familiar with purchasing cases, power supplies, fans, cases. Additionally, I have never installed or set up a RAID array, so I would gladly welcome any assistance there as well. Thanks in advance.

 

[DaveSimmons]

Windows or Linux OS?

Have you considered just getting a pre-built dual xeon box from Dell?

RAID-1 or RAID-5? If your total data needs will fit in under 250 GB RAID-1 is probably easier since you can just pair up 2 IDE or SATA drives. Modern 7200 RPM IDE drives can read a 20 MB image in half a second so they should be fast enough (the bottleneck will be CPU time for the image processing).

 

[biochemist]

We use Linux OS.

Prebuilts from Dell and similar sites seem to cost about $2k more than the system components alone. I am confident in my ability to physically assemble the system, I just want someone with more experience to assure that all the components are compatible.

I was probably thinking RAID-1. Is it possible to array more than 2 SATA drives? About 200 GB would be ok for now, but expandibility will certainly extended the lifetime of the system.

Thanks for all assistance.

 

[zephyrprime]

Don't just get IDE drives. You want the reliability of SCCI or the Maxtor Proline IDE or the Raptor or something like that. Even with RAID 1.

 

[biochemist]

Yeah, I am starting to get that idea from your suggestions and others elsewhere. I am actually starting to lean towards Opteron chips. Anybody have any experience with Opterons/Linux (RedHat) and SCSI RAIDS?

 

[mmnatas]

If you're going to need a lot of space and want good speed, I'd also recommend SCSI drives. If you use RAID 5, you can pretty painlessly increase your disk space as you need it; if you get a good SCSI controller, you can put a decent amount of memory onboard which will further improve performance, especially since RAID 5 isn't quite as fast as RAID 1.

 

[stultus]

Originally posted by: biochemist
structure refinements

X-ray or NMR? What lab are you with?

Does it really make sense to spend thousands on these super servers anymore? Seems like you could just create a farm with cheaper off-the-shelf consumer stuff and beat processing times. Sure, it's not as cool as being able to say "yeah. we have a dual xeon with raid" blah blah. Actually, I don't know the answer to this, and was wondering if anyone had done a cost/performance evaluation. You can get cheap network drives for data backup and bypass the raid thing if you went with a farm...

 

[Monoman]

Originally posted by: stultus

Originally posted by: biochemist structure refinements

X-ray or NMR? What lab are you with? Does it really make sense to spend thousands on these super servers anymore? Seems like you could just create a farm with cheaper off-the-shelf consumer stuff and beat processing times. Sure, it's not as cool as being able to say "yeah. we have a dual xeon with raid" blah blah. Actually, I don't know the answer to this, and was wondering if anyone had done a cost/performance evaluation. You can get cheap network drives for data backup and bypass the raid thing if you went with a farm...

absolutely.. do you know the potential power a beowulf cluster in redhat with $250 a piece Athalon 1Ghz machines?

I agree that very intensive data crunching is better suiter for a farm than one solo box. Red Hat 9 clusters fairly easy,

good luck!

 

[drag]

Beowolf requires that you program your computer so that the computation gets divided between everything... If your apps were designed to run on a single image computer you'd have to rewrite them to run on a beowolf.

I am pretty sure....


However if your apps are very multithreading, were they start lots of different types of threads/proccesses, you can look into a OpenMosix cluster. Openmosix is a kernel modification and some user space stuff so that threads get migrated from one computer to another. One machine will be the "master", however you can start the app anywere and it will move eventually to a machine with the most resources aviable to proccess it. OF course each thread will only go as fast as your fastest computer, but multitasking will be made easier and faster depending the proccess type and other variables like the speed of the network....

Think openmosix like a gigantic SMP machine. The main advantage of a openmosix cluster is that apps don't have to be rewritten to take advantage of it.

Plus the nice thing about linux clusters is that a cluster can be BOTH openmosix and beowolf.


Openmosix is pretty innocuous to set up. It realy doesn't affect how the machine operates other then the migrating of the proccesses... The main issue that I am aware of is that if your using a program that (for instance) runs on Althon XP using SSE instructions and it moves to a older PC that doesn't support SSE instructions then it will crash the program. So any other sort of optimizations and instructions need to be present and supported on all the machines in the cluster.


Maybe you can add a bit of planning in your purchase, a upgrade path. For you, I'd say get one Opteron SMP machine. If it proves a bit over budget you can get a one proccessor in and add a second one later. Set up a nice decent RAID 5 array, and a couple DVD burners for backups.

Then maybe work on getting your apps multithreading support.

Later when it comes to upgrading the system begin purchasing Opteron-based workstations for people to use in the lab. Hook them up to a Gigabyte ethernet switch to the main SMP server. Then you set something like a OpenMosix cluster were the threads and individual proccesses can move from one machine to another very quickly.

Then you can use your expensive RAID stuff on one or two machines, have the benifit of highspeed I/O on your server, with the proccessing power aviable from the other computers that get left on in your labs.

Something to think about. OpenMosix, for more details and corrections on my ignorance-based misinformation


For your main machine rightnow I think maybe unless your sure of what your doing check out the prices on name brand stuff like Dell and the sort. Not as sexy as building it yourself, but you know it will work and you have a place for replacement parts always avaible 24/7 if something breaks.


edit:

Guess Dell doesn't have a opteron system, does any mainstream PC reseller have them?

 

[drag]

Here I went to colfax-intl.com. I don't know anything about them or if they are a good company or not, but I thought it may give a good Idea what you are looking in terms of $$.

I went to "Customize your AMD Dual Opteron High End Pedestal Server below"

Thunder K8S Pro (S2882) SMP motherboard.
2 Opteron 248 DP proccessors.
4 banks of 1024Meg DDR ram.
Chenbro SR107 Case.
550 watt power supply.
Plextor CD-R/RW drive.
4x Western Digital 36gig 10,000RPM SATA drives.
SATA hotswappable cage.
4-port Escalade 8506-4LP 3ware SATA raid controller.

No nic(only offered gigabyte fiber in this model)/keyboard/mouse/monitor/floppy drive

Upgradeable with a second RAID of SCSI or SATA drives. (need extra controller or a 8-port controller of your favorite SCSI or SATA flavor) And mobo can support up to 16GB of ram on 8 banks.

With Redhat 9.0 pre-installed. (SuSE/Redhat enterprise editions offered, a bit more expesive, but you get many benifits including unlimited (i think) tech support for a year, Phone and webbased. Also upgrade support.)

$6042

I don't know if this is super expesive or cheap or whatever. For pricing your own system from peices check out Newegg.com, they have good prices and always been good to me.

 

[biochemist]

I appreciate all the great help so far.

As far as the applications we are running, they are single threaded and normally each person in the lab is only running one refinement at a time. (BTW, we are an xray crystallography lab).

I am not a computer programmer and will/can not re-write the code to be multi-threaded. Therefore, we are looking at the fastest processors we can buy. We would put a single opteron computer at everyone's bench except for the space/noise/heat and nuisance.

I found a great website last night after 2 weeks of looking that benchmarked the software which we us:
http://asdp.bnl.gov/asda/LSD/Benchmarks/benchmark_cns.html
This is our bread-and-butter program and often times routines can take 10's of hours to a week, therefore, the apparent 50% improvement in speed on the 64-bit processors (Itanium and Opteron) over the Xeon's would be tremendously beneficial.
Additionally, it is comforting to know that another lab of similar ilk has gotten these programs to run under linux with a 64-bit processor, which was a worry of mine.

The system above seems dead-on our needs, and $6k is within our budget probably. Through this and other forums however, I have been strongly adviced to get a SCSI array rather than SATA due to its maturity and proven reliability. Any more thoughts?

I guess I am at heart a scientist, so now that I "think" I have decided on a processor (Opteron 248), the next question is motherboards. I did extensive research on the Intel Xeon Chipsets, thinking I would buy/build a Xeon, but perhaps someone here can help shorten that process for the Opteron mobo's and give me a quick summary, if they don't mind.

Thanks

 

[drag]

SCSI is indeed very superior.

However if you get SATA, in a RAID 5 configuration (and be sure to get a couple extra drives for backup when the model you choose supply begins dry up) you can begin to approach the reliablity/speed of SCSI. When one of the drives fails you can pop a new one in a hotswappable drive setup and the files will replicate themselves with absolutely no downtime. However if 2 drives fail at the same time everything would be lost.

Then if it is very important to keep things running get SCSI for best. The next step above a RAID 5 SCSI setup is to get 2 Raid 5 arrays and mirror them, keeping a identicle image on both arrrays.

I don't know what sort of output you get from your program, but if it can fit on a single CDROM and/or DVD drive that will be ideal. Maybe a tape drive or even a extra IDE drive. Then put your information on it and employ a off-site archiver to store it. Say you do a complete backup on the weekends. Then if the drive array got run over by a truck, failed miserably, building burnt down or got blown away. Worst case is that you'd loose a week of work.


I figured a SATA array would be OK since your on a budget. Proper backups would be more important then getting a SCSI array.The money would then go to get faster proccessors.

But that's for you to decide. Is the major price difference between say a dual/single 200 to 248 opteron worth the amount of performance increase VS the spending the money instead on a high output/high avaible SCSI Raid 5 array?

Or could the difference in CPU power knock a day off of a week long proccessing of a big job?

Could be or not. Performance difference may be very small compared to the cost of getting down time due to a slightly inferior Drive setup.... OR you may find that with the more powerfull CPU that isn't the bottleneck in performance anymore and the I/O becomes the bottle neck since the SATA drives can't feed the information fast enough to the dual CPU's. I have no idea.


Or maybe the money would be better spent on proper support from a bigname OEM computer maker so that you don't get fired when the computer you built catches on fire. Instead the OEM gets sued for the computer catching on fire?

I don't have much experiance in this stuff, some people swear by good corporate tech support, others swear at it.

 

[biochemist]

I would definitely be interested in having someone else put together the system, as it keeps getting increasingly complicated. I just don't know of anyone making Dual Opteron Servers, until you mentioned colfax-intl.com. I would gladly let someone else decide if the components match up, make sure everything is loaded, such as RAID and linux, etc... The purpose of this is for the lab to get a fast computer, not for me to build a machine. I just went to Dell first and was amazed at the 2k premium that charged over buying the parts from someplace like newegg.

We have 2 categories of files. First are the data images, which are about 10-25 MB a piece and around 100-200 images in a data set, which we currently back up via a DVD-burner and will continue to do. Once the data is reduced, the files become much smaller, with program outputs on the order of 10's of MB at most and we back those directories up on CD's using a SCSI Plextor that has burnt hundreds, if not thousands, of CD's without a single error. The reason for looking into RAID solutions is 2 fold. First, we have not had a drive crash in over 5 years (knock on wood), so we are due. Secondly, we do not have a single adminstrator who backs up everyones files, rather its a do it yourself job. As with most things procrasintation sets in and easily months go by before things are backed-up.

The premium here is on processor speed. The RAID/SCSI/ATA is secondary, as for the most part, none will provide a bottleneck. Also, with each data set being 2-5 GB's a 36 gig drive can get eaten up in a hurry, so I definitely think that if SATA RAIDS are realible, they would suit our needs best.

 

[jose]

Check out RedHat Enterprise 3.0 ES you can get it for $176, It has improved performance over RH9.0 etc.

It also supports 6 diff platforms. I'd take a look at a 7505 based mobo w/ 4 gigs of memory.
Mylex has some good 64bit 133mhz raid 5 controllers. Pair them w/ 5 scsi drives in raid 5 .. ie 15K Maxtor Atlas drives.

Regards,
Jose

 

[biochemist]

I have further researched this system and have just a few more questions. Thanks for all of your assistance so far.

First off, I am leaning (with my boss's nudging) of going with SATA drives. My question is about RAID-5. What happens if there is a power failure or other unscheduled reboot? Because I work in a medical center, attached to the hospital, there is an on-site coal power station. We have only had power flicker once in 4+ years I have been here, but it did cause an unscheduled reboot of our computers. Should I look into a UPS if I have a RAID?
Also, I am likely to go with a case that does not have hot-swap bays. If a drive fails, is it ok to shut the system down to replace the bad disc with a new one?

We will continue to back up files as we do now (multiple, redundant DVD-R's). But being paranoid, and not having a disc fail in 4 years, we are due. This is why I was thinking about a RAID solution. Should I be looking at other solutions?

Thanks again.

 

[drag]

UPS is fine, It's cheap insurance and protects expesive equipment from power fluctuations. The only bad thing is that every couple years or so you have to replace the batteries, and that can get expesive or some models will be hard to find.

I don't know about the raid stuff. Probably be OK.

 

[Swanny]

I'd go with an Opteron system with some Raptors in RAID 0.

 

[Matthias99]

A UPS is always a good idea for any sort of critical server. Of course, all that allows is basically for a clean shutdown if there is any sort of extended power failure, but it will keep you up and running smoothly through a flicker. It also smooths out any irregularities in your power lines, which can cause instability problems, or power supply failure over time.

With most hardware RAID solutions, you should be more or less OK if the power does drop. You may have an inconsistency on the block(s) that were currently being written when the power failed (which would cause corruption in whatever file was being written, but at least it would be noticed!), but the rest of the data will be alright. If only a single disk has an error, it *should* be able to rebuild properly. It's a very good idea to do a live test of things like this (unplug a drive while running, unplug the system while running and see what's there when you power up) once you get it set up -- if it's not truly fault-tolerant, you want to know about it *before* something bad happens.

If you don't have hot-swap bays, your only choice is to shut down the system to replace a bad disk. Whether or not this is an acceptable solution is up to you -- it takes significantly longer to open the case up and replace a bad disk than to just swap a drive from a bay. If you're Amazon.com, that's bad. Your lab, however, may be able to tolerate an hour of downtime for the significant price savings.

IMHO (as someone who works in the computer storage business), SCSI disks themselves are not inherently more fault-tolerant than IDE/SATA drives. That said, high-end SCSI disks are certainly of better quality than cheap IDE ones, but I doubt you're going to see a significant change in failures going between IBM 10KRPM 74GB SCSI disks and, say, WD Raptor SATA drives. If you're concerned about drive failure, use RAID1, RAID0+1, or RAID5 to provide redundancy. Many controllers can 3- or 4-way mirror data in RAID1 as well, but if your backups are pretty regular, that's probably overkill. If you want to make backups easier and more regular, you might want to look into NAS (Network Attached Storage), and some automated backup software. Might be out of your budget, though.

I understand your position, having worked in a neuroscience lab in college that did a lot of number-crunching (mostly DSP stuff on 32- and 64-channel high-resolution intracellular recordings, with dataset sizes up in the 1-8 GB range). For bizarre reasons that I don't entirely understand, scientific computing packages seem to have practically no support for multithreading OR distributed computing, even though they're the sort of apps that could most make use of them! If an x86-64 (Opteron) version of your software is available, that's probably your best bet. Itanium solutions are prohibitively expensive in my experience, and you'll get a lot more speed than Xeons or Athlon MPs that way if you have a 64-bit app.

 

[HokieESM]

You've gotten some good advice from some people who know WAY more than I about the hardware/computing part. But being a grad student in computational mechanics, I have picked up a few things about number crunching... specifically about parallelization/multiple runs. Definitely check the memory/CPU usage for each run before deciding to go with a single machine with dual CPUs. Some codes are memory hogs--and the shared memory can be a problem. Also, if your code is extraordinarily memory bandwidth dependent, the dual CPU can pose a problem.... I found that it was actually easier for me (for my bandwidth-hog code) to build two single CPU machines as opposed to a dual CPU machine (there is some redundancy--HDDs, optical drive.... but the cheaper parts can offset this... not to mention, it can be nice to have a separate SYSTEM to do a backup).

Anyways... good luck. Glad to hear you're getting your own resources from your university... as opposed to my situation, where I now have four computers running in my apartment (two crunching numbers continuously).

 

[HokieESM]

Originally posted by: Matthias99
For bizarre reasons that I don't entirely understand, scientific computing packages seem to have practically no support for multithreading OR distributed computing, even though they're the sort of apps that could most make use of them! If an x86-64 (Opteron) version of your software is available, that's probably your best bet. Itanium solutions are prohibitively expensive in my experience, and you'll get a lot more speed than Xeons or Athlon MPs that way if you have a 64-bit app.

Matthias... it all depends. Most mechanics-based solutions (or full-complement problems) are inherently un-parallelizable. Every node/point effects every other node/point. Sadly, this is why a cluster isn't of much use for some scientific applications--you're pretty much limited to the speed of a single node. My code only sees a 20% gain using a second processor and only an additional 1% for using a 3rd. Now some molecular dynamics situations can iteratively solve and use multiple threads or a cluster-type environment... or any simulation where one can completely separate portions of the solution from another.

This is also why we're not seeing an EXPLOSION in scientific computing right now--if we could just make bigger and bigger clusters to solve individual problems, then we would be seeing massive massive growth (rather than just Moore's Law)... computers are getting individually faster AND cheaper. But big clusters DO allow simultaneous numerical experiments..... I can run 20 runs on 20 nodes of VT's new System X and not affect anyone else!

 

[biochemist]

Thanks. I feel more confident about the RAID solution. However, I wasn't planning on using Raptor quality drives, rather the WD SATA 7.2k rpm drives in the 160-250 gb range. The down time of a few hours to replace a bad drive (because no hot-swap bay) is perfectly acceptable. What kind of UPS should I be looking for, what characteristics that is? Also, I read a little bit this afternoon and some of them will shut down your system for you. How is this acheived and is it easy/possible under Linux? This sounds like an ideal solution, obviously.

As far as scientific software not being multi-threaded... I can't do much about that right now. I can only live with it and spend the outrageous bucks for the fastest single processors I can find. Too bad, I'd love to make a farm of 24 Athlon 1GHz and let them all hum away, but alas. The 2 gigs of RAM, 1 per processor, will be more than sufficient. Other machines we run (which aren't "quite" as fast) don't come close to using their 1 gig RAM.

 

[biochemist]

Opps. Hit enter twice... Sorry.

Thanks for the advice everyone.

 

[Cerb]

Even I wouldn't have expected such a major gap between the Itanium/Opteron and other systems on that benchmarking page. For the price of that one Itanium box, you could make 3 or 4 Opterons, so if the budget rules you, Opteron it is. Also note that you can now get Opterons 20% faster than the 1.8GHz used for that benches you found.
Also, as has been noted already, you want a UPS.

SCSI/IDA/SATA:
If you're going to have reads, then a few seconds or minutes before another write (as opposed to constantly doing both), the difference between IDE and SCSI is basically that the plain IDE drives will be a bit more prone to failure. Without a fair bit of drive access, there isn't much difference in performance. Use hardware RAID 1 or RAID 5 and you're set there. Get more memory than you think you need, and that will also help with the drives, since it will be longer before they MUST be accessed (and you can painlessly get to 4GB RAM, up to 6 or 8 depending on the mobo).

If you want the physical part built for you, you might also want to try http://monarchcomputer.com

 

[biochemist]

Haha. Great recommendations. I posted this to another forum about 30 minutes ago.

Ok, so I am getting down to the actual ordering phase now. First up, mobo and such.

I am going to order a AMD Combo from monarchcomputers.com. They will assemble and test the mobo, processors, heatsinks/fans, and memory. For NO charge. As this is the stage I am worried about assemblying and messing up and monarch's prices are very similar, if just slightly higher than newegg.com, I think I will take this option.

Combo -
Tyan Thunder K8S Pro (S2882UG3NR)
2 x Opteron 248
2 x Thermal Grease
2 x 1gb (2 peices of 512) Corsair memory

Now one remaining question, the Corsair PC3200 memory is only 2 dollars more expensive than the PC2700. This seems odd for leading edge technology, normally you pay a premium. Is there a quality/reliability issue that I am missing with the PC3200 RAM? Or is this just a good deal and I should take advantage of it? Thanks.

Price with PC3200 RAM - $3,121.

Any comments are welcome, such as, "I ordered from Monarch and the mobo burnt my building down." Or similar.

 

[Cerb]

I just went and looked, and if that's the same Corsair I'm seeing, it's not ECC, while the others are.
You might want to look under their servers. A bit more expensive than the combos, but with nice cases (especially the Chenbro 3U ones) and redundant PSUs. Also, still not much more expensive than Newegg (particularly given that you can get redundant PSUs).