Setting up a home server

[EtherealDragon]

Hello all,

I have purchased a server for my home, and I know what I would like to do, I am just not sure how to go about getting it accomplished.

The SIMPLE plan is to have this server set up as a NAS (running ZFS), and also running Plex Media Server. Hardware allowing, perhaps other functions in the future as well.

What comes with the server:

• Supermicro SC826TQ-R800LPB Chassis (12 hot swap 3.5" drives) with rails
• Supermicro X8DTN+ Motherboard
• 2x Intel Xeon E5620 2.4ghz 12m Cache CPU's
• SIMLP-3+ IPMI Remote Access Card
• 6x 4Gig PC3-10600R Memory (24 Gig) (no info on brand or model)
• 2x Supermicro PWS-801-1R 800W Power Supplies

What I already have:

• 5x WD WD40EFRX 4TB SATA HDD's
• 2x OCZ Agility 3 60 Gig SATA SSD's
• 4-8 Seagate 7200.11 ST3500320AS 500 Gig SATA HDD's (Gave some away for builds, not sure how many left and still good)
• Simple telco 2 post rack cut to about 3.5 feet tall from a previous home network install (will likely be used in attic or garage for small equipment there, not for the server)

What I need:

• 4 post server rack - eventually I will want something with locking doors, but that isn't a priority now, as it will likely sit on a table in the basement until I get a proper rack.

The BASE OS I am leaning towards FreeBSD, as both ZFS and Plex Media Server will run on it. Now, I have LOTS of reading and research to do, because my experience with OS's other than Windows is praciclly non-existant. We do have linux (I think) where we run CLI searches, make copies, of router configs and IOS files. Sometimes have to create configs with vi and change permissions, but all pretty basic stuff. So, ZFS won't be virtualized, but Plex Media Server (PMS) will be running on a virtual machine.

Can anyone suggest good reading material on FreeBSD and virtual machines?? Any input would be most appreciated.

My plan is to first make sure it is all clean and not choked with dust. I will then remove the CPU's and put some fresh thermal compound on them. Once that is all done, I plan to test the memory, but I am not sure with what. I have Memtest86, but is that good for server ECC memory too?? Any suggestions?

From here it will be installing the main OS, FreeBSD, and seeing where I can go from there.

 

[mfenn]

I think that you'll find that starting with a base FreeBSD OS will be harder road than going with NAS distribution such as FreeNAS (based on FreeBSD with Plex support built in). If you just want to get up and running, then that's probably your best bet.

If you want to have something to tinker with, I would recommend going with Linux rather than FreeBSD. You'll find that a lot more resources and software are available for Linux. Running something like Ubuntu LTS will give you access to about a zillion pre-packaged pieces of software and more online documentation than you can shake a stick at. And don't worry, ZFS does run on Linux.

Also, I don't see the point in running Plex in a VM unless you have to. You'll have to give it access to all your media anyway, which kind of defeats the purpose of running a VM.

As for diagnosing the system, a great advantage of server hardware is the system event log. You can access this through the IPMI card, and it'll tell you anything that goes wrong with the system. It's especially nice for hunting down memory errors because you'll see a log entries like:

Threshold of correctable ECC errors reached: DIMM A1
Uncorrectable ECC error: DIMM B2

 

[EtherealDragon]

Thank you very much for the reply. I had actually had several people suggest running with Ubuntu, so I will put that at the top of the list and start research. Thank you as well for the ZFS on Linux link.

As for running Plex, I haven't run any VM's yet, but I was thinking that it would be best to run any software inside of a VM than on the bare metal host.

For the system event log, are you saying that I don't need to worry about checking the memory for stability and just keep an eye on the event log? Furthermore, if a VM host has a memory error, would that error show up ONLY in the event log of that particular VM, or would there also be a corresponding error in the main OS (Sorry, not sure of the term to use for the main non VM OS)?

 

[XavierMace]

Keep in mind that your drive backplane has 12x 3Gb SAS ports but your motherboard only has 6x 3Gb SATA ports. I'd recommend picking up an M1015 on ebay (http://www.ebay.com/itm/311289289402?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT) with a pair of SAS Forward breakout cables (http://www.monoprice.com/Product?c_id=102&cp_id=10254&cs_id=1025406&p_id=8186&seq=1&format=2) and flash the M1015 to IT mode (http://www.servethehome.com/ibm-serveraid-m1015-part-4/). That will give you connections for 8 drives.

Since you're not running a bare metal hypervisor, I'm with mfenn and don't see the point to running Plex in a VM.

 

[sao123]

What comes with the server:

• Supermicro SC826TQ-R800LPB Chassis (12 hot swap 3.5" drives) with rails
• Supermicro X8DTN+ Motherboard
• 2x Intel Xeon E5620 2.4ghz 12m Cache CPU's
• SIMLP-3+ IPMI Remote Access Card
• 6x 4Gig PC3-10600R Memory (24 Gig) (no info on brand or model)
• 2x Supermicro PWS-801-1R 800W Power Supplies

just out of curiosity, are you buying this new or used? What did it cost?

 

[EtherealDragon]

I purchased this used. Cost including shipping was just over $340. I feel that it was a pretty good deal. I have been looking to BUILD a server like this in a Supermicro 4U chassis, but the costs were just too high. I was ABOUT to email someone on Craigslist about a 4U chassis for $450, but then started looking at 2U chassis all over. After mulling it over for about a month I pulled the trigger.

This is the one I got with the dual Xeon E5620:
http://www.ebay.com/itm/291363497930?_trksid=p2060778.m2749.l2649&ssPageName=STRK:MEBIDX:IT


They also have one with same specs aside from dual Xeon L5520 for $20 less:
http://www.ebay.com/itm/371243279654?_trksid=p2060778.m1438.l2649&ssPageName=STRK:MEBIDX:IT


They have maybe a half dozen more options with different CPU's and different amounts of RAM in the same chassis, some with a few drives, some with none, but the prices climb quickly.

It's only 08:15, and I have already checked the front porch 4 times this morning... Like a kid on Christmas morning waiting for FedEx to pull up, lol.

 

[EtherealDragon]

XavierMace,

Thank you for the link to the IBM M1015 card. I will be purchasing one of those today actually. I forgot about no card being in the chassis and only half of the drive bays being connected.

Thank you also for the link to flashing to IT mode, bookmarked.

[EDIT]SIDE Question: Would it be advisable to get TWO of the M1015 cards to have ALL of the main storage drives connected to the cards rather than the motherboard? I am not sure if that would offload of the processing to the cards rather than the mobo.[/EDIT]


[EDIT2]YAY!! Santa (a.k.a the FedEx guy) just dropped off the server. Ran a quick test to make sure both PSU's and the fans work, and they are all good, powers up from either PSU without issue. I am actually surprised, it IS noisy, but not QUITE as noisy as I was expecting. This is going to basically be installed in the basement in a long utility closet which also holds the furnace. The volume from the server is basically more or less the same, MAYBE a bit quieter than the furnace. Time will tell. Maybe I will just turn it on, then tell my wife it is the furnace running 24x7. Lol.

I also just bought a bag of screws for the drive sleds for a couple bucks, and a front locking bezel for $22 shipped. I don't care if it locks now, but more concerned about dust and pet hair. The bezel LOOKS to have a fine mesh grill that I would imagine would catch much of it, if not, I can fashion a filter to fit in there and keep it out of the server.

I opened it up, and WOW... I was expecting DUST CITY, but this this is IMMACULATE. Not even the FANS have a SPEC of dust on them. It was either in a really nice filtered environment, or someone took MUCH care to clean thsis like crazy. There are only a few scratches on the top and bottom where other rack components would have been slid in/out right above/below it. I am very happy with the purchase so far, and it isn't doing anything but sitting on a table behind me.

I will down the road get rails for it too, but that is a cost I don't need to dive into until I get a proper 4 post rack for it. [/EDIT2]

 

[dave_the_nerd]

Another vote for Ubuntu Server - ZFS-On-Linux works fine, if you really want to use ZFS for storage. Also a lot easier to configure, better tutorials, more open community willing to help n00bs, etc.

I put Plex and a few other things in VMs just so I can reboot their machines or tinker with them without necessarily bringing down the system. It's more about compartmentalization than about pure performance or efficiency.

[EDIT]SIDE Question: Would it be advisable to get TWO of the M1015 cards to have ALL of the main storage drives connected to the cards rather than the motherboard? I am not sure if that would offload of the processing to the cards rather than the mobo.[/EDIT]

Wouldn't matter - with ZFS, the CPU is handling all of the processing anyway - and you want it too. The SATA/SAS controllers should be working in "dummy" or JBOD mode and letting the file system on the CPU do the hard work.

 

[EtherealDragon]

Thank you Dave for the replay. That was my same thought with having Plex in a VM. Still not quite sure how I will handle it in the end, we will see.

I also ordered the M1015 card just now so that I can have all 5 drives plus boot drives online. I am not sure how I will handle the cables though. I know that the straight connectors on the SATA side will FIT, but it is a tight bend, not sure if that is good for it. Also, will that card in a 2U chassis allow for the straight SFF-8087 connector, or would that need a right angle as well?

I may also look into having the stock fans replaced with PWN 80x38 or 80x25 mm fans. We will see.

 

[Carson Dyle]

Do your _really_ want a rackmount server for just a dozen hard drives? Unless you have other rackmount equipment, or you plan on adding additional rackmount servers in the future, I'd seriously avoid going that route. A rack requires a lot of floor space for just a single server, plus there's the noise.

Also look at the LSI 9211-8i card, which is the native LSI branded equivalent of the IBM M1015. They can be had from Chinese sellers on ebay for as little as $85. Currently there's a seller with them for $100.

http://www.ebay.com/itm/281409936741

 

[dave_the_nerd]

Thank you Dave for the replay. That was my same thought with having Plex in a VM. Still not quite sure how I will handle it in the end, we will see.

I set up NFS exports from the main system (working as a file server) for the VMs to connect to. It gives me shared home directories, which is nice. But I can mount /NAS/ARRAY_1/Movies and /NAS/ARRAY_2/TV as /media/Movies, /media/TV, and so on on the PLEX server.

I have the same basic thing set up for Crashplan, which I definitely am glad I did, since seems to gum Java up and need a reboot every now and again. (Not as frequently as it used to though - both PLEX and Crashplan are way more stable running in an Ubuntu VM then they were as Jails/Plugins on FreeNAS. Holy cow.)

 

[EtherealDragon]

Yes, I am all in currently for a server rack. At this point, I already have the server sitting on a table behind me.

As for space, it isn't really a question of wasting space, as I have plenty of room for it. I have a utility space which is just under 8 feet long, 3 feet deep, and around 5.5 feet tall with which to put equipment.

With racks in general, I had a rack set up in my previous home, but it was just 40" tall 2 post 19" telco rack. (See photo below) I have both halves and I suppose I CAN fashion a homemade 4 post solution, but I would prefer NOT having to be bending over all the time like I was with the old rack (it was in a crawl space under a landing for the stairs).

This rack is going to be EVENTUALLY the place where the house wiring terminates, so there will be pretty much ALL of the networking equipment for the house. This will include a few patch panels for Ethernet and Coax, may POSSIBLY try to run a fiber from the basement to the attic, and POSSIBLY up to the garage. This isn't a HUGE priority, but may eventually do it. There will also be switches, cable & DSL modems, hopefully a UPS in the future as well. I have a 1500 Kva UPS for my desk which powers my worklaptop, and dual monitor setup should there be a power loss. This has already come in handy twice in 8 months.

 

[Red Squirrel]

Basic rule of thumb try to keep storage as separate as possible from the rest. The last thing you want is something causing an issue where you need to reboot that since you have to now reboot everything else too.

I would look at eventually building a separate VM server then that would run everything else and use the storage off the NAS. You would just need a SSD or even USB stick in VM server to run the hypervisor and then map NFS shares.

Rack is a very good idea. Makes it easier to keep things neat and tidy. You'll probably want a decent UPS too so at very least your stuff can shut down gracefully if power goes out.

 

[Carson Dyle]

Yes, I am all in currently for a server rack. At this point, I already have the server sitting on a table behind me.

As for space, it isn't really a question of wasting space, as I have plenty of room for it. I have a utility space which is just under 8 feet long, 3 feet deep, and around 5.5 feet tall with which to put equipment.

Only three feet deep? Are you sure that's enough? That's pretty shallow for most four post cabinets. I assume you have access to the rear of the space, otherwise it could be difficult mounting rails and managing cables.

This rack is going to be EVENTUALLY the place where the house wiring terminates, so there will be pretty much ALL of the networking equipment for the house. This will include a few patch panels for Ethernet and Coax, may POSSIBLY try to run a fiber from the basement to the attic, and POSSIBLY up to the garage.

I don't know what your living situation is, i.e. whether you'll ever move from that house, but if so, you might consider terminating the cabling in a more permanent place, like a nearby wallmount rack (typically just 6" or so deep) or in a modular cabling box, or even a separate two-post telco rack bolted to the floor. If/when you move, leaving a full-sized network rack isn't likely to be welcome by everyone. Nor is leaving bare cables or punched-down patch panels hanging or lying on the floor.

 

[mfenn]

Only three feet deep? Are you sure that's enough? That's pretty shallow for most four post cabinets. I assume you have access to the rear of the space, otherwise it could be difficult mounting rails and managing cables.

Agree. 36" is a really shallow 4 post. The standard is more like 42".

 

[mfenn]

As for running Plex, I haven't run any VM's yet, but I was thinking that it would be best to run any software inside of a VM than on the bare metal host.

The nice thing about UNIX (and UNIX-based systems like Linux), is that they're modular by design. You don't need to reboot to solve a problem with a particular service, just restart the service.

For the system event log, are you saying that I don't need to worry about checking the memory for stability and just keep an eye on the event log? Furthermore, if a VM host has a memory error, would that error show up ONLY in the event log of that particular VM, or would there also be a corresponding error in the main OS (Sorry, not sure of the term to use for the main non VM OS)?

Yeah, pretty much. The SEL is a firmware level construct. It doesn't know or care about VMs or anything like that, it'll see all memory errors as long as you use ECC.

 

[EtherealDragon]

Basic rule of thumb try to keep storage as separate as possible from the rest. The last thing you want is something causing an issue where you need to reboot that since you have to now reboot everything else too.

I would look at eventually building a separate VM server then that would run everything else and use the storage off the NAS. You would just need a SSD or even USB stick in VM server to run the hypervisor and then map NFS shares.

Rack is a very good idea. Makes it easier to keep things neat and tidy. You'll probably want a decent UPS too so at very least your stuff can shut down gracefully if power goes out.

Yes, I MAY look down the road to have the VM's moved off of this hardware to to another setup, but that won't be any time soon. First I have to learn the ropes of whatever OS I end up going with, most likely Ubuntu, and get everything running to my satisfaction.

I am not quite sure what you mean in the middle part. I have not read a HUGE amount on ZFS, but I remember reading somewhere about not running ZFS on a VM, as there were some issues. Perhaps it was a one off thing, or there really are issues, that I have no idea at this point.

Side note, I followed your build over on [H], nice setup. I liked the setup you have at the top of your Belkin rack for monitoring different things around the house, and would like to be able to do something similar here. That is a ways down the road though.

Only three feet deep? Are you sure that's enough? That's pretty shallow for most four post cabinets. I assume you have access to the rear of the space, otherwise it could be difficult mounting rails and managing cables.



I don't know what your living situation is, i.e. whether you'll ever move from that house, but if so, you might consider terminating the cabling in a more permanent place, like a nearby wallmount rack (typically just 6" or so deep) or in a modular cabling box, or even a separate two-post telco rack bolted to the floor. If/when you move, leaving a full-sized network rack isn't likely to be welcome by everyone. Nor is leaving bare cables or punched-down patch panels hanging or lying on the floor.

With the 3 feet, my plan was to have a rack turned 90* with one side up against the foundation, so that the front and rear are accessible. While having the rack at this point isn't much of a priority, I am still looking to see if I can find any steals out there. I am not going with a 42U rack, something much smaller, I believe 27U or thereabouts.

We just purchased the house back in June of '14, and we plan to be here for the long haul. I am not yet sure how I am going to terminate the cables, or where exactly, but I will get there. What is there for now works, it is just that I have cables hanging right now, none of the existing stuff was connected to ANYTHING. I plan to pull it all and start over, slowly. Starting with the basement and 1st floor, then working up to the 2nd, which SHOULD be a breeze.

When I left the old house, I DID take the rack, but I build a nice wooden 19" rack to mount the patch panel which all of the house wiring was terminated to. I hadn't gotten it labeled by the time we moved out, but it was all terminated and tested and working.

 

[Red Squirrel]

Yes, I MAY look down the road to have the VM's moved off of this hardware to to another setup, but that won't be any time soon. First I have to learn the ropes of whatever OS I end up going with, most likely Ubuntu, and get everything running to my satisfaction.

I am not quite sure what you mean in the middle part. I have not read a HUGE amount on ZFS, but I remember reading somewhere about not running ZFS on a VM, as there were some issues. Perhaps it was a one off thing, or there really are issues, that I have no idea at this point.

Side note, I followed your build over on [H], nice setup. I liked the setup you have at the top of your Belkin rack for monitoring different things around the house, and would like to be able to do something similar here. That is a ways down the road though.

Oh no the ZFS server would be physical, I meant VMs for everything else like Plex, torrent box, or whatever else you may have at some point.

Some people DO virtualize their storage though, but I'm not really a big fan of that idea.

 

[XavierMace]

SIDE Question: Would it be advisable to get TWO of the M1015 cards to have ALL of the main storage drives connected to the cards rather than the motherboard?

Personally I would. No potential oddities from having your pool on two different types of controllers, SAS support in case you need it later, and cleaner cabling (1 SAS breakout vs 4 SATA cables).

Regarding the bending, be as gentle as you can with the cabling and you should be fine. I've never needed a right angle connector.

 

[EtherealDragon]

I picked up an extra 4TB drive, because I plan to be running a single VDEV with RaidZ2, so ~16TB usable. I got 2 sets of SFF-8087 to SATA breakout cables, but will need to get another one. The first one I got I didn't notice was 0.5 meter, and just too short. I MEANT to get the 1.0 meter cable, but wasn't paying attention. After getting it, I didn't like the bend radius, so I ordered a pair of right angle SFF8087 connectors, but naturally, they WON'T work with the card. Oh well, the price I pay for being lazy and not checking into that. Will be selling 2x 1.0 meter right angle cables, and 2x 0.5 meter breakout cables. PM me if anyone is interested in them.


I have been reading a few things about ZIL and a drive for L2ARC. I have a pair of 60 gig OCZ Agility 3 SSD's that I was planning on using for this, but I am not sure if I should or not. I read a few places that it really isn't going to make any difference at all. These were NOT purchased for the projects, but old drives from a laptop and a desktop which just aren't in use. Will setting these up have any DECREASE in performance? The things I read said basically that investing in RAM results in better performance than SSD's for ZIL & L2ARC if the RAM isn't maxed. As I already have these on hand, I would like to use them for something so they aren't just sitting unused.

The plan is to then steal the 250 Gig SSD out of my PS3 and swap in a 2.5" 1TB drive I have to the PS3, and use the SSD for the base OS on the server. That is, if 250 Gig will be enough, or if I should use the 1 TB drive as the OS drive, I can do that too...

Thoughts?

 

[mfenn]

250GB is plenty for an OS install.

The idea behind a dedicated ZIL (ZFS Intent Log) mirror is to reduce latency on writes. Writes can be logged to the SSD very quickly and then flushed from memory to disk whenever it's convenient. That's right, in normal operation, the ZIL is write-only. It's important for a dedicated ZIL device to be mirrored, because if it fails and the server goes down uncleanly, you've lost data. The ZIL device doesn't need to be very large, 8GB is enough for even a server with 10GbE. Note that if you don't have a dedicated ZIL device, there is still a ZIL, it's just written out to the normal disks (at higher latency).

L2ARC (Level 2 Adaptive Replacement Cache) is a read cache. It's called level 2 because the level 1 cache is in main memory. It can be useful if you have a workload whose working set of random access data is larger than the system's memory. Otherwise, they don't do much. L2ARC is just a copy of data that's already in the zpool, so it doesn't need to be mirrored.

Neither of these should decrease performance unless your Agility drives are at the end of their lives and have problems with latency. However, neither is likely to make an appreciable difference in a low-intensity workload.

A common practice when dealing with a pair of SSDs is to create two partitions on each, one that's 8GB or so in capacity and the other that fills the remainder of the disk. Add the two smaller partitions as a mirrored log devices (ZIL) and the larger partitions as independent cache devices (L2ARC).

 

[EtherealDragon]

So, I am starting to get time to actually work on this server. The current install of Ubuntu Server will be redone, because it was set up on one of the 60 gig SSD's that will be used for L2ARC and ZIL. Once I get my 250 (or 240 or whatever it is SSD) that is in my PS3, THAT will be the OS drive. Off the bat, I will have 6 drives, all 4TB drives in the chassis. That will leave me with 6 more empty slots that I will likely fill up in the near future. The only problem I have, is ZERO place to put the other drives with the hotswap bays being strictly for the storage drives.

That said, the Motherboard has several PCI-X slots which I will not be using. They are the perfect length and width to hold some drives. My INITIAL plan was to build a drive tower for 3 drives.

I chose styrene for this, as it is plastic, won't conduct, is cheap, and SHOULDN'T MELT. The melting point is pretty high, and I BELIEVE it won't soften under 95*C. I am not quite sure what temps to expect out of the server, so I will be keeping a close eye on this to monitor.

I STARTED to measure everything out, and found that the styrene sheet was JUST wide enough for 3 equal sized pieces, and would PERFECTLY fit 2 drives back to back on it. So this gave me a tower for 6 drives total. I ended up short 8 nylon spacers, as I expected to do a shelf for 3 drives, not 6. I picked the extra 8 up today at the hardware store, all good to go now.

The thing I am not sure about is OS drive. Would it be a good idea to go with a MIRRORED setup with 2x SSD's in case one fails here?? Just thinking, I will HAVE the room, and I happen to have identical 240 gig drives, just need to be stolen out of an old laptop and my PS3....


Initial Plan:

Pre Glue mock up:

Ready for glue:

Server Drivebay Location:

Glued base sitting in place:

 

[mfenn]

The thing I am not sure about is OS drive. Would it be a good idea to go with a MIRRORED setup with 2x SSD's in case one fails here??

If uptime is very important to you, then by all means do RAID1 on the OS. Otherwise, a good backup plan will serve you better.

Also, please don't use Agility 3's for ZIL. Their performance consistency (latency variance over time) is not great, and that's the primary goal of adding a dedicated log device.

 

[PliotronX]

If uptime is very important to you, then by all means do RAID1 on the OS. Otherwise, a good backup plan will serve you better.

Also, please don't use Agility 3's for ZIL. Their performance consistency (latency variance over time) is not great, and that's the primary goal of adding a dedicated log device.

Good point, though I ran into a case of a fileserver a few weeks ago at a law firm whose RAID-1 mirror drive had not synchronized since 2013. RAID-1 is only smart if you are on top of the status of the arrays. The guy taking care of the law firm clearly was not D:

 

[EtherealDragon]

Well, I am not familiar with that particular metric when it comes to drives. I guess I will scrap those drives from the build then, or at least from the ZIL/L2ARC portion. Would the Agility 3's be sufficient in Raid 0 for the OS do you think?

The other 2 drives that I have are Toshiba Q Series HDTS225XZSTA. Would these suffice for L2ARC and ZIL partitions?

Additionally, I have some extra 2.5" 5400 laptop drives maybe I will put into the drive tower to just fill it up, and they can be used for storing back ups of the OS, or something which wouldn't need great performance.