ZFS build questions

[nvsravank]

I have been enamoured by ZFS for its ability to protect from silent data at rest corruptions.
I am a photographer and I have this unexplainable fear of loss. One file being bad is not the end of the world for most but it is for me. So with reasons as to why I want zfs, taken care of let me ask my questions.

- Do the current ZFS systems allow adding disks easily to upgrade like what drobo does? (I use it for backups). Do I have to learn the internals of how best to setup the disks like what raid forces me to?

- What protocol support is necessary if I want my windows applications to treat it as local disk? I use Lightroom and it does not handle remote network shares well. So do I need CIFS or ISCSI shares?

- I want to have the ability to do backup from the ZFS to drobo.

- I have a four 2TB drives and two 1TB drives. Regular drives and not specific to raid TLER disks. The data currently is around 2TB with expectation of going up to 4TB in the next 2 to 3 years. I would like to use the 2TB disks in the DROBO, and so would like to know best disks for this ZFS box

- I want very good performance so I plan to add one or two SSDs. Want to know how many needed. I would like the storage to be automatically tiered spo that when I get the photos from my current shoot they are on SSD for fast access and go to hard disk once I stop using them. I want the storage system to manage it. Similar to what is advertised by the rack boxes from drobo( note I said advertised, since I have no idea how it works in real life. ) I am planning to buy a higher mp camera and already my raid1 disk subsystem is giving trouble.

- I want build my own ZFS box with nas4free or ZFSGuru(read a bit about it today). So what specs do I need?

- What ZFS settings to use for my purpose?

 

[CiPHER]

Why ZFS
ZFS offers you not just protection against corruption, but also detection. That is already so valuable! Finally you just know when your data is GOOD or when parts of it are corrupted. Silent corruption will be something of the past. ZFS is simply the way to store data in 2010+.


ZFS expansion
ZFS has one big limitation for home users though: expansion doesn't work as regular RAID solutions work. ZFS can do expansion methods that regular RAID will not be able to do, so it's not just all negative. But the feature is missed that allows you to add one disk at a time to a parity RAID pool. Instead, ZFS expansion works by adding vdevs, meaning that you add multiple disks at the same time together with redundancy. So if you have one 5-disk RAID-Z, you can add 3-disks in RAID-Z or 6 disks in RAID-Z2 to the same pool. But you cannot add one disk and get a 6-disk RAID-Z.

It is not so bad, because ZFS saves you a lot of money on the controller, BBU, UPS, and expensive TLER/uBER 10^-16 disks. The money saved can be invested on more disks; problem solved! Plus you get much more capacity for free.


How to use ZFS on your Windows desktop?
You can use either SMB/CIFS ("windows filesharing") or other protocols. In your case you probably work with Adobe-products, which have the nasty habbit of sticking their nose where it doesn't belong. Adobe-products mess around with swap - where no application should ever mess with. And if you say they discriminate against network disks that doesn't surprise me.

The result is that you have to use iSCSI. But know that part of the protection of ZFS, will go down the drain. Essentially you are storing your data on a legacy filesystem (NTFS) and the virtual harddrive with NTFS on it, will be stored on ZFS. That is what happens when you use iSCSI. It means ZFS can be corruption free, but your legacy filesystem still has corruption, detected or undetected.

You can use both iSCSI and CIFS though. So only use one iSCSI volume for Photoshop/Adobe-products and use CIFS for the rest. But the limitation here is that you cannot easily shrink the iSCSI volume. You can expand it however.


Backup
Backup is not different from other solutions. But know that ZFS works best with ZFS send/receive, so that during the transport all checksum protections apply and the transfer of data is secured. Whereas normal copying is insecure and will not notice corruption spread to the backup. Rsync is possible with checksums, but incredibly slow. ZFS send/receive requires that both origin and destination are ZFS, however.


What disks for ZFS?
Use normal consumer-grade disks. The whole point of ZFS is that you can use cheap consumer-grade hardware. This was the whole point of RAID as well (hence the 'i' means Inexpensive). But companies like EMC have perverted the origin and made RAID into something expensive thanks to market segregation. But with ZFS you no longer have to be exploited, and can use regular cheap technology.

That means: WD Green and sorts. I recommend WD Green 4 to 6TB. WD Red works well too, but with TLER means it has no protection against bad sector recovery - since TLER basically disables the recovery functionality of the drive. For redundant ZFS pools this doesn't matter. But it will matter (slightly) if you have lost your redundancy. ZFS itself uses redundant metadata even if running as single disk or degraded RAID-Z, so ZFS itself cannot be damaged by bad sectors.

For mass storage, choose disks with as low rpm as possible, and with high as possible data density. Then you get high performance at low noise and low power consumption and high reliability. If you want IOps use SSD L2ARC caching - high rpm drives suck just as well they just suck slightly less than 5400rpm class harddrives.


SSDs
SSDs can be used by ZFS to accelerate the pool as L2ARC cache or sLOG buffer. But it doesn't work like you said. Big files (1MB) that get read from begin to end, will not be cached by the SSD. Only non-contiguous reads will be cached; random reads. This is where harddrives are bad at. Harddrives are very good at doing I/O on large files. 1MB is already large by this definition.


What hardware do you need for ZFS?
Tip: first decide on the motherboard. It will decide the rest of the hardware. You may have a look at this overview of motherboards.


What settings to use?
Default settings. But if you mean what pool configuration, i would suggest at least RAID-Z2. One SSD for caching metadata might help slightly, but not all that much.

 

[smitbret]

Why ZFS
ZFS offers you not just protection against corruption, but also detection. That is already so valuable! Finally you just know when your data is GOOD or when parts of it are corrupted. Silent corruption will be something of the past. ZFS is simply the way to store data in 2010+.


ZFS expansion
ZFS has one big limitation for home users though: expansion doesn't work as regular RAID solutions work. ZFS can do expansion methods that regular RAID will not be able to do, so it's not just all negative. But the feature is missed that allows you to add one disk at a time to a parity RAID pool. Instead, ZFS expansion works by adding vdevs, meaning that you add multiple disks at the same time together with redundancy. So if you have one 5-disk RAID-Z, you can add 3-disks in RAID-Z or 6 disks in RAID-Z2 to the same pool. But you cannot add one disk and get a 6-disk RAID-Z.

It is not so bad, because ZFS saves you a lot of money on the controller, BBU, UPS, and expensive TLER/uBER 10^-16 disks. The money saved can be invested on more disks; problem solved! Plus you get much more capacity for free.


How to use ZFS on your Windows desktop?
You can use either SMB/CIFS ("windows filesharing") or other protocols. In your case you probably work with Adobe-products, which have the nasty habbit of sticking their nose where it doesn't belong. Adobe-products mess around with swap - where no application should ever mess with. And if you say they discriminate against network disks that doesn't surprise me.

The result is that you have to use iSCSI. But know that part of the protection of ZFS, will go down the drain. Essentially you are storing your data on a legacy filesystem (NTFS) and the virtual harddrive with NTFS on it, will be stored on ZFS. That is what happens when you use iSCSI. It means ZFS can be corruption free, but your legacy filesystem still has corruption, detected or undetected.

You can use both iSCSI and CIFS though. So only use one iSCSI volume for Photoshop/Adobe-products and use CIFS for the rest. But the limitation here is that you cannot easily shrink the iSCSI volume. You can expand it however.


Backup
Backup is not different from other solutions. But know that ZFS works best with ZFS send/receive, so that during the transport all checksum protections apply and the transfer of data is secured. Whereas normal copying is insecure and will not notice corruption spread to the backup. Rsync is possible with checksums, but incredibly slow. ZFS send/receive requires that both origin and destination are ZFS, however.


What disks for ZFS?
Use normal consumer-grade disks. The whole point of ZFS is that you can use cheap consumer-grade hardware. This was the whole point of RAID as well (hence the 'i' means Inexpensive). But companies like EMC have perverted the origin and made RAID into something expensive thanks to market segregation. But with ZFS you no longer have to be exploited, and can use regular cheap technology.

That means: WD Green and sorts. I recommend WD Green 4 to 6TB. WD Red works well too, but with TLER means it has no protection against bad sector recovery - since TLER basically disables the recovery functionality of the drive. For redundant ZFS pools this doesn't matter. But it will matter (slightly) if you have lost your redundancy. ZFS itself uses redundant metadata even if running as single disk or degraded RAID-Z, so ZFS itself cannot be damaged by bad sectors.

For mass storage, choose disks with as low rpm as possible, and with high as possible data density. Then you get high performance at low noise and low power consumption and high reliability. If you want IOps use SSD L2ARC caching - high rpm drives suck just as well they just suck slightly less than 5400rpm class harddrives.


SSDs
SSDs can be used by ZFS to accelerate the pool as L2ARC cache or sLOG buffer. But it doesn't work like you said. Big files (1MB) that get read from begin to end, will not be cached by the SSD. Only non-contiguous reads will be cached; random reads. This is where harddrives are bad at. Harddrives are very good at doing I/O on large files. 1MB is already large by this definition.


What hardware do you need for ZFS?
Tip: first decide on the motherboard. It will decide the rest of the hardware. You may have a look at this overview of motherboards.


What settings to use?
Default settings. But if you mean what pool configuration, i would suggest at least RAID-Z2. One SSD for caching metadata might help slightly, but not all that much.

This is quite possibly the best write-up on ZFS that I have seen that puts it in layman's terms.

I nominate for a sticky.

 

[gea]

Very good summary.

I want to add the main problems for any storage without ZFS as ZFS mainly adds two principles to the storage world: End to end checksums and CopyOnWrite.

The problem:

- If there is any problem with your PSU, disk-driver, cabling, backplane or disk, resulting in a disk error: you will never know without realtime checksums on OS level.

- if there is any silent disk corruption due to radioactivity, magnetic fields or pure statistics : you will never know without realtime checksums on OS level.

- Any problem during a write (power failure ot other) may result in a corrupted filesystem. Best protection is a CopyOnWrite filesystem where a write is done completely or not at all.

- ZFS Raid is not backup but many problems are irrelevant with read only ZFS snaps=previous file versions .


The answer:
ZFS (and other modern filesystems like btrs, ReFS or WAFL) where only ZFS is free or best of all.

 

[nvsravank]

ZFS expansion
The feature is missed that allows you to add one disk at a time to a parity RAID pool. Instead, ZFS expansion works by adding vdevs.

I did a little more reading on this. So because i cannot increase the number of disks in a Vdev, I can increase the size of the Z pool in two ways.
1. Add another Vdev to the pool.
2. Replace each disk one at a time with resilvering operation in between to get the status back to normal.

add another Vdev to the Pool
If we add another Vdev to the zpool does ZFS automatically then start trying to write to the new Vdev to get the Vdev populated equally?

I read that if you have multiple Vdevs then it basically is like raid 0 stripes accross the vdevs. Does this mean that a file can be split between two vdevs?

Replace disk and resilver
I am assuming this increases the risk of disk failure and irrecoverable data loss. And takes a long long time.

Am I better off creating a new zpool with the new vDev and doing a ZFS based copy to the new Zpool? (Backup or snapshot?)

Mixing disks of different sizes
Mixing disks of different sizes in a Raid-Z volume will give the overall size based on the size of the smallest disk. So it is better to do the RaidZ based on same size disks. So based on all of the above, I am thinking a RaidZ2 with four of my 2 TB disks would be the best to get a 4TB drive which will give me a cushion for around 2 to 3 years.
Then just keep the two 1 TB disks in the Drobo and add disks of the next size (4TB) to it until i am ready to do another vDev/Zpool to increase the size.

How to use ZFS on your Windows desktop?
The result is that you have to use iSCSI. But know that part of the protection of ZFS, will go down the drain. Essentially you are storing your data on a legacy filesystem (NTFS) and the virtual harddrive with NTFS on it, will be stored on ZFS. That is what happens when you use iSCSI. It means ZFS can be corruption free, but your legacy filesystem still has corruption, detected or undetected.

So basically what you are saying is create Datasets and share them into the OS environment and not do a volume creation in ZFS as volumes dont give the full protection that datasets give.

Backup
Backup is not different from other solutions. But know that ZFS works best with ZFS send/receive, so that during the transport all checksum protections apply and the transfer of data is secured. Whereas normal copying is insecure and will not notice corruption spread to the backup. Rsync is possible with checksums, but incredibly slow. ZFS send/receive requires that both origin and destination are ZFS, however.

Can i have two Z pools on the same hardware? (Except the disks of course)
If os can i setup one pool as back up of the other pool? Or do you think that doesnt add any value? In this backup to a different location is the only real use of backups? Trying to figure out a backup strategy that is necessary with ZFS.

What disks for ZFS?
Use normal consumer-grade disks.
...
For mass storage, choose disks with as low rpm as possible, and with high as possible data density. Then you get high performance at low noise and low power consumption and high reliability. If you want IOps use SSD L2ARC caching - high rpm drives suck just as well they just suck slightly less than 5400rpm class harddrives.

So buy the largest sized disks even if they are slow. Dont worry about RAID features and you are better off with regular disks than those so called raid disks (TLER).
Wont that cause more time for resilvering?

SSDs
SSDs can be used by ZFS to accelerate the pool as L2ARC cache or sLOG buffer. But it doesn't work like you said. Big files (1MB) that get read from begin to end, will not be cached by the SSD. Only non-contiguous reads will be cached; random reads. This is where harddrives are bad at. Harddrives are very good at doing I/O on large files. 1MB is already large by this definition.

Is it better off to have an SSD in RAID 1 for working files then? Use the ZFS volume for long term storage but use a pair of SSD for working files and then migrate to the ZFS volume once the regular usage is done.

What hardware do you need for ZFS?
Tip: first decide on the motherboard. It will decide the rest of the hardware. You may have a look at this overview of motherboards.

Great i didnt know i can get a board with the CPU with it for this kind of application! I think the decision i need to make is how many SATA ports to have. Currently i am thinking of having a vDev with 4 disks and if i want to expand easily then i would need to have another 6 ports to put the disks for the new Zpool (to avoid resolvering). Add a couple for SSDs for cacheing that puts the ports at 12!. That seems like a lot of ports. Add chassis to hold so many disks is not easy to find either! Any other options?

What settings to use?
Default settings. But if you mean what pool configuration, i would suggest at least RAID-Z2. One SSD for caching metadata might help slightly, but not all that much.

I meant what "Features" to think about.
Reading more i dont think deduplication will help me much. With photos there is not much duplication really.
Compression will not help again. The photos I store are RAW files which use internal compression that is much more effective than any generic compression can achieve.

I am thinking i can add ZIL and L2ARC as add on later. I can do that right?

if i am doing raidZ2 i dont need to turn on duplicates right?

PS: _CiPHER_ are you the same person as is the dev in ZFSguru?

 

[Essence_of_War]

I did a little more reading on this. So because i cannot increase the number of disks in a Vdev, I can increase the size of the Z pool in two ways.
1. Add another Vdev to the pool.
2. Replace each disk one at a time with resilvering operation in between to get the status back to normal.

add another Vdev to the Pool
If we add another Vdev to the zpool does ZFS automatically then start trying to write to the new Vdev to get the Vdev populated equally?

I read that if you have multiple Vdevs then it basically is like raid 0 stripes accross the vdevs. Does this mean that a file can be split between two vdevs?

ZFS will try to push more of future writes to balance the vdevs. What it won't do is move data automatically if there aren't writes happening. So if you have an enormous file, and you add another vdev, until you actually modify that file, it's going to sit on the vdevs it was originally committed to.

ZFS dynamically stripes all writes to the pool across all vdevs. So every vdev you've got is effectively a new stripe of the pool. Two vdevs are a two-way stripe, 3 vdevs are a three-way stripe, etc. So if you lose a vdev, you lose the pool. Plan redundancy for your vdevs as appropriate, and make back-ups too.

Replace disk and resilver
I am assuming this increases the risk of disk failure and irrecoverable data loss. And takes a long long time.

Am I better off creating a new zpool with the new vDev and doing a ZFS based copy to the new Zpool? (Backup or snapshot?)

The amount of time/IO for a resilvering operation depends on the redundancy. Mirror rebuilds tend to be MUCH faster and easier than raidz1/z2 rebuilds. If you're switching to larger disks, as long as you have adequate backups and redundancy, it's fine to swap disks and rebuild.