Question NVME / U.2 vs U.3 / Cheap NVME high capacity

[Tech Junky]

So, I found something interesting when it comes to higher capacity NVME / PCIE drives that has me scratching my head as to why technophiles are giving their money to companies for M2 based options when they can get double the capacity for the same price in some cases.

It doesn't seem like a difficult proposition depending on how you want to do things. There's the PCIE card adapters and cages for 5.25" holes in the case.

There's some issues though that have me wondering about compatibility between the 2 though since they both use the same 8639 connector. Now, in theory the backplane / controller should dictate which drives work with which controller.

Some cards/cages say they work with both versions while others say they don't.

A U.3 NVMe drive is backward compatible to U.2 drive bays, but a U.2 NVMe drive can't be used in a U.3 drive bay

So, this is similar to backwards compatibility of say a Gen4 drive being put into a Gen3 slot and just reducing the BW of the drive. However, U.2 won't work in a U.3 position even though the connector is the same. So, if someone wants to take advantage of this storage option it would be best to get a U.3 card / cage and drive. However the price difference is significant depending on the capacity.

8TB U.2 Gen3 - $400 - https://www.amazon.com/dp/B0C6B2PGMC
8TB U.2 Gen4 - $500 - https://www.amazon.com/dp/B0B7Y4TFJR
16TB U.2 Gen4 - $1000 - https://www.amazon.com/dp/B0C6BR89C2

8TB U.3 Gen4 - $600 - https://www.amazon.com/dp/B0BHCMYH9Z
16TB U.3 Gen4 - $1000 -https://www.amazon.com/dp/B0BHGSXB4T

U.2 PCIE adapters
$21 - https://www.amazon.com/dp/B099185SSV
$40 - https://www.amazon.com/dp/B01D2PXUAQ

U.3 PCIE adapters
$15 - https://www.amazon.com/dp/B0C3C6MHH3
$16 - https://www.amazon.com/dp/B0C3HGSBGD
$27 - https://www.amazon.com/dp/B0C8L6DPZX
$42 - https://www.amazon.com/dp/B09RTZSN9X

Cages
U.2/3 2X $230 - https://www.amazon.com/dp/B0BHLTDZZ8
U.2/3 4X $400 - https://www.amazon.com/dp/B07B9HK4QG

With how newer MOBO's are killing off slots to make room for more M2 options it's a toss up in which direction to go but, with a 4TB NVME Gen4 being ~$200 it's compelling to look at these U options. 8TB G4 ~$800 and no 16TB options in the M2 format.

4 x 4TB runs about $800 shipped
8TB x 2 U.2 same price but, in my instance would switch from R10 to R1
Bumping back to a 16TB array though would be a significant change in affordability at ~$2000 / R1.

But, considering the higher capacity being the same price as 8TB NVME M2 dives it's considerably more affordable at 50% of the cost for capacity.

Evolving Storage with SFF-TA-1001 (U.3) Universal Drive Bays

U.3 is a term that refers to compliance with the SFF-TA-1001 specification, which also requires compliance with the SFF-8639 Module specification.

www.storagereview.com

SSD Form Factors | SNIA

Updated March 2023 Solid-state drives (SSDs) are commonly used in client, hyperscale and enterprise compute environments. They typically come in three flavors: NVMe®, SAS, and SATA. Since SSDs are made from flash memory, they can be built in many different form factors. This resource guide is...

www.snia.org

https://www.amazon.com/s?k=Oculink+SFF-8611+card+x4&i=electronics&s=price-asc-rank&crid=VM5LJIBEWY28&qid=1692213693&sprefix=oculink+sff-8611+card+x4%2Celectronics%2C148&ref=sr_st_price-asc-rank&ds=v1%3AqNKm69wi0NEUsBx0HFbdRxwkiYTmnI%2F9WpZ94SJ7bnA

Then there's Oculink SFF-8611 cards / cables / adapters where you can either use a PCIE slot or M2 to convert to U2/3 format as well.

I could swap the M2 drives for these @ $15/ea - https://www.amazon.com/NFHK-22x80mm-SFF-8639-Mainboard-SFF-8643/dp/B0BDYKG9RQ
Grab the cables - $29/ea https://www.amazon.com/Micro-SATA-Cables-SFF-8639-SFF-8643/dp/B07ZBKCFND

$45/drive to convert over might be a budget issue though for the ability to up your storage game. High density MOBO options with Gen4 slots might be an issue though to find w/o going with the M2 conversion method. I've had some 5-6 slot options in the past though but, the consistency of the slot speeds / sizes / electrical wiring vary. Moving to something like a X299 though would give tons of slots all at Gen3 speeds.

 

[Igo69]

I dont get it why are consumer nvmes are way more expensive than this is it because these is a new tech or new interface or less reliable?

 

[Tech Junky]

I dont get it why are consumer nvmes are way more expensive than this is it because these is a new tech or new interface or less reliable?

Well, it's a combo of ease and lack of mobo's having the option built in which requires thought by the consumer. They're getting cheaper though but, for whatever reason they weren't on my radar for U series drives.

Consumer gear needs to get in line with enterprise options and it will drive prices down and make it simpler to innovate. Considering I just redid my setup w/ 4xM2 vs 4xHDD for storage it's got me thinking about regressing and going with the U route instead since I ended up blowing away my Raid setup anyway and starting over instead of hoarding all of the media I was storing on it. I can getaway with just doing 2 drives and simply sync them w/ rsync and avoid the whole raid setup. I just have to look at things a bit deeper and come up with a solution that works for me at this point.

 

[Tech Junky]

@Igo69

So, I specced out both options U2/3 w/ 2 8TB drives and adapters / cables needed. Both drives being Gen4 options so it's apples to apples for speed / capacity w/ the only difference being the connector on the end of the cable.

U.3 x 2 / 16TB
M2 - 22.88
Cable - 25.88
Drive - 575.77
Total - 1,249.06
Shipped - $1,352.12

U.2 x 2 / 16TB
M2 - 18.94
Cable - 18.88
Drive - 465
Total - 1,005.64
Shipped - $1,077.78

Difference - 274.34

Current
Z790 / DDR5 - $240.50
4 drives - $824.20
Shipped - $1064.7

Not much of a difference other than my prior MOBO doesn't have ample Gen4 M2 sockets to plugin to. I could swap in PCIE adapters though

If I shipped my new setup back MOBO/Drives/RAM ($1064) and then swapped in the new U2/PCIE card adapter ($1093) I would still get the same results but regress back to a Z690 chipset / DDR4 for a $30 difference.

If I kept the MOBO/DDR5 and swapped just the drives it's a net cost of 237.50 more to switch things up to U2.

So, yeah, I'm not sure I would do another rebuild to use U2's unless I was ready to pony up the cash for the 16TB drives which for is when it would make sense to convert over. Though dealing w/ 2xU2 would simplify things vs R10 w/ M2's.

 

[Igo69]

Damn, this is a lot cheaper than consumer 8TB NVMe (the cheapest i found is ~$800 for 8TB).

 

[Tech Junky]

Damn, this is a lot cheaper than consumer 8TB NVMe (the cheapest i found is ~$800 for 8TB).

Yup, I pivoted and ordered a 16tb micron and going single disk instead. Reverted back to my other mobo/ram and the difference in price is under $100. Going from spinners at 400MB/s to nvme gen 4 in a 2.5" format that does 6GB/s. Also, achieve the weight loss I'm aiming for with the spinners being out of the case.

Lack any redundancy but, shouldn't be an issue to add that later when ARL comes out next fall and I switch things up again. Can always add a bigger drive to my laptop for backups of important stuff.

 

[mxnerd]

a 16tb micron

This one U.3

Amazon.com

$1068 6800MB read / 5600MB write

instead U.2 https://www.amazon.com/Micron-15-36TB-Enterprise-Solid-State/dp/B07SK8GSYZ $1499 ? 3500MB read/write

A huge difference, wow. Micron U.3 is faster and even cheaper than U.2 now.

 

[mxnerd]

Micron 7450 Pro U.3 8TB (7680GB) $585

Amazon.com

https://media-www.micron.com/-/media/client/global/documents/products/product-flyer/7450_nvme_ssd_product_brief.pdf

PCIE 4.0 x 4 max speed 8GB/s (64 Gbps) is more than enough for most people. It will even saturate a 40Gbps link.

 

[Igo69]

Please let us know how this works out. I am very curious as i want to make this too because I want to build a new PC in a near future.

 

[Tech Junky]

@mxnerd

I went U3 / 7450 / 16tb. There were really only a handful of options that seemed worthy. Only a couple of brands and the Intel drives didn't really deserve a premium. Going with the 9400 micron didn't make much sense either with the two being about the side moon performance. The 9400 being about $400 more.

I was debating raid 1 with 2*8tb but figured prices will come down as they age a bit and just pick up a second 16tb later. I'm considering consolidating my os if it's bootable from a pcie slot and then just backup the OS to the M2. That's a quick test to determine whether or not it works.

@Igo69

I will keep updating the thread. While poking around though it's a bit of a macgyver situation with a few options to pick from. There's the pcie adapter plus drive, M2+cable+drive option if you want to keep a slot free but the M2 options are hit or miss on speed with most seemingly to only do 4gbps where the pcie card unleashes the full speed. Then there's oculink but the M2 adapters come in both oculink and msas options. Maas being slightly cheaper. Of course going with a true server board would be easiest but costs more to do everything.

 

[Tech Junky]

Found a handy parts list for Micron to decipher whether it's a 7mm / 15mm design and the other options.

https://media-www.micron.com/-/media/client/global/documents/products/part-numbering-guide/7450-product-catalog.pdf?la=en&rev=e457ad303bf14160aee9d7233982c63d

 

[Igo69]

A cheaper alternative 2 of these (SAMSUNG 870 QVO) for ~15TB or 3 for ~24tb RAID.

This is QLC though

https://www.newegg.com/samsung-8tb-870-qvo-series/p/N82E16820147784

 

[Tech Junky]

SAMSUNG 870 QVO

I considered them but the price V performance matrix doesn't add up to restrict yourself to 500MB/s and legacy SATA.

My latest was 4*4tb gen4 and switching to the single 16tb option was less than $100 and the same speed and simpler.

16tb was my target from the spinner edition though I wiped all of that data to start a new collection. I trimmed back the collection though under 8tb for the nvme switchover and made things fit.in the mean time while sending things back to Amazon and waiting on the new stuff to show up I threw a couple of cheap 1TB NVME drives in to tide me over. I put them in R0 for capacity if I get frisky on DL's over the next week but, they're not the best for this sort of thing as they are both Gen3 and use Phison controllers which means they take a nose dive on larger sustained transfers.

My other target was to keep this adventure around $1K total. It's got me thinking though my next build should be AMD based though if they can get all of the gremlins worked out and a stable platform. I can then use a dual U2 card and bifurcate the slot into x4/x4 but, those cards are $200 and only Gen3 speeds. though there are some dual Oculink options that are cheaper for running X8 and splitting to 2 drives.

It's a bit of a wash depending on the approach of dives / controllers / cables /etc. Raid adds to the cost even though it's controlled by the MOBO / OS since you need more drives for redundancy purposes and potentially switching to R5 from R10 to get capacity and parity for less than 4x drives. There's juts a budgetary break point vs just doing it.

I'm happy with the idea of getting away from using M2's though even though the option is there. The board I picked up worked fine w/ 5 M2 sockets it's just over complicated and more expensive.

 

[Tech Junky]

Adapter card showed up today and looks like the drive is showing up tomorrow instead of next week.

 

[Tech Junky]

For everything installed and started copying data and something spazzed. Got IO errors on the new drive and then couldn't repartition it again but Linux can see it electrically just can't put data on it. Swapped it to a different system to rule that out and couldn't do anything with it there either.

While it was working I was getting transfer speeds from my sn850 around 2.5GB/s which isn't full performance but still pretty decent crossing different buses. Ordered a different card to test with before shipping everything back and ordering a new drive again. Price dropped on the drive though which makes up for the cost of the card. Jait sucks to have to wait again for new hw to show up.

Maybe it's just the cheap cn card causing the issue but my gut says it's the drive at this point since it didn't work in either system. It's always a chance for something to fail when it comes to tech. Round 2 hopefully goes better and is stable.

 

[mxnerd]

Seems the drive you bought from is a small seller, is the drive completely sealed and looks new?

 

[Tech Junky]

Seems the drive you bought from is a small seller, is the drive completely sealed and looks new?

Everything was sealed. I suppose I could run a SN check on it as well.

It's not unusual to have a DOA / failure though. It just happens. I had one of my several 8TB HDDs arrive DOA in the past but, have had those in service 24/7 for ~8 years w/o issue. I just haven't had a drive spazz out and then drop the partition table in quite a long time.

 

[mxnerd]

Could be contact pins, use rubbing alcohol on the adapter/drive.

 

[Tech Junky]

https://www.superwarehouse.com/micr...3-15mm-non-sed-mtfdkcc15t3tfr-1bc1zabyyr.html
$1,069.43 shipped / no tax

Seems like a bit of savings considering w/ tax Amazon for me comes out to $1,135.54 / $65 more w/ tax

 

[Tech Junky]

Ok, 2nd Micron drive borked out and lost data. This one lasted almost a week compared to the other that did it the same day. I though I was out of the danger zone but, went to add some data and got an error. Was able to READ data or at least list it in the drive but, it was acting funny and rebooted the system and bam it was all toast like the last drive.

I'm not even going to mess with it at this point and looking at some different brands instead. 2 failures is enough for me.

Looking at the Samsung PM1733 or Intel D7-P5520 both come in the 15.36TB configuration same as the Micron but, aren't Micron.... I even bothered to update the firmware on this one which I couldn't on the last one since it freaked out and I wasn't aware there was an update until I went TSing the damned thing. So, I swapped the adapter card the drive sits in w/ the new drive as well to rule out cheap CN crap being the issue at the same time. I don't get why they keep failing though. What are the odds of 2 drives from different batches doing the same thing? Maybe there's a reason they're showing up for sale? Maybe the 9 series version is more stable?

 

[mxnerd]

I would avoid buying products that have very few reviews if I can.

 

[mxnerd]

New tech early adopters always have to pay high price and also take risk at the same time.

 

[Tech Junky]

@mxnerd

Enterprise gear won't have typical reviews so that is nonsense here.

 

[Shmee]

It may be that the seller is selling scam modded/defective/fake drives. I wouldn't think that enterprise Micron SSDs would have any sort of issues if new, or even if used but legit.

On a side note, could anyone recommend NVMe drives and adapters for a PCIe gen3 system? Asus X99 deluxe with Xeon E5 1660v3, so plenty of lanes.

 

[Tech Junky]

@Shmee they're new and two different sellers. If I rill the dice again I'm going. Samsung or Intel I think and probably rebuilding with AMD to use bifurcation since I can get 2*8tb drives for $50 more than an 8tb M2 and $200 less than a 16tb single drive.

It's kind of a lateral move though from 12700k >> 7900x in terms of performance but gain a couple of threads to maybe do more with. I've been contemplating the switch to AMD for a few years and it's just time to say goodbye to Intel.