Question NVME / U.2 vs U.3 / Cheap NVME high capacity

[mxnerd]

@Shmee See if following post helps. Googled info. Didn't read through the whole post.

Don't know if you need to do BIOS mod to add bifurcation functionality to the motherboard to make all 4 nvme drives to be seen if you want to have more than one nvme drive on a single adapter.

[Request] PCIe Bifurcation on Asus X99 Deluxe

Hello @Lost_N_BIOS & @davidm71 Today I received my new M2 adapter card which consists 4 M2 slots and looks it works greatly. Even the issue I mentioned before seems somehow disappear (still need verification). I have 3 M2 nvme devices installed on that card and the performance is quite...

winraid.level1techs.com

JEYI Quad NVMe PCIe 4.0 Expansion Card, Required Bifurcation

Amazon.com

 

[Shmee]

Yeah I came across that very post, it looks like something I would have to do to get more than a single SSD per PCIe slot. I was thinking about doing the u.2 option, with something like this https://www.amazon.com/Septpenta-Fu...dp/B0BHWQ1RS6/ref=psdc_13983711_t3_B0B7BHM954 and something like two of these https://www.ebay.com/itm/2663843369...60kaWrVHIxPebNwGR/e4c02jF3|tkp:Bk9SR6KVqr_QYg

 

[mxnerd]

You still need the motherboard to support bifurcation as long as you want more than one drive per single x16 adapter even if you use U.2 or U.3 form factor since all of them only support PCIE (1.0 to 4.0/5.0) x4 interface

another post regarding BIOS mod.

[REQUEST] Asus x99-a 3.1 4x4x4x4x bifurcation

Hello, I am in desperate need of enabling 4x4x4x4x

www.bios-mods.com

 

[Shmee]

You still need the motherboard to support bifurcation as long as you want more than one drive per single x16 adapter even if you use U.2 or U.3 form factor since all of them only support PCIE (1.0 to 4.0/5.0) x4

True, but from my understanding reading the thread you linked on the Asus X99 Deluxe bifurcation, it is doable.

As for choosing u.2 SSDs, any reason to not go with some from the link I posted? It looks like those should be Samsung PM 1733 drives, which I understand are pretty good.

 

[mxnerd]

Sorry, but I'm not really familiar with enterprise stuff, I just like to update my knowledge by reading a lot of articles.

 

[Shmee]

@aigomorla what are your thoughts on the PM 1733 compared to the PM 983 drives you have?

 

[Tech Junky]

@Shmee having blown through 2 micron's at this point my next would be SS or Intel. I've seen more of those in use per reviews. I was just hoping for better writes with the micron drives since the others are slightly lower. If you go with the 8tb options there's Intel and WD options for $400. I'm thinking about pairing one of each on the AMD setup and either just use them separately or have one as a backup destination and use rsync to automate things.

 

[Shmee]

Well, I ended up ordering 2 Samsung PM 1733 drives for a good deal. Ordered 2 cables and a card as well. Should all be here this week.

 

[igor_kavinski]

Saw this: https://www.newegg.com/kioxia-cd6-r-7-68tb/p/N82E16820988003?Item=9SIAD07K083553

Pretty cheap if all you want is reads.

1 DWPD is pretty good at this price.

U.3 to PCIe 4.0 x4 adapter: https://www.amazon.com/Ableconn-PEXU3-132-2-5-inch-SFF-TA-1001-Carrier/dp/B09RTZSN9X

 

[Tech Junky]

@igor_kavinski

That's a decent price for Kioxia that I didn't see before. Though I found a couple of options with WD / SS that beat the price by $50/drive.

https://www.amazon.com/dp/B08WGNKJTH - WD
https://www.amazon.com/dp/B0C6B2PGMC - Intel
https://www.amazon.com/dp/B0B7Y4TFJR - SS a bit more @ $530 / G4

Only difference is they're Gen 3 in terms of speed. Sure, it would be nice to have something as fast as my SN850 for moving data back and forth but, reality is it's not necessary. That was the initial appeal of the Micron drives being somewhat on the same playing field in terms of performance on paper at least since they both spazzed out and lost their partition table.

I figure if the problem follows across brands then it's a ME problem and not a U problem. I'm just pausing and ruminating before diving into the next drive / system build after 2 experiences that give pause to the idea. For some reason coincidence or not the last micron caused the OS drive to have a bloat issue that required a redo of it as well to purge it from the system. I tackled that from different approaches and spent most of a day working out the kinks. Of course the backup files helped save some configuration time but, not quite as quick of an operation could be. Might adjust it to a smaller set of data since it's a bit of a PITA either way. But, it's a svelte ~14GB now w/o all of the clutter. Whatever gremlin was in there had it up to 900GB+

 

[Shmee]

Update here, I successfully modded the Asus X99's BIOS in the Truenas to bifurcate one of the PCIe slots. I also learned how to make a Windows to go VHDx for a Ventoy USB in the process, which I used to test and flash the BIOS. Now both the Samsung PM 1733 drives are showing up successfully.

 

[Tech Junky]

@Shmee

Good job! I ended up switching to a card+cable vs the direct mount PCIE card / drive option I started with and just waiting on the new drive to show up Tues. For some reason they ship these things pretty slow no matter the dealer or site you use.

The 1733's came up a lot but, something stuck out with the connector being mentioned that didn't sit right with me as for the bandwidth. I couldn't find any pics of the connector readily though I admit with the red flag gut feeling I got from the mention of it I didn't put much effort into finding them.

I've been talking with someone though on another board thar recently went down the Micron avenue and upgraded their system last week and while it's only been a couple of days the only thing they've noticed so far is slightly higher temps on the drives and put some fans on them directly to bring the temps down to a more moderate 40's level.

So, when you said you had to hack your BIOS.... X99 leads me to think Intel and that's a feat to deal with when it comes to bifurcation w/o spending more to go to the higher platforms. Ventoy though is a nice little swiss army of a tool.

Oh, the drive I switched to is a Kioxia KCD8XRUG15T3 15.36TB. I had my eye on a different model that was cheaper but, the seller sold them all by the time I got done researching options and landing on that one to be the best at the time. Maybe it was a blessing as the one I went with has better performance in terms of R/W speeds.

Since this is an equiv in terms of speed with the current OS SN850 I'm using in the server I'm considering just moving the OS to the drive as well and consolidating the bulk storage into a single drive. Then again in the back of mind is backing up some of the vital settings becomes a bit more difficult though on the rebuild most of the stuff I ended up pulling were from one directory as dropping the system onto a fresh format didn't work so well. But, that's all a whole different story but, the micron experience lead to a data bloat that consumed the free space of the SN850 for some reason.


###########################
General info I've come across since starting this...

U.2/3 drives use SFF-8639

SAS 12 / 1.5GB/s - SFF-8643
SAS 24 / 3GB/s - G3 / SFF-8654
Oculink 32 / 8GB/s - G4 . SFF-8611/8612

While this falls into the standards of each device type... There are hints since these are just electrical adapters the mentioning of the standard / speed doesn't apply as they're pass through devices and can hit top end speeds for a Gen4 drive. I saw some reviews on different cards / cables / adapters showing the ability to hit 7GB/s but, in my gut and with the problems thus far didn't want to push my luck yet again.

When looking at comparing these different setups though it's reasonable to expect to cost ~$50/drive to get it up and running. The initial appeal with the card adapters was the sub $20 per card though they do go up to $50 as well. There are some M2 / cable adapters as well that vary in costs and the guy I've been talking to is using those to leave room for PCI slots to be used for other things.

 

[Shmee]

Yes X99 is Intel, but it is one of the past HEDT platforms, and many X99 boards support Xeon CPUs, such as my Asus X99 Deluxe. As for cards and cables, I got this card https://www.amazon.com/dp/B0BVMK2K1Z?ref=ppx_yo2ov_dt_b_product_details&th=1and 2 of these cables https://www.amazon.com/dp/B07JH92TD9?psc=1&ref=ppx_yo2ov_dt_b_product_details

As you can see, since I have bifurcated the PCIe X16 slot the card is in to x4x4x4x4, I can order another two cables and two additional drives in the future.

 

[Tech Junky]

bifurcated

That's a given to me. Afaik though it's slim pickings on being able to in the Intel side unless you go X/hedt level. I was looking at those as well until digging into the speed of the links based on connection types.

Since you found the key to the kingdom here's another option.

Amazon.com

Amazon.com

A little cleaner cable management but, same result. Shouldn't be a problem later on when higher performance drives finally trickle down as well. If I did an AMD rebuild that's probably what I would use. Though, the need for space shouldn't be an issue for quite awhile now.

 

[Tech Junky]

UPDATE....

Switched to a PCI adapter and cable setup since the slotted drive adapters x 2 failed. So,,,,,,,

Started with the card + 100CM cable and nada.
Switched to a 50CM cable and got signs of life through the system renumbering the NIC IF's on boot which is normal to see when it picks up +/- a PCI device.
Go look for the new disk.... NADA! WTF!?!? double checked all the cabling and rebooted again and NADA.
Decided to move the card down to another slot which happens to still be G4 instead of the G5 top slot both of the other cards were in w/ Micron's and moved my NIC to that slot and watched for lights on power on and they started dancing like they should so, the slot isn't dead and works fine other than anything associated with these drives. Go figure.
Boot to the desktop and open "disks" and sure as S*&^ it shows up in the list.
Changed my mount points on the M2 to backup and put the U3 on Storage by UUID as this avoids the renaming complex Linux has when booting. Reboot and verify.
Things up, mount, etc.

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme2n1     259:0    0 931.5G  0 disk
└─nvme2n1p1 259:1    0 931.5G  0 part /mnt/backup
nvme1n1     259:2    0    14T  0 disk /mnt/Storage
nvme0n1     259:3    0 931.5G  0 disk
├─nvme0n1p1 259:4    0     1G  0 part /boot/efi
└─nvme0n1p2 259:5    0 930.5G  0 part /

OS - 09:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01)
U3 - 0a:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller CD8 (rev 01)
M2 / backup - 0c:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01)

Now, to see how this one works AFA lasting more than a week. Monitoring temps a bit closer as well due to mentions of high temps on the Micron's someone else added to their system. Right now though it's sitting at 38C which is in line with the 2 x M2's in the system currently.

Right now it's just sitting in the case not actually mounted though and will need to figure that out once it lives longer than the return period and proves itself. So, the other thing that's going to make my OCD go haywire is the slot issue with the X16 G5 being flakey. No real reason it shouldn't be able to work with the adapter / cable / drive. The other option is to switch the drive adapter to an M2 format and push the NIC back down to the original slot. Either way both options result in the drive having G4 x4 speeds.

 

[Shmee]

UPDATE....

Switched to a PCI adapter and cable setup since the slotted drive adapters x 2 failed. So,,,,,,,

Started with the card + 100CM cable and nada.
Switched to a 50CM cable and got signs of life through the system renumbering the NIC IF's on boot which is normal to see when it picks up +/- a PCI device.
Go look for the new disk.... NADA! WTF!?!? double checked all the cabling and rebooted again and NADA.
Decided to move the card down to another slot which happens to still be G4 instead of the G5 top slot both of the other cards were in w/ Micron's and moved my NIC to that slot and watched for lights on power on and they started dancing like they should so, the slot isn't dead and works fine other than anything associated with these drives. Go figure.
Boot to the desktop and open "disks" and sure as S*&^ it shows up in the list.
Changed my mount points on the M2 to backup and put the U3 on Storage by UUID as this avoids the renaming complex Linux has when booting. Reboot and verify.
Things up, mount, etc.

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme2n1     259:0    0 931.5G  0 disk
└─nvme2n1p1 259:1    0 931.5G  0 part /mnt/backup
nvme1n1     259:2    0    14T  0 disk /mnt/Storage
nvme0n1     259:3    0 931.5G  0 disk
├─nvme0n1p1 259:4    0     1G  0 part /boot/efi
└─nvme0n1p2 259:5    0 930.5G  0 part /

OS - 09:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01)
U3 - 0a:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller CD8 (rev 01)
M2 / backup - 0c:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01)

Now, to see how this one works AFA lasting more than a week. Monitoring temps a bit closer as well due to mentions of high temps on the Micron's someone else added to their system. Right now though it's sitting at 38C which is in line with the 2 x M2's in the system currently.

Right now it's just sitting in the case not actually mounted though and will need to figure that out once it lives longer than the return period and proves itself. So, the other thing that's going to make my OCD go haywire is the slot issue with the X16 G5 being flakey. No real reason it shouldn't be able to work with the adapter / cable / drive. The other option is to switch the drive adapter to an M2 format and push the NIC back down to the original slot. Either way both options result in the drive having G4 x4 speeds.

So it sounds like it wasn't the Micron drives that were the problem, but maybe the previous adapter(s)? Were you able to test out the Micron drives again, or did you already return them? Anyway, sounds like you got it working for now. What motherboard btw?

 

[Tech Junky]

@Shmee

Well, it's been a daunting project so far with 3 different setups but with these types of drives it's all been on a z690 steel legend. 2 different cards with the drive mounted on them in the same slot each time. Third card being the current one in a different slot now.

So, here's what I ended up with:
Card - https://a.co/d/etr1u9U
Cable - https://a.co/d/4TLR3Rt
Drive - https://www.ebay.com/itm/204388229185

I sent the Micron's back when they lost the ability to store data. Unfortunately they didn't work in a different system so, moving the card to the other slot likely wouldn't have resolved the issue. However, I also changed my fstab mappings from /dev/xyz to /dev/UUID as Linux likes to renumber things on each boot.

I thought the first failure was a fluke and the second one lasting nearly a week looked promising until it did the same thing. The second one was just idle though overnight and then it wouldn't R/W data but listed it and upon reboot disappeared like the first one. So, drives / adapter / slot? Who knows but, the slot works fine with a quad port NIC I've been using for quite awhile in a different slot. There must be something with how the slot communicates with these drives vs a NIC or some other device.

I'm contemplating going forward with my AMD rebuild idea though now with discovery of the issue with the slot not being 100%. Still not enough to bite the bullet on spending another $800 but, I have all the parts spec'd out for it if need be and can get them next day.

The other thing is these drives are finicky with the cable length due to the signal attenuation is my guess to maintain the speed. The cable issue wasn't a known thing when starting out though but, figured the shorter the connection traces the better the performance hence going with the PCB mount vs cable originally. While waiting on things to show up on the 3rd attempt I went digging around looking for any gotcha points people have posted over the years and that was one of them. Unfortunately it's impossible to find a ready made cable that's short than 50CM at a decent price or I would have opted for something shorter for the future when G5 drives are more common and the price comes down on them.

The only thing left to consider is using a M2 adapter instead of the card but, not sure it's worth the hassle at this point to experiment with it while it's working just fine. Time will tell though if this is goin to be the final solution for the time being or if things will go wonky again and have to rethink a ground up rebuild to resolve it.

Drive temp is hovering high 30's / low 40's though which is where it should be most of the time. Since it's considerably faster than the spinners in raid copying data shouldn't tax it too hard and bump the temps to unreasonable levels. If it's stable I might might repartition it and put the OS on there as well to simplify things even further. Can still backup the OS to the M2 and get a little bit of a speed upgrade from the SN850 to this beast. Just have a few other ideas bouncing around before locking it down as complete.

 

[thigobr]

What about power consumption? Do these U2/U3 drives have proper ASPM going into low power states?

 

[Tech Junky]

What about power consumption? Do these U2/U3 drives have proper ASPM going into low power states?

Not sure about that particular parameter but, the Kioxia I have now is showing this for the SMART output.

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W   25.00W       -    0  0  0  0   500000  500000
 1 +    12.00W   12.00W       -    1  1  1  1   500000  500000

They're going to be using a bit more power than some other options. The Micron's though had more power profiles showing in the output but, I don't have a capture of that since they've been returned already.

 

[Singularity9]

Forgive me as I am new to all this

Is there a consensus on the best method to connect 2 u.2 ssds at full gen4 speed? So far it looks like im going with this card: https://www.amazon.com/gp/product/B0BRV7RQLK/ref=ox_sc_act_title_2?smid=A2OKSEIX7OEK0B&psc=1
and this cable: https://www.amazon.com/gp/product/B0BMQ1L9T4/ref=ewc_pr_img_1?smid=A2OKSEIX7OEK0B&psc=1
My Motherboard supports 4x4x4x4 bifurcation.

Seems these are quite expensive in comparison to some other adapters. Is this the most optimal way to do it?

 

[igor_kavinski]

So far it looks like im going with this card: https://www.amazon.com/gp/product/B0BRV7RQLK/ref=ox_sc_act_title_2?smid=A2OKSEIX7OEK0B&psc=1

I have no experience with this thing but any time I see VROC, the only thing I can think of is bad news and stay the hell away! VROC requires a compatible mobo and you have to pay an Intel license fee. I don't know much about it other than that.

 

[Singularity9]

I have no experience with this thing but any time I see VROC, the only thing I can think of is bad news and stay the hell away! VROC requires a compatible mobo and you have to pay an Intel license fee. I don't know much about it other than that.

Guess that card is out. Hope i can find an alternative. I was pretty hopeful as the only amazon review for it verifies that it works for my exact use case.

 

[igor_kavinski]

I was pretty hopeful as the only amazon review for it verifies that it works for my exact use case.

No wonder it worked fine with Threadripper. AMD isn't as evil as Intel.

 

[Singularity9]

No wonder it worked fine with Threadripper. AMD isn't as evil as Intel.

My motherboard is x570 using a 5600x so perhaps it would still work for me? I just want to really be sure before I get this stuff. Also found something like this: https://www.amazon.com/Septpenta-Fu...e6-99a5-67937da79f51&pd_rd_i=B0BHWQ1RS6&psc=1.

 

[igor_kavinski]

Also found something like this: https://www.amazon.com/Septpenta-Function-Indicator-Lossless-Transmission/dp/B0BHWQ1RS6

That could work with this cable: https://www.microsatacables.com/u-2-to-internal-minisas-hd-4x-sff-8673-cable-l-100cm-pcie-4-0

But pretty expensive for a cable.