Affordable 10Gbit copper nics?

[boed]

Hello,

I was looking a the X540-T1 but was wondering if there were cheaper cards I should consider. I have over 20TB of data I need to backup on a regular basis. My built in nics on my two systems are crap. I've gotten much better throughput when using real 1Gbit cards. These are fast drives on a fast lsi controller so I know they can go well over 1Gbit in transfer speeds although I realize they won't come close to 10Gbit.

 

[Pandasaurus]

At the moment, I think the only way you'll get "10Gbit" and "affordable" to appear in the same sentence is to include "Infiniband" in the same sentence. CX4 10Gb HCA's can be had on eBay for <$60, 10Gb switches for <$150. 10GbE? Not so much.

Edit: If this is strictly for storage, you might also consider Fibre Channel. 4Gb FC is quite affordable these days.

Edit 2: Just checked eBay again, looks like pricing for Infiniband has changed somewhat. Switches are around $200 now, but HCA's are $30-40 for PCIe.

 

[Hanpan]

I would go fiber instead. (Please note am not affiliated in any way with any of the links below)

First, get yourself 2 brocade 1020.

I was able to get two for $56 shipped from http://www.ebay.com/itm/171446971447?_trksid=p2059210.m2749.l2648&ssPageName=STRK:MEBIDX:IT

Now if you just want to go direct between two pc's you can simply get an active twinax cable and you are done.

For example:

http://www.ebay.com/itm/BROCADE-10G...055?pt=LH_DefaultDomain_0&hash=item3f40097c0f

Or, you can get fiber lamps. These are reported to work well:

http://www.fiberstore.com/new-broca...p-850nm-300m-optical-transceiver-p-31443.html

https://forums.servethehome.com/ind...0-cna-10gbe-pcie-cards.2955/page-4#post-31135

You'll need some fiber. I would suggest OM3 such as:

http://www.fiberstore.com/lc-lc-duplex-10g-om3-50-125-multimode-fiber-patch-cable-p-12018.html

Finally, if you want a switch,

try:

http://www.ebay.com/itm/Quanta-LB4M...38?pt=US_Network_Switches&hash=item3f3ff8ad42 (Might be loud)

or

http://www.ebay.com/itm/181364813454

The first is reported working with the brocade cards:

https://forums.servethehome.com/ind...0-cna-10gbe-pcie-cards.2955/page-5#post-31247

The second however I have not found anyone using the setup, though it should work...

 

[XavierMace]

Edit: Nevermind, just read the thread you linked to. Definitely economical, but looks to be extremely picky on hardware. I'd be hesitant to go with anything that picky.

 

[Hanpan]

I would definitely look at infiniband then. Are you working in a switched environment, or direct connect?

 

[azazel1024]

What is the distance you are looking at and can it be direct connected, networked or not?

Twinax 10GbE is fairly inexpensive. You can generally find a cable and pair of GBIC+ for around $100. Just need a NIC that can take GBIC+ then, which are typically around $50-80 a pop. The switch side of things might be a little more expensive, but you can find bare GBIC+ switches for not too much.

The limitation is going to be 10 meters maximum length.

 

[azazel1024]

Oh, also what operating system? Windows 8, 8.1 and server 2012 can all do SMB3.0 (or 3.01) and SMB multichannel. This means that the OS will "bundle" multiple NICs of the same speed for faster transfers. Only works between the operating systems, the SMB3.0 specs do NOT have SMB Multichannel, it is a proprietary Microsoft extension to SMB (doesn't mean no one could implement it, AFAIK, but no one who HAS implemented SMB3.0 has SMB Multichannel right now, other than Microsoft).

Advantages are, you don't need need to do NIC teaming (actually, you can't and get SMB Multichannel to work) and it works over dumb switches. I have a pair of Intel Gigabit CT adapters in my server and desktop and can get 235-237MB/sec transfer speeds when I am not disk bound (windows 8 server, windows 8.1 desktop).

A cheap used Intel quad port gigabit NIC in both machines might be the way to go. In theory you are talking 4Gbps transfer speeds then, and I am guessing that is roughly the performance you could get out of the disk arrays (unless that is a BIG array, like 6+ disks in RAID5 as most RAID5 is very fast read, but a lot slower write speeds).

One final question, do you have to transfer all 20TB at a time? Or are you just talking it is 20TB of data, and what has to be backed up of that is a resonable fraction of 20TB? Cause if you are backing up all of it each time, why not do versioning or check only for updated data (like using Synctoy or similar backup tools)?

 

[boed]

Thanks for all the good questions and the helpful links on the fiber. Wouldn't I also have to buy transceivers for those cards? I'm not really up to speed on fiber solutions.

To save money I was going to go direct connected. - About 12' apart. I'm in an area where things are constantly moving about so I was trying to avoid fiber as it can be a bit more delicate than copper. The onboard nic would be fine for internet and the dedicated direct connect would just be for transfers between machines.

I do have onboard 2 nics in my PC as well as an intel CT card (which is not supported for Windows 8 really - it does work but no intel drivers). Not sure if I can do the SMB solution without a switch that supports some sort of link aggregation.

I often transfer 2 TB at a time (every other day) - today I'm restoring (I just expanded the array) so I need to restore 25TB.

I'm running Windows 8.1 x64 enterprise.

That's a good idea bout the smb multichannel - I'll start a new thread on this as this might be a good interim solution.

I'm not familiar with infiniband. Any additional links to a proposed solution would be helpful.

 

[azazel1024]

If you look, what I posted is not fiber. It is twinaux. It is coaxial cable, but with coax inside coax (hence, twinaux). Twinaux is a reciever on each end + cable, for around $100. Some inexpensive SFP+ cards that can take the transcievers are going to be around $70-100 each.

 

[boed]

Thanks - so if I order these http://www.ebay.com/itm/171446971447...%3AMEBIDX%3AIT - all I need is a cable? http://www.ebay.com/itm/BROCADE-10G-...item3f40097c0f - the one that expired? Just those 2 things - no fiber or switches or transceivers - just those two cards with that cable in between?

 

[alkemyst]

Look at differential backups too. No reason to do a full 20TB backup at a time.

 

[boed]

I usually only have to backup about 2TB at a time (changed data). I use freefilesync which is wonderful to backup what has changed.

There is a need to restore 25TB when I reformatted the drive and there is nothing on it though. I'm restoring 25TB now on the newly expanded array and I'm about to do the same in the opposite direction next week.

 

[alkemyst]

Quick solution is to mount that drive in the other device.

 

[boed]

Thanks - I could not easily mount an array with 8 drives attached that is inside one PC inside another PC that already has 8 drives in an array.

 

[alkemyst]

Thanks - so if I order these http://www.ebay.com/itm/171446971447...%3AMEBIDX%3AIT - all I need is a cable? http://www.ebay.com/itm/BROCADE-10G-...item3f40097c0f - the one that expired? Just those 2 things - no fiber or switches or transceivers - just those two cards with that cable in between?

You will need those cards, the 10G transceivers and the matching cable between them.

Also the drivers for your OS/machine.

 

[boed]

I would go fiber instead. (Please note am not affiliated in any way with any of the links below)

First, get yourself 2 brocade 1020.

I was able to get two for $56 shipped from http://www.ebay.com/itm/171446971447?_trksid=p2059210.m2749.l2648&ssPageName=STRK:MEBIDX:IT

Now if you just want to go direct between two pc's you can simply get an active twinax cable and you are done.

For example:

http://www.ebay.com/itm/BROCADE-10G...055?pt=LH_DefaultDomain_0&hash=item3f40097c0f

Or, you can get fiber lamps. These are reported to work well:

http://www.fiberstore.com/new-broca...p-850nm-300m-optical-transceiver-p-31443.html

https://forums.servethehome.com/ind...0-cna-10gbe-pcie-cards.2955/page-4#post-31135

You'll need some fiber. I would suggest OM3 such as:

http://www.fiberstore.com/lc-lc-duplex-10g-om3-50-125-multimode-fiber-patch-cable-p-12018.html

Finally, if you want a switch,

try:

http://www.ebay.com/itm/Quanta-LB4M...38?pt=US_Network_Switches&hash=item3f3ff8ad42 (Might be loud)

or

http://www.ebay.com/itm/181364813454

The first is reported working with the brocade cards:

https://forums.servethehome.com/ind...0-cna-10gbe-pcie-cards.2955/page-5#post-31247

The second however I have not found anyone using the setup, though it should work...

Thanks, if I just want to connect the cards direct if I get the cards in the first link, the cable in the second. What other thing do I need if I don't want a hub/switch? Can you like the transceiver if it they are necessary and the is the fiber necessary in addition? Sorry for the multiple questions but I'm not familiar with this solution.

 

[boed]

You will need those cards, the 10G transceivers and the matching cable between them.

Also the drivers for your OS/machine.

Thanks - I'm not sure who is right as I'm not familiar with this tech. THe original post said I just needed the cards and the cables. I agree I'll need drivers but I'm more concerned about what hardware I need right now.

 

[alkemyst]

The cards don't work by themselves. The SFP/Transceivers need to be installed. They do 10G, but I am almost positive 1G modules can be installed too.

You then need the proper fiber (OM3 would be it) to connect them. You don't need a switch really.

A switch would be needed if you were wanting to network more than two devices. No one really uses hubs anymore.


Basically the card goes into the computer, the transceiver into the card and the fiber into the transceiver at both ends.

 

[master_shake_]

infiniband is cheaper.



a lot cheaper.

i run a 4 computer infiniband network and a switch

i think i am about 400 in everything.

 

[alkemyst]

infiniband is cheaper.



a lot cheaper.

i run a 4 computer infiniband network and a switch

i think i am about 400 in everything.

There can be issues with Infiniband, it can be a good solution too.

http://www.vladan.fr/homelab-storage-network-speedup/

 

[master_shake_]

i wouldn't touch the mellanox drivers.

open fabric or nothing.

but for some reason anything over 3.1 won't work very odd.

 

[boed]

If you read the last post on this page - apparently there are two types - one uses the fiber and transceivers and the other uses just the twinax cable
https://forums.servethehome.com/ind...0-cna-10gbe-pcie-cards.2955/page-5#post-31247

 

[XavierMace]

The Twinax cable originally linked to has transceivers on it already.

 

[XavierMace]

There can be issues with Infiniband, it can be a good solution too.

http://www.vladan.fr/homelab-storage-network-speedup/

That's because his setup is a complete hack job. My Fibre Channel setup was cheaper than his Infiniband setup and I didn't have to jump through hoops. No driver installs, just install the adapter, rescan storage. His setup has the added "bonus" of potentially breaking with an ESXI update as the tweaks may no longer function. It's the same with tweaking ESXI to run OSX on non-Apple hardware. You can't update ESXI as it breaks the tweaks. My Fiber Setup:

Brocade 4100 32 port 4Gb FC Switch - $100
Pair of QLogic 2464 Quad Port 4Gb HBA - $45
8x 2m LC/LC Fiber Cables - $80

Enable, Multipathing, ta-da. 16Gb/s between host and storage. No driver installs or hacks. Each additional host adds $60, can run 7 hosts, each with 16Gb/s with that setup.

 

[master_shake_]

That's because his setup is a complete hack job. My Fibre Channel setup was cheaper than his Infiniband setup and I didn't have to jump through hoops. No driver installs, just install the adapter, rescan storage. His setup has the added "bonus" of potentially breaking with an ESXI update as the tweaks may no longer function. It's the same with tweaking ESXI to run OSX on non-Apple hardware. You can't update ESXI as it breaks the tweaks. My Fiber Setup:

Brocade 4100 32 port 4Gb FC Switch - $100
Pair of QLogic 2464 Quad Port 4Gb HBA - $45
8x 2m LC/LC Fiber Cables - $80

Enable, Multipathing, ta-da. 16Gb/s between host and storage. No driver installs or hacks. Each additional host adds $60, can run 7 hosts, each with 16Gb/s with that setup.

really?

installing a driver and open sm on one computer is soooo hard.