Going from 5GbE to 40GbE with my homelab
Right now I have my all-flash NAS connected directly to my PC via 5gbe ethernet link. It's actually not the fastest link, since my NAS can do 7GB/s+ speed easily. I'm currently using Realtek 8126 NIC on both ends. I was thinking of getting a 10gbe NIC (either RJ45 or SFP+ would be fine) so I can have faster speed. So I continued researching...
The first thing I check is to see if there is cheap 10gbe NIC for sale here. I found the one that's available here is either Intel X540 T2 10G with RJ45 port ($20), or INTEL X520 DA2 with SFP+ ($20). If I choose the former, I will not need to change any cabling at all, since I already use cat 6a and it's just 15 meters. If I choose the latter, I would need to buy 10gb SFP+ and a fiber patch cord, which would add more cost.
In all curiousness, I also checked if there is 25gb or 40gb NIC available for a reasonable price. There were some potential NIC I can use, for example this Mellanox ConnectX-4 25GbE Dual Port SFP28 CX4121A, which uses SFP248 transceiver.
It's a good card, however the price tag is not good. It cost around $100 for the NIC alone, which is out the question. I need to buy 2 of these, and also 2 transceiver, so I eliminate 25GbE from the list. That leave only 10GbE and 40GbE.
As for 40gb, I found that there is Mellanox ConnectX-3 40G QSFP (HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP adapter) for $30. However, this card use a FLOM to PCIE adapter, which I don't like. It adds another point of failure. I searched for other Mellanox card, and found a candidate.
Mellanox CX354A QCBT, listed by the seller as 10GbE NIC and just for $20. It's weird because why this card has QSFP+ port but only 10GbE support? I searched to why this card only has 10GbE, and found some interesting post on STH.
It turns out, you can cross-flash this card from QCBT (40Gb/s IB / 10GbE) to FCBT variant (40Gb/s IB / 40GbE) NIC ! I took the bullet and bought the card.
As for the transceiver, I searched what kind of transceiver this card supports. There is some clue from STH that the card only supports up to 1.5w QSFP+.
From this post, Mellanox ConnectX-3 card only supports up to 1.5w QSFP+
This limit myself in finding the transceiver and I need to look at the datasheet before buying. I just searched Tokopedia for the cheapest QSFP+ transceiver I could find and read the specsheet. After some hours, I found this Finisar FTL410QD3C QSFP 40GBASE SR4. It only needs 1.2W and it does 40GbE.
Note that it has MPO connector, so I need to find a suitable cable with MPO connectors. There is a very good detail on MPO connector available on fs.com
Moris
After reading that for a while, I went with a 12-core MPO patch cable with 15 meters long for $10. I waited for all the ingredients to arrive.