Cross-flash and Using Mellanox ConnectX-3 CX354A 40GbE NIC on my Proxmox VE 8 homelab
Earlier I posted about how I'm going to go from 5GbE to 40GbE. I went to Malaysia shortly after posted that so I was not able to do the testing. Fast forward 7 days later, I'm back home and everything arrived.
I plug the Mellanox card into my Windows 11 main PC and tried to crossflash it. As per STH forum, I downloaded the exact version of MFT 4.18.0 (Mellanox Flash Tool) to make sure I can use it on older card from https://network.nvidia.com/products/adapter-software/firmware-tools/.
First I checked to see if the card is detected.
# Start Mellanox Flast Tool Service
mst start
# Check available Mellanox card in the system
mst status
Since the card is being detected, I got the PCIE address which is mt4099_pci_cr0. I downloaded the firmware for MCX354A-FCBT (the one I need to crossflash to) from https://network.nvidia.com/support/firmware/connectx3ib/.
I then continue to backup the original firmware, original rom, original product id.
# Backup the original firmware
flint -d mt4099_pci_cr0 ri orig_firmware.bin
# Backup original firmware config
flint -d mt4099_pci_cr0 dc orig_firmware.ini
# Backup original ROM
flint -d mt4099_pci_cr0 rrom orig_rom.bin
# Backup original product data
mlxburn -d mt4099_pci_cr0 -vpd > orig_vpd.txtThe command used along with explanation
Once I've done all of that, I proceed to flash the card with the new firmware, with -allow_psid_change to allow the flasher to flash the card to different psid firmware, basically forcing it to accept me transforming itself from QCBT to FCBT.
# FLash the card with FCBT .bin with allow_psid_change to allow flashing using mismatched PSID (Firmware ID)
flint -d mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn
Once done I checked to see if the firmware was updated correctly.
# Check card detail, such as SR-IOV, link status (Infiniband or Ethernet)), etc
mlxconfig -d mt4099_pci_cr0 query
# Check card detail, such as MAC address, Firmware version, etc
mlxfwmanager --query
At this point, the card is ready to use. I plug the card into my proxmox node, and found it as ens1 and ens1d1. I checked it using ethtool to see information about the card.
As you can see, the card fully supports 10Gb, 40Gb, and 56Gb speed. Although 10 and 40Gb/s are easily achieveable, the 56Gb/s only achieveable to use with Mellanox own DAC. Note that proxmox sometimes still loads mlnx4_ib driver, which is used for infiniband. You need to blacklist it and modprobe mlnx4_en if you're going to use it on ethernet mode.
By the way, I also flashed another card to use on my main PC, with only PCIE 4.0 x4 bandwidth. Although I have PCIE 4.0 slot, the mellanox card only use PCIE 3.0, so I'm stuck with PCIE 3.0 x4 bandwidth, with theoretical speed up to 28Gb/s. Still a big jump over my 5GbE.
I moved the QSFP transceiver to the 2nd port on the proxmox node, and I plug the other end to my PC. This time it didn't work, I checkec there was no light from my Main PC NIC. I thought this maybe just a driver issue, because the card is being detected as IPoIB on windows device manager.
I checked NVIDIA website for the driver (Mellanox was bought by NVIDIA). The ConnectX-3 card uses WinOF and not WinOF-2 driver.
I downloaded and installed WinOF driver for my Main PC. After driver installation, I re-plugged the transceiver, and the link led is turned on, which means the card is now connected with my proxmox NIC.
I checked the link speed on my proxmox host to see thecurrent link speed.
The link speed shows up as 40000Mb/s, so it recongnized the max transceiver speed. I checked my main PC ethernet status as well.
Now both my Main PC and Proxmox Host connected through 40GbE link ! Next up is testing either TrueNAS or UnRAID on this proxmox as VM, as I need them to store my data in RAID 5/ RAIDZ1, as well as storing my CCTV footage.