Adding Mellanox Connectx3 Dual Port 40GbE and Intel Pro/1000 VT Quad Gigabit Nic to Proxmox thru IOMMU & SRIOV

The focus of this guide is to install a Mellanox Connectx3 40GbE and a Intel Pro/1000 VT [ET2] quad nic to a Proxmox install, making them available for Ubuntu VM’s within Proxmox. The difficulty many have with this install can likely be traced to certain chipsets, specifically Intel Z170/270/370 implementation, where the CPU’s PCI-e slots share 1 and only one IOMMU group, which makes splitting certain hardware to share between the host and VM very difficult, if not impossible. I hope this guide helps others who are setting a homelab on consumer hardware.

Hardware specs for reference:
CPU – i5 8400 [6-core]
Mobo – MSI Z370 SLI Plus

1. Prerequisites, on system BIOS
   a. Virtual Technology – Enabled
   b. VT-d – Enabled
   c. Above Gen4 Encoding – Enabled
   
2. Install Proxmox, open terminal, update and reboot:
   a. Disable commercial repo:
   $ sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/pve-enterprise.list
   b. Add PVE Community Repo and upgrade:
   $ echo "deb http://download.proxmox.com/debian/pve $(grep "VERSION=" /etc/os-release | sed -n 's/.((.)).*/\1/p') pve-no-subscription" > /etc/apt/sources.list.d/pve-no-enterprise.list
$ apt update
$ apt upgrade
$ reboot
   
3. Edit /etc/kernel/cmdline:
   $ nano /etc/kernel/cmdline
   And append the line to:
   root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt
   Then Ctrl-O to write out [save] the file and Ctrl-X to exit nano.
   
4. Configure grub and update:
   $ nano /etc/default/grub
   And append the line to:
   GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
   Then Ctrl-O to write out [save] the file and Ctrl-X to exit nano.
   And update grub:
   $ update-grub
   
5. Edit /etc/modules:
   $ nano /etc/modules
   Add the following:
   vfio
vfio_iommu type1
vfio_pci
vfio_virqfd
   Then Ctrl-O to write out [save] the file and Ctrl-X to exit nano.
   
6. Update the initramfs and reboot [you must repeat this command AFTER everytime you update grub]:
   $ update-initramfs -u -k all
$ reboot
   
7. Download prerequisites and install latest MFT [Mellanox utility]:
   $ apt-get install infiniband-diags
$ apt install gcc dkms
$ apt-get install pve-headers-5.11.22-7-pve [for this, use $ uname -r to confirm your PVE version]
$ wget https://www.mellanox.com/downloads/MFT/mft-4.17.0-106-x86_64-deb.tgz
$ tar -xvf mft-4.17.0-106-x86_64-deb.tgz
$ cd mft-4.17.0-106-x86_64-deb/
$ ./install.sh
   
8. Configure the Connectx3 for SRIOV:
   $ mst start
$ mst status
$ mlxconfig -d /dev/mst/mt4099_pciconf0 q
$ mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4
$ reboot
   
9. Check status:
   $ ibstat
   
10. Edit /etc/modprobe.d/mlx4_core.conf to configure VFS and ports:
    $ nano /etc/modprobe.d/mlx4_core.conf
    Add this line:
    options mlx4_core num_vfs=2,2,0 port_type_array=2,2 probe_vf=2,2,0
    Then Ctrl-O to write out [save] the file and Ctrl-X to exit nano.
    $ update-initramfs -u -k all
$ reboot

At this point, you will be able to add your Mellanox Connectx3 to the PC. I recommend using the lowest/last PCI-e slot that is controlled by the chipset, as this slot will break out the dual port into 2 VFS for each physical port. You can now “Add Hardware” to a VM as a “PCI Device” with the following options:
– All Functions – Unchecked
– ROM-Bar – Unchecked
– Primary GPU – Unchecked

With regards to the Intel quad nic, the above procedure ensured that the quad nic is also activated with its own IOMMU group and assignable thru SRIOV. BUT since the chipset assigns a single IOMMU group to PCI-e devices above the chipset PCI-e slots, I was not successful in assigning any of the quad ports thru SRIOV to the VM’s. Instead, these Intel PRO/1000 VT/ET2 cards are made for virtualization, so I was able to assign a “Linux Bridge” to each port and then “Add Hardware” to assign a “Network Device” for the virtualized ethernet port [vmbr] to the VM and achieve full line speeds.

References:
1. PVE 5.0 beta 2, Mellanox Connectx-3, and SR-IOV – Proxmox Forum
2. Separated networks, two NIC, two vmbr – Proxmox Forum
3. PROXMOX (Debian 10, KVM) enabling SR-IOV for Mellanox Infiniband cards – khmel.org
4. Enable Proxmox PCIe Passthrough – Thomas Krenn
5. 40gbs Mellanox Infinityband – Proxmox Forum
6. Mellanox Firmware Tools (MFT)
7. Getting started with Mellanox Firmware tools (MFT) for Linux
8. Mellanox ConnectX-3 VFs stuck in same SR-IOV group – Proxmox Forum
9. Single Root IO Virtualization (SR-IOV)
10. Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware – ServeTheHome Forum
11. Firmware Update Instructions – Mellanox/nVidia
12. New Proxmox build- no iommu detected – Proxmox Forum

Liked it? Take a second to support RedEyeNinja on Patreon!