unable to bind Mellanox device to DPDK compatible driver

pband2 · ‎04-19-2019

Hi All,

I am trying to compile DPDK with Mellanox driver support and test pktgen on Ubuntu 18.04/16.04

on Azure. I have created the VM Ubuntu 18.04/16.04 with two interfaces with accelerated networking

enabled. I have compiled DPDK with MLX4/5 enabled successfully followed by PKTGEN with appropriate

targets. However when the Mellanox interfaces are bound to mlx4_core using dpdk-devbind.py

the interfaces doesn't show up as DPDK network devices and rather show up as using Network Devices.

However when the interfaces are bound to igb_uio driver the interfaces show up as DPDK compatible

network devices.

DPDK version used: dpdk-19.02

PKTGEN: pktgen-3.6.5

After installing all dependencies, DPDK is compiled with the following steps

Make config T=x86_64-native-linuxapp-gcc

To enable MLX driver:

sed -ri 's,(MLX._PMD=)n,\1y,' build/.config.

and

make install T=x86_64-native-linuxapp-gcc DESTDIR=<PKTGEN_DIR>

Please note that I have installed libibverbs and have not installed full version of OFED i.e. MLNX_OFED_LINUX.

Installation in /home/<>/dpdk-19.02// complete

Similarly PKTGEN is also compiled and built with the already compiled DPDK SDK.

What I notice is that these interfaces show up as DPDK compatible

interfaces when bound to igb_uio but when bound to mlx4_uio, they show up as Kernel

Output of dpdk-devbind.py with -b with igb_uio and mlx4_core

When bound to igb_uio

Network devices using DPDK-compatible driver

============================================

0001:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' drv=igb_uio unused=

0002:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' drv=igb_uio unused=

0003:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' drv=igb_uio unused=

When bound to mlx4_core

Network devices using kernel driver

===================================

0001:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' if=rename20 drv=mlx4_core unused=igb_uio

0002:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' if=enP2s2 drv=mlx4_core unused=igb_uio

0003:00:02.0 'MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004' if=enP3s3 drv=mlx4_core unused=igb_uio

Please note that the devices are using kernel driver and not DPDK MLX driver.

~/dpdk-19.02$ lspci

0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)

0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)

0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)

0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)

0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA

0001:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

0002:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

0003:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

dpdk-19.02$ ifconfig -a

enP1p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:08:6b

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:9773 errors:0 dropped:0 overruns:0 frame:0

TX packets:15939 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:7116204 (7.1 MB) TX bytes:3611453 (3.6 MB)

enP2p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:f5:2c

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:15 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:1690 (1.6 KB)

enP3p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:f0:7c

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:15 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:1690 (1.6 KB)

eth0 Link encap:Ethernet HWaddr 00:0d:3a:f2:08:6b

inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0

inet6 addr: fe80::20d:3aff:fef2:86b/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:11519 errors:0 dropped:0 overruns:0 frame:0

TX packets:15414 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:8502266 (8.5 MB) TX bytes:3579927 (3.5 MB)

eth1 Link encap:Ethernet HWaddr 00:0d:3a:f2:f5:2c

inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0

inet6 addr: fe80::20d:3aff:fef2:f52c/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:2 errors:0 dropped:0 overruns:0 frame:0

TX packets:15 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:762 (762.0 B) TX bytes:1690 (1.6 KB)

Do I need to do anything else to get this working? I mean bind Mellanox interfaces

As DPDK compatible interfaces? Packet generation is not working which I think is due

To this issue. In summary while the compilation goes through, DPDK MLX drivers doesn’t

Seem to get loaded and attach to the interfaces.

I have referred to https://doc.dpdk.org/guides/nics/mlx4.html as well while working on compiling DPDK.

Other thing I would like to know is if installing MLNX_OFED_LINUX is a must in this environment. I am facing

Issues while installing OFED drivers but DPDK compilation seems to go through fine with just libibverbs. Please

Let me know if you would like to know info. Any help will be greatly appreciated.

pband2 · ‎04-21-2019

I have a similar issue on CentOs as well ( on Azure ). Post compilation I see that interfaces are using mlx4_core drivers but am unable to bind the

interfaces to DPDK drivers. However this time I get an error that DPDK drivers are not at all loaded. Seems like I am missing something basic.

[root@phanidpdkcentos usertools]# ./dpdk-devbind.py -b igb_uio 0002:00:02.0

Error - no supported modules(DPDK driver) are loaded

Any hints at what probably is missing?

I am following the instruction from

https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk and

https://doc.dpdk.org/guides/nics/mlx4.html

for compilation

[root@phanidpdkcentos usertools]# lspci -s 0002:00:02.0 -vv

0002:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

Subsystem: Mellanox Technologies Device 61b0

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0

Region 2: Memory at fe0800000 (64-bit, prefetchable) [size=8M]

Capabilities: [60] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset+ SlotPowerLimit 0.000W

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-

MaxPayload 128 bytes, MaxReadReq 128 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <64ns, L1 <1us

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [9c] MSI-X: Enable+ Count=24 Masked-

Vector table: BAR=2 offset=00002000

PBA: BAR=2 offset=00003000

Capabilities: [40] Power Management version 0

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Kernel driver in use: mlx4_core

Kernel modules: mlx4_core