SR-IOV failed on Intel Xeon-D 1541's X552 10gbe NIC

[CHsia9]

Hi all,

I cannot figure out why I cannot enable SR-IOV on Intel Xeon-D 1541's X552 10gbe NIC, it must be the intel's latest ixgbe driver issue because on the same SoC board, the Intel i350 1gbe NIC's sr-iov can be enabled.

Following is the pci device info and also its ixgbe info

mailto:root@pve1:/sys/bus/pci/devices/0000:03:00.1# root@pve1:/sys/bus/pci/devices/0000:03:00.1# lspci -vnnk -s 03:00.0

03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]

Subsystem: Super Micro Computer Inc Device [15d9:15ad]

Physical Slot: 0-1

Flags: bus master, fast devsel, latency 0, IRQ 25

Memory at fbc00000 (64-bit, prefetchable) [size=2M]

Memory at fbe04000 (64-bit, prefetchable) [size=16K]

Expansion ROM at 90100000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] MSI-X: Enable+ Count=64 Masked-

Capabilities: [a0] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

Capabilities: [1b0] Access Control Services

Capabilities: [1c0] Latency Tolerance Reporting

Kernel driver in use: ixgbe

mailto:root@pve1:/sys/bus/pci/devices/0000:03:00.1# root@pve1:/sys/bus/pci/devices/0000:03:00.1# modinfo ixgbe

filename: /lib/modules/4.2.8-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

version: 4.1.5

license: GPL

description: Intel(R) 10 Gigabit PCI Express Network Driver

author: Intel Corporation, mailto:linux.nics@intel.com

srcversion: 9781CEF8A3110F93FF9DBA8

alias: pci:v00008086d000015ADsv*sd*bc*sc*i*

alias: pci:v00008086d00001560sv*sd*bc*sc*i*

alias: pci:v00008086d00001558sv*sd*bc*sc*i*

alias: pci:v00008086d0000154Asv*sd*bc*sc*i*

alias: pci:v00008086d00001557sv*sd*bc*sc*i*

alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*

alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*

alias: pci:v00008086d00001528sv*sd*bc*sc*i*

alias: pci:v00008086d000010F8sv*sd*bc*sc*i*

alias: pci:v00008086d0000151Csv*sd*bc*sc*i*

alias: pci:v00008086d00001529sv*sd*bc*sc*i*

alias: pci:v00008086d0000152Asv*sd*bc*sc*i*

alias: pci:v00008086d000010F9sv*sd*bc*sc*i*

alias: pci:v00008086d00001514sv*sd*bc*sc*i*

alias: pci:v00008086d00001507sv*sd*bc*sc*i*

alias: pci:v00008086d000010FBsv*sd*bc*sc*i*

alias: pci:v00008086d00001517sv*sd*bc*sc*i*

alias: pci:v00008086d000010FCsv*sd*bc*sc*i*

alias: pci:v00008086d000010F7sv*sd*bc*sc*i*

alias: pci:v00008086d00001508sv*sd*bc*sc*i*

alias: pci:v00008086d000010DBsv*sd*bc*sc*i*

alias: pci:v00008086d000010F4sv*sd*bc*sc*i*

alias: pci:v00008086d000010E1sv*sd*bc*sc*i*

alias: pci:v00008086d000010F1sv*sd*bc*sc*i*

alias: pci:v00008086d000010ECsv*sd*bc*sc*i*

alias: pci:v00008086d000010DDsv*sd*bc*sc*i*

alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*

alias: pci:v00008086d000010C8sv*sd*bc*sc*i*

alias: pci:v00008086d000010C7sv*sd*bc*sc*i*

alias: pci:v00008086d000010C6sv*sd*bc*sc*i*

alias: pci:v00008086d000010B6sv*sd*bc*sc*i*

depends: ptp,dca,vxlan

vermagic: 4.2.8-1-pve SMP mod_unload modversions

parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)

parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)

parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)

parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)

parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)

parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)

parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)

parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)

parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)

parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)

parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)

parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)

parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)

parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)

parm: FdirPballoc:Flow Director packet buffer allocation level:

1 = 8k hash filters or 2k perfect filters

2 = 16k hash filters or 4k perfect filters

3 = 32k hash filters or 8k perfect filters (array of int)

parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)

parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)

parm: LRO:Large Receive Offload (0,1), default 1 = on (array of int)

parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)

parm: dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)

parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

[st4]

Hi EluRex,

Thank you for the post. Let me check on this.

rgds,

wb

[st4]

Hi EluRex,

Did you resolve the issue by updating the driver?

Rgds,

wb

[CHsia9]

Hi wb_intel,

the problem is not solved with the latest driver.

ixgbe driver version 4.3.13

PCIe Device Info and driver info

Still fail to turn on sriov

[st4]

HI EluRex,

Thanks for the update. Let me check on this.

rgds,

wb

[CHsia9]

I have tried on driver 4.3.15... still the same

x552 not able to enable sriov

However, for x520 and i350 on the same system, sriov can be enabled

[VincentT_T_Intel]

Hi EluRex,

Considering that SR-IOV works after switching to another kernel (PVE 4.1 to Ubuntu 15.10), this suggests the issue is with the kernel and not the user tools, BIOS or driver. Suggest to contact Proxmox regarding this issue. you may file issue report thru https://bugzilla.proxmox.com/ https://bugzilla.proxmox.com/

Regards,

Vince

[CHsia9]

Dear Vincent,

I found the solution for it.

It is using /etc/modprobe.d/ixgbe.conf to turn on SR-IOV. It seems to me that sysfs write permission has been removed.

Chris

[st4]

Hi EluRex,

Good day. Please try using drive version 4.3.13 (this is the latest version) for X552 series network adapter

https://downloadcenter.intel.com/download/14687/Network-Adapter-Driver-for-PCI-E-10-Gigabit-Network-Connections-under-Linux- Download Network Adapter Driver for PCI-E* 10 Gigabit Network Connections under Linux*

rgds,

wb