intel igb i354 issue with lack of multiqueue (intel atom C2758 board)

[ITria]

Hi there,

I've been having some trouble getting igb/i354 to enable multiple queues.

Despite compiling the latest igb module (5.2.9.4 at the time of writing this) I have not been able to enable it.

I have ensured that MSI-X interrupts are used, and even set RSS values to 8 (and IntMode to 2 to be safe again) and rebuilt initrd and have googled extensively as well as followed all the information/advice I've seen in intel's documentation for installing the modules. The kernel I am running is 2.6.32 on a proxmox installation (though I tried with a 3.10 kernel as well with same results).

The motherboard is a Supermicro A1SRi-2758F atom server board and there is nothing I can see in the BIOS that looks like it could be causing this issue.

I am starting to think that I am missing something obvious here...

Any help/advice would be much appreciated.

Regards,

Iakovos

From lspci ( lspci -vv at the bottom of this post due to size):

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)

00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)

00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02)

00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)

00:0b.0 Co-processor: Intel Corporation Atom processor C2000 nCPM (rev 02)

00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)

00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)

00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)

00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)

00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)

00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)

00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)

00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02)

00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02)

00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02)

00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)

00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)

01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)

02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

03:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

From dmesg:

igb 0000:00:14.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20

igb 0000:00:14.0: setting latency timer to 64

igb 0000:00:14.0: irq 35 for MSI/MSI-X

igb 0000:00:14.0: irq 36 for MSI/MSI-X

igb 0000:00:14.0: Intel(R) Gigabit Ethernet Network Connection

igb 0000:00:14.0: eth0: (PCIe:integrated:integrated)

igb 0000:00:14.0: eth0: MAC: 0c:c4:7a:xx:xx:xx

igb 0000:00:14.0: eth0: PBA No: 010A00-000

igb 0000:00:14.0: LRO is disabled

igb 0000:00:14.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

igb 0000:00:14.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21

igb 0000:00:14.1: setting latency timer to 64

igb 0000:00:14.1: irq 38 for MSI/MSI-X

igb 0000:00:14.1: irq 39 for MSI/MSI-X

igb 0000:00:14.1: Intel(R) Gigabit Ethernet Network Connection

igb 0000:00:14.1: eth1: (PCIe:integrated:integrated)

igb 0000:00:14.1: eth1: MAC: 0c:c4:7a:xx:xx:xx

igb 0000:00:14.1: eth1: PBA No: 010A00-000

igb 0000:00:14.1: LRO is disabled

igb 0000:00:14.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

igb 0000:00:14.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22

igb 0000:00:14.2: setting latency timer to 64

igb 0000:00:14.2: irq 41 for MSI/MSI-X

igb 0000:00:14.2: irq 42 for MSI/MSI-X

igb 0000:00:14.2: Intel(R) Gigabit Ethernet Network Connection

igb 0000:00:14.2: eth2: (PCIe:integrated:integrated)

igb 0000:00:14.2: eth2: MAC: 0c:c4:7a:xx:xx:xx

igb 0000:00:14.2: eth2: PBA No: 010A00-000

igb 0000:00:14.2: LRO is disabled

igb 0000:00:14.2: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

igb 0000:00:14.3: PCI INT D -> GSI 23 (level, low) -> IRQ 23

igb 0000:00:14.3: setting latency timer to 64

igb 0000:00:14.3: irq 43 for MSI/MSI-X

igb 0000:00:14.3: irq 44 for MSI/MSI-X

igb 0000:00:14.3: Intel(R) Gigabit Ethernet Network Connection

igb 0000:00:14.3: eth3: (PCIe:integrated:integrated)

igb 0000:00:14.3: eth3: MAC: 0c:c4:7a:xx:xx:xx

igb 0000:00:14.3: eth3: PBA No: 010A00-000

igb 0000:00:14.3: LRO is disabled

igb 0000:00:14.3: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)

igb 0000:00:14.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

igb 0000:00:14.1: eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

igb 0000:00:14.2: eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

igb 0000:00:14.3: eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

From cat /proc/interrupts |grep eth:

35: 0 0 0 0 0 0 1 0 PCI-MSI-edge eth0

36: 919 901 926 916 920 936 882 904 PCI-MSI-edge eth0-TxRx-0

38: 0 1 0 0 0 0 0 0 PCI-MSI-edge eth1

39: 589 629 635 616 609 582 609 615 PCI-MSI-edge eth1-TxRx-0

41: 0 0 0 0 0 1 0 0 PCI-MSI-edge eth2

42: 644 620 577 604 602 616 603 615 PCI-MSI-edge eth2-TxRx-0

43: 0 0 1 0 0 0 0 0 PCI-MSI-edge eth3

44: 599 593 601 618 600 592 640 630 PCI-MSI-edge eth3-TxRx-0

Values I am passing on modules.conf & modprobe.d/options (for 2.6/3.x):

igb RSS=8,8,8,8 IntMode=2,2,2,2

# modinfo igb

filename: /lib/modules/2.6.32-32-pve/kernel/drivers/net/igb/igb.ko

version: 5.2.9.4

license: GPL

description: Intel(R) Gigabit Ethernet Network Driver

author: Intel Corporation, <</span>mailto:e1000-devel@lists.sourceforge.net e1000-devel@lists.sourceforge.net>

srcversion: E377200391EBF74638FEDA2

alias: pci:v00008086d000010D6sv*sd*bc*sc*i*

alias: pci:v00008086d000010A9sv*sd*bc*sc*i*

alias: pci:v00008086d000010A7sv*sd*bc*sc*i*

alias: pci:v00008086d000010E8sv*sd*bc*sc*i*

alias: pci:v00008086d00001526sv*sd*bc*sc*i*

alias: pci:v000080...

[PAUL_S_Intel]

Hi Iakovos -

I am an Intel Applications engineer. Could you please send your information to one of the following for a response:

1) File a bug on the e1000 SourceForge mailing list (e1000.sourceforge.net) or

2) email mailto:Linux-nics@intel.com Linux-nics@intel.com

Then we should be able to get you some help with your issue.

thanks;

Paul

[ITria]

Hi Paul,

Many thanks for your reply. It's much appreciated and nice to see that intel employees show interest in customers' problems (makes a nice change sometimes)!

As I needed a working implementation within a certain timeframe I was left with little option but to reinstall using a different distro. My troubleshooting pretty much proved that the issue was related with some options set within the kernels packaged with proxmox. I am afraid I run out of time to troubleshoot this further at this point. At a guess if you had similar hardware at hand you shouldn't have much trouble reproducing the issue

Using the very same driver/module within a debian vanilla kernel (both stable and testing) works a treat. As a matter of fact I had multiqueue support out of the box with the debian kernel with an igb versions of 5.0.5-k (if memory serves) though I compiled the latest version of the module regardless.

I will however make notes of the above information for future reference.

Many thanks for your time,

Iakovos