Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

Some VFs are assigned three IRQs and some receive only two

ALyak
Beginner
1,403 Views

Greetings all,

We are using Ubunti-Precise 3.2.0-29-generic # 46, with KVM version qemu-kvm-1.0 and libvirt 0.9.8. We are using Intel NIC 82599 with ixgbe driver 3.11.33 and in-tree ixgbevf driver 2.2.0-k. We are attaching VFs to KVM instances.

In lspci we see that MSI-X table for each VF has three entries, like this:

03:17.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

Subsystem: Intel Corporation Device 7a11

Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Region 0: [virtual] Memory at df478000 (64-bit, non-prefetchable) [size=16K]

Region 3: [virtual] Memory at df578000 (64-bit, non-prefetchable) [size=16K]

Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

But when attaching the VF to KVM, we see that most of the VFs receive only two IRQs, while only a few receive three IRQs.

For example:

Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.448999] ixgbevf 0000:03:17.3: irq 359 for MSI/MSI-X

Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.449016] ixgbevf 0000:03:17.3: irq 360 for MSI/MSI-X

Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.449033] ixgbevf 0000:03:17.3: irq 361 for MSI/MSI-X

Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490908] pci-stub 0000:03:12.2: irq 138 for MSI/MSI-X

Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490924] pci-stub 0000:03:12.2: irq 139 for MSI/MSI-X

Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730559] pci-stub 0000:03:1c.1: irq 335 for MSI/MSI-X

Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730580] pci-stub 0000:03:1c.1: irq 336 for MSI/MSI-X

Initial debugging shows, that KVM reads the PCI configuration space to determine the number of IRQs to assign:

static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)

{

AssignedDevice *adev = DO_UPCAST(AssignedDevice, dev, pci_dev);

uint16_t entries_nr = 0, entries_max_nr;

int pos = 0, i, r = 0;

uint32_t msg_addr, msg_upper_addr, msg_data, msg_ctrl;

struct kvm_assigned_msix_nr msix_nr;

struct kvm_assigned_msix_entry msix_entry;

void *va = adev->msix_table_page;

pos = pci_find_capability(pci_dev, PCI_CAP_ID_MSIX);

entries_max_nr = *(uint16_t *)(pci_dev->config + pos + 2);

entries_max_nr &= PCI_MSIX_FLAGS_QSIZE;

entries_max_nr += 1;

This yields entries_max_nr=3, as expected.

But then:

/* Get the usable entry number for allocating */

for (i = 0; i < entries_max_nr; i++) {

memcpy(&msg_ctrl, va + i * 16 + 12, 4);

memcpy(&msg_data, va + i * 16 + 8, 4);

/* Ignore unused entry even it's unmasked */

if (msg_data == 0)

continue;

entries_nr ++;

}

And adding some prints shows that the third entry yields msg_data==0, so it is skipped.

ALEXL: PCIDEV(pci-assign) entries_max_nr=3

ALEXL: PCIDEV(pci-assign) entry # 0 msg_data=16817

ALEXL: PCIDEV(pci-assign) entry # 1 msg_data=16833

ALEXL: PCIDEV(pci-assign) entry # 2 msg_data=0

ALEXL: PCIDEV(pci-assign) entries_max_nr=3

ALEXL: PCIDEV(pci-assign) entry # 0 msg_data=16849

ALEXL: PCIDEV(pci-assign) entry # 1 msg_data=16865

ALEXL: PCIDEV(pci-assign) entry # 2 msg_data=0

ALEXL: PCIDEV(pci-assign) entries_max_nr=3

ALEXL: PCIDEV(pci-assign) entry # 0 msg_data=16674

ALEXL: PCIDEV(pci-assign) entry # 1 msg_data=16706

ALEXL: PCIDEV(pci-assign) entry # 2 msg_data=0

We don't know what is the implication of having only two IRQs assigned instead of three. From overall perspective, those KVM instances function normally. Can anybody please comment on this?

When using SR-IOV cards from another vendor, we see that those other cards have 4 entries for each VF, and all those entries have IRQs attached. So we see this issue only with Intel cards.

Thanks,

Alex.

0 Kudos
3 Replies
Patrick_K_Intel1
Employee
516 Views

Hi Alex,

Our guess is that you have a mis-aligned PF and VF version. We did some modifications of how many interrupts we use per VF a while back and went from 3 to 2. Can you try the latest and greatest drivers located at http://sourceforge.net/projects/e1000/files/ http://sourceforge.net/projects/e1000/files/ and let us know how it went?

The latest IXGBE PF is 3.19.1 and the latest IXGBEvf drivers is 2.12.1.

thanx,

Patrick

0 Kudos
ALyak
Beginner
516 Views

Hi Patrick,

with the latest drivers you mentioned, we still see that MSI-X table for each VF has three entries, but each VF receives only two IRQs:

03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0

Region 0: [virtual] Memory at d1300000 (64-bit, prefetchable) [size=16K]

Region 3: [virtual] Memory at d1200000 (64-bit, prefetchable) [size=16K]

Capabilities: [70] MSI-X: Enable+ Count=3 Masked-

but still PFs and VFs receive two IRQs each:

Jan 27 02:32:00 la-01-04 kernel: [4.887658] ixgbe 0000:03:00.1: (unregistered net_device): SR-IOV enabled with 32 VFsJan 27 02:32:00 la-01-04 kernel: [4.911463] ixgbe 0000:03:00.1: irq 117 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [4.911481] ixgbe 0000:03:00.1: irq 118 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [4.913246] ixgbe 0000:03:00.1: PCI Express bandwidth of 32GT/s availableJan 27 02:32:00 la-01-04 kernel: [4.913251] ixgbe 0000:03:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)Jan 27 02:32:00 la-01-04 kernel: [4.913335] ixgbe 0000:03:00.1: eth5: MAC: 2, PHY: 9, SFP+: 4, PBA No: FFFFFF-0FFJan 27 02:32:00 la-01-04 kernel: [4.913339] ixgbe 0000:03:00.1: f8:4a:bf:56:ce:27Jan 27 02:32:00 la-01-04 kernel: [4.913342] ixgbe 0000:03:00.1: eth5: Enabled Features: RxQ: 1 TxQ: 1Jan 27 02:32:00 la-01-04 kernel: [4.913356] ixgbe 0000:03:00.1: eth5: Intel(R) 10 Gigabit Network ConnectionJan 27 02:32:00 la-01-04 kernel: [4.913637] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)Jan 27 02:32:00 la-01-04 kernel: [4.913650] ixgbevf 0000:03:10.0: setting latency timer to 64Jan 27 02:32:00 la-01-04 kernel: [4.976594] ixgbevf 0000:03:10.0: PF still in reset state. Is the PF interface up?Jan 27 02:32:00 la-01-04 kernel: [4.976600] ixgbevf 0000:03:10.0: Assigning random MAC addressJan 27 02:32:00 la-01-04 kernel: [4.976705] ixgbevf 0000:03:10.0: irq 119 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [4.976722] ixgbevf 0000:03:10.0: irq 120 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [4.976729] ixgbevf: eth%d: ixgbevf_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1Jan 27 02:32:00 la-01-04 kernel: [4.977184] ixgbevf: eth6: ixgbevf_probe: Intel(R) 82599 Virtual FunctionJan 27 02:32:00 la-01-04 kernel: [4.977188] a2:b2:a3:e9:8e:d3Jan 27 02:32:00 la-01-04 kernel: [4.977192] ixgbevf: eth6: ixgbevf_probe: GRO is enabledJan 27 02:32:00 la-01-04 kernel: [4.977195] ixgbevf: eth6: ixgbevf_probe: Intel(R) 10 Gigabit PCI Express Virtual Function Network DriverJan 27 02:32:00 la-01-04 kernel: [4.977348] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)Jan 27 02:32:00 la-01-04 kernel: [4.977362] ixgbevf 0000:03:10.2: setting latency timer to 64Jan 27 02:32:00 la-01-04 kernel: [5.040503] ixgbevf 0000:03:10.2: PF still in reset state. Is the PF interface up?Jan 27 02:32:00 la-01-04 kernel: [5.040512] ixgbevf 0000:03:10.2: Assigning random MAC addressJan 27 02:32:00 la-01-04 kernel: [5.040632] ixgbevf 0000:03:10.2: irq 121 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [5.040649] ixgbevf 0000:03:10.2: irq 122 for MSI/MSI-XJan 27 02:32:00 la-01-04 kernel: [5.040656] ixgbevf: eth%d: ixgbevf_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1Jan 27 02:32:00 la-01-04 kernel: [5.041162] ixgbevf: eth7: ixgbevf_probe: Intel(R) 82599 Virtual FunctionJan 27 02:32:00 la-01-04 kernel: [5.041166] 8e:03:15:99:05:5eJan 27 02:32:00 la-01-04 kernel: [5.041170] ixgbevf: eth7: ixgbevf_probe: GRO is enabledJan 27 02:32:00 la-01-04 kernel: [5.041173] ixgbevf: eth7: ixgbevf_probe: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver

Is a firmware upgrade perhaps needed to make MSIX table have two entries?

0 Kudos
Patrick_K_Intel1
Employee
516 Views

We have recently added Mulit-Queue support to our SR-IOV solution. Multi-Queue (MQ) uses 3 interrupts, non MQ only uses 2; however the code still requests 3. I've asked engineering to look into this and see if we can only reqeust 2 if only going to use 2.

To use MQ, and all 3 interrupts, when you compile both the PF and VF use CFLAGS_EXTRA="-DIXGBE_ENABLE_VF_MQ".

Hope this helps,

Patrick

0 Kudos
Reply