xl pci-detach failure when using 82599 NIC

[zye5]

Hi,

I've come across a need to hotplug PCI devices between dom0 and domU using SR-IOV NIC. But I'm experiencing problems when trying to detach VF more than one PV guests.

I can attach VF to DomU successful as follow:

# xl pci-assignable-list

0000:05:10.0

0000:05:10.2

# xl pci-attach 1 05:10.0

# xl pci-attach 2 05:10.2

But when I can't detach VF and it report errors as follow:

# xl pci-detach 2 05:10.2

libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/2/0 not ready

And only 05:10.0 can detach successful :

# xl pci-detach 1 05:10.0

# xl pci-assignable-list

0000:05:10.0

Each guest config is the same like this:

name = "ubuntu-pv-1"

bootloader = "pygrub"

memory = 256

vcpus = 1

vif = [ 'bridge=xenbr0' ]

disk = [ 'file:/home/ye/ubuntu-pv/ubuntu-pv-1/ubuntu-pv-1.img,xvda,rw' ]

pci_permissive = 1

Follow xl dmesg log:

# xl dmesg | grep -i vt-d | grep -i enable

(XEN) Intel VT-d Snoop Control enabled.

(XEN) Intel VT-d Dom0 DMA Passthrough enabled.

(XEN) Intel VT-d Queued Invalidation enabled.

(XEN) Intel VT-d Interrupt Remapping enabled.

(XEN) Intel VT-d Shared EPT tables enabled.

# xl dmesg | grep "I/O virt"

(XEN) I/O virtualisation enabled

And libxl-driver log:

# cat /var/log/libvirt/libxl/libxl-driver.log

xc: detail: sysctl operation failed -- need to rebuild the user-space tool set?

libxl: error: libxl.c:4364:libxl_get_physinfo: getting physinfo: Permission denied

xc: debug: hypercall buffer: total allocations:7 total releases:7

xc: debug: hypercall buffer: current allocations:0 maximum allocations:1

xc: debug: hypercall buffer: cache current size:1

xc: debug: hypercall buffer: cache hits:6 misses:1 toobig:0

I'm running in Ubuntu 14 using Xen-4.6. I have tested in other machine using the same environment but encountering the same problem.

[idata]

Hi Zero123,

Thank you for posting in Wired Communities. We will further check on this.Regards,Sharon T

[idata]

Hi Zero123,

Please refer to this third party website, there is additional steps required prior attaching to virtual machine which you might find it helpful: https://wiki.xenproject.org/wiki/Xen_4.2:_xl_and_pci_pass-through(Please note this is third party website wherein we do not have control over the content, for your reference only) You may also try posting your inquiry at Xen forum.Regards,Sharon TIntel Customer SupportAgent under contract to Intel

[idata]

Hi Zero123,

Please feel free to update me if you have refer to the website and post at Xen forum for further assistance. Regards,Sharon TIntel Customer SupportAgent under contract to Intel

[zye5]

Hi,

Thank you for your answer.

I have posted the problem I encountered to the Xen forum, but nobody answered me. And in fact, I just configured the 82599 network card according to the Xen official website, but it didn't work smoothly.

[idata]

Hi Zero123,

Please attach the full dmesg log here and provide the exact NIC model, is this an Intel retail version 82599 (X520-series) or OEM NIC? You may refer to the website below to check on the device and vendor ID for us to identify the NIC: https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.htmlRegards,Sharon TIntel Customer SupportAgent under contract to Intel

[zye5]

Hi,

My NIC is the Intel retail version 82599 (X520-series) and dmesg log al follow:

[root@ccrfox141 ubuntu-pv-4]# dmesg | grep ixgbe

[ 0.000000] Command line: placeholder root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet max_loop=64 ixgbe.max_vfs=4 intel_iommu=on pci_pt_e820_access=on xen-pciback.hide=(05:10.0)(05:10.1)(05:10.2)(05:10.3)(05:10.4)(05:10.5)(05:10.6)(05:10.7)

[ 0.000000] Kernel command line: placeholder root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet max_loop=64 ixgbe.max_vfs=4 intel_iommu=on pci_pt_e820_access=on xen-pciback.hide=(05:10.0)(05:10.1)(05:10.2)(05:10.3)(05:10.4)(05:10.5)(05:10.6)(05:10.7)

[ 7.244711] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.2.1-k

[ 7.244713] ixgbe: Copyright (c) 1999-2015 Intel Corporation.

[ 7.244961] ixgbe 0000:05:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

[ 7.875441] ixgbe 0000:05:00.0 0000:05:00.0 (uninitialized): SR-IOV enabled with 4 VFs

[ 7.899478] ixgbe 0000:05:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4

[ 7.899605] ixgbe 0000:05:00.0: PCI Express bandwidth of 32GT/s available

[ 7.899607] ixgbe 0000:05:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

[ 7.899689] ixgbe 0000:05:00.0: MAC: 2, PHY: 18, SFP+: 5, PBA No: E68785-006

[ 7.899691] ixgbe 0000:05:00.0: 00:1b:21:80:62:6c

[ 7.900988] ixgbe 0000:05:00.0 eth0: IOV is enabled with 4 VFs

[ 7.901045] ixgbe 0000:05:00.0: Intel(R) 10 Gigabit Network Connection

[ 7.901317] ixgbe 0000:05:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

[ 9.480375] ixgbe 0000:05:00.1 0000:05:00.1 (uninitialized): SR-IOV enabled with 4 VFs

[ 9.504290] ixgbe 0000:05:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4

[ 9.504418] ixgbe 0000:05:00.1: PCI Express bandwidth of 32GT/s available

[ 9.504420] ixgbe 0000:05:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

[ 9.504503] ixgbe 0000:05:00.1: MAC: 2, PHY: 1, PBA No: E68785-006

[ 9.504504] ixgbe 0000:05:00.1: 00:1b:21:80:62:6d

[ 9.505757] ixgbe 0000:05:00.1 eth1: IOV is enabled with 4 VFs

[ 9.505781] ixgbe 0000:05:00.1: Intel(R) 10 Gigabit Network Connection

[ 9.566898] ixgbe 0000:05:00.0 p5p1: renamed from eth0

[ 9.588483] ixgbe 0000:05:00.1 p5p2: renamed from eth1

[ 18.729913] ixgbe 0000:05:00.0: registered PHC device on p5p1

[ 18.899330] ixgbe 0000:05:00.0 p5p1: detected SFP+: 5

[ 19.142256] ixgbe 0000:05:00.0 p5p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 27.375825] ixgbe 0000:05:00.0: setting MAC aa:bb:cc:dd:ee:00 on VF 0

[ 27.375827] ixgbe 0000:05:00.0: Reload the VF driver to make this change effective.

[ 27.377196] ixgbe 0000:05:00.0: setting MAC aa:bb:cc:dd:ee:01 on VF 1

[ 27.377199] ixgbe 0000:05:00.0: Reload the VF driver to make this change effective.

[ 27.378777] ixgbe 0000:05:00.0: setting MAC aa:bb:cc:dd:ee:02 on VF 2

[ 27.378780] ixgbe 0000:05:00.0: Reload the VF driver to make this change effective.

[ 27.380242] ixgbe 0000:05:00.0: setting MAC aa:bb:cc:dd:ee:03 on VF 3

[ 27.380245] ixgbe 0000:05:00.0: Reload the VF driver to make this change effective.

[ 27.381752] ixgbe 0000:05:00.1: setting MAC aa:bb:cc:dd:ee:04 on VF 0

[ 27.381755] ixgbe 0000:05:00.1: Reload the VF driver to make this change effective.

[ 27.381757] ixgbe 0000:05:00.1: The VF MAC address has been set, but the PF device is not up.

[ 27.381759] ixgbe 0000:05:00.1: Bring the PF device up before attempting to use the VF device.

[ 27.383237] ixgbe 0000:05:00.1: setting MAC aa:bb:cc:dd:ee:05 on VF 1

[ 27.383240] ixgbe 0000:05:00.1: Reload the VF driver to make this change effective.

[ 27.383241] ixgbe 0000:05:00.1: The VF MAC address has been set, but the PF device is not up.

[ 27.383243] ixgbe 0000:05:00.1: Bring the PF device up before attempting to use the VF device.

[ 27.384722] ixgbe 0000:05:00.1: setting MAC aa:bb:cc:dd:ee:06 on VF 2

[ 27.384725] ixgbe 0000:05:00.1: Reload the VF driver to make this change effective.

[ 27.384727] ixgbe 0000:05:00.1: The VF MAC address has been set, but the PF device is not up.

[ 27.384728] ixgbe 0000:05:00.1: Bring the PF device up before attempting to use the VF device.

[ 27.386238] ixgbe 0000:05:00.1: setting MAC aa:bb:cc:dd:ee:07 on VF 3

[ 27.386241] ixgbe 0000:05:00.1: Reload the VF driver to make this change effective.

[ 27.386243] ixgbe 0000:05:00.1: The VF MAC address has been set, but the PF device is not up.

[ 27.386244] ixgbe 0000:05:00.1: Bring the PF device up before attempting to use the VF device.

[16770.518670] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[16770.547190] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[16770.602526] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 2

[16770.629248] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 3

[16771.273580] ixgbe 0000:05:00.0 p5p1: VF 0 attempted to override administratively set MAC address

[16772.760880] ixgbe 0000:05:00.0 p5p1: 6 Spoofed packets detected

[16774.764837] ixgbe 0000:05:00.0 p5p1: 3 Spoofed packets detected

[16776.768870] ixgbe 0000:05:00.0 p5p1: 1 Spoofed packets detected

[16778.772826] ixgbe 0000:05:00.0 p5p1: 2 Spoofed packets detected

[16780.776864] ixgbe 0000:05:00.0 p5p1: 4 Spoofed packets detected

[19434.491860] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[19860.712951] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[19860.767860] ixgbe 0000:05:00.0 p5p1: VF 0 attempted to override administratively set MAC address

[19901.516353] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[19910.552027] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[19910.607840] ixgbe 0000:05:00.0 p5p1: VF 0 attempted to override administratively set MAC address

[82087.324219] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 0

[82918.563915] ixgbe 0000:05:00.1: registered PHC device on p5p2

[83100.481367] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[83100.522953] ixgbe 0000:05:00.0 p5p1: VF 1 attempted to override administratively set MAC address

[83101.123400] ixgbe 0000:05:00.0 p5p1: 2 Spoofed packets detected

[83103.127463] ixgbe 0000:05:00.0 p5p1: 2 Spoofed packets detected

[83106.232597] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 2

[83106.272828] ixgbe 0000:05:00.0 p5p1: VF 2 attempted to override administratively set MAC address

[83948.672430] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[86372.337957] ixgbe 0000:05:00.0 p5p1: VF 0 is trusted

[86452.400834] ixgbe 0000:05:00.0 p5p1: VF 1 is trusted

[86452.420369] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[86456.888314] ixgbe 0000:05:00.0 p5p1: VF 2 is trusted

[86461.127892] ixgbe 0000:05:00.0 p5p1: VF 3 is trusted

[86587.891895] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 2

[103593.696425] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[103597.987892] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 2

[103812.951160] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 1

[103812.992804] ixgbe 0000:05:00.0 p5p1: VF 1 attempted to override administratively set MAC address

[103819.403340] ixgbe 0000:05:00.0 p5p1: VF Reset msg received from vf 2

[103819.445374] ixgbe 0000:05:00.0 p5p1: VF 2 attempted to override administratively set MAC address

[idata]

Hi Zero123,

Thank you for the information. We will further investigate.Regards,Sharon TIntel Customer SupportAgent under contract to Intel

[idata]

Hi Zero123,

Please refer to this link for the configuration of SR-IOV on Xen, check if you can detach the vF from the Dom-U. https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/ethernet-x520-suse-linux-xen-tech-brief.pdf It is recommended to use the latest driver, you may download from https://sourceforge.net/projects/e1000/files/. Feel free to update us the result.Regards,Sharon TIntel Customer SupportAgent under contract to Intel

[idata]

Hi Zero123,

Please feel free to update us if you have tried the latest driver. Regards,Sharon TIntel Customer SupportAgent under contract to Intel