Dual port X710 for 10GbE SFP+ (rev 02) VF failure on port 2

[NikhilA]

Hi,

We have x710 dual port NIC with following details

37:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

While no issue is observed with VFs from first port of X710 i.e PCI-37:00.0, issue happens for always VFs of 2nd port i.e on PCI-37:00.1

Following are dmesg from both VM and HOST machines. 

Attachements:

1. VM & HOST dmesg extracts - VM-dmesg.txt, host-dmesg.txt.

2. ethtool and lspci o/ps of X710 - NIC-and-driver-details.txt   

Issue: VM is not able to access VF from 2nd port i.e 37:00.1.

Error on VM (dmesg) on bootup:

i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
------------[ cut here ]------------
WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: ens11 (i40evf): transmit queue 1 timed out
Modules linked in: sunrpc vfat fat nfit libnvdimm snd_hda_codec_generic snd_hda_intel snd_hda_codec iosf_mbi snd_hda_core crc32_pclmul ghash_clmulni_intel snd_hwdep snd_pcm aesni_intel ppdev lrw snd_timer gf128mul snd glue_helper ablk_helper soundcore cryptd i40evf parport_pc sg i2c_piix4 parport virtio_balloon joydev pcspkr ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy ata_piix drm crct10dif_pclmul crct10dif_common e1000 crc32c_intel libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 0 Comm: swapper/5 Not tainted 3.10.0-957.el7.pw.1.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<IRQ> [<ffffffff92d61e11>] dump_stack+0x19/0x1b
[<ffffffff92697648>] __warn+0xd8/0x100
[<ffffffff926976cf>] warn_slowpath_fmt+0x5f/0x80
[<ffffffff92c65bb8>] dev_watchdog+0x248/0x260
[<ffffffff92c65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff926a7f58>] call_timer_fn+0x38/0x110
[<ffffffff92c65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff926aa3bd>] run_timer_softirq+0x24d/0x300
[<ffffffff926a0f05>] __do_softirq+0xf5/0x280
[<ffffffff92d7832c>] call_softirq+0x1c/0x30
[<ffffffff9262e675>] do_softirq+0x65/0xa0
[<ffffffff926a1285>] irq_exit+0x105/0x110
[<ffffffff92d796c8>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffff92d75df2>] apic_timer_interrupt+0x162/0x170
<EOI> [<ffffffff92d69a70>] ? __cpuidle_text_start+0x8/0x8
[<ffffffff92d69c76>] ? native_safe_halt+0x6/0x10
[<ffffffff92d69a8e>] default_idle+0x1e/0xc0
[<ffffffff926366f0>] arch_cpu_idle+0x20/0xc0
[<ffffffff926fc3ba>] cpu_startup_entry+0x14a/0x1e0
[<ffffffff92657db7>] start_secondary+0x1f7/0x270
[<ffffffff926000d5>] start_cpu+0x5/0x14
---[ end trace 74ab1529c3e517c9 ]---
i40evf 0000:00:0b.0 ens11: NIC Link is Up 10 Gbps Full Duplex
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Device is still in reset (-16), retrying
i40evf 0000:00:0c.0: Failed to communicate with PF; waiting before retry 

Dmesg on Host machine on VF creation on both port of x710:

iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 3.2.3-k
Copyright (c) 2013 - 2018 Intel Corporation.
iavf 0000:37:02.0: enabling device (0000 -> 0002)
iavf 0000:37:02.1: enabling device (0000 -> 0002)
iavf 0000:37:02.0: Device is still in reset (-16), retrying
iavf 0000:37:02.1: Device is still in reset (-16), retrying
iavf 0000:37:02.0: Invalid MAC address 00:00:00:00:00:00, using random
iavf 0000:37:02.0: irq 314 for MSI/MSI-X
iavf 0000:37:02.0: irq 315 for MSI/MSI-X
iavf 0000:37:02.0: irq 316 for MSI/MSI-X
iavf 0000:37:02.0: irq 317 for MSI/MSI-X
iavf 0000:37:02.0: irq 318 for MSI/MSI-X
iavf 0000:37:02.0: Multiqueue Enabled: Queue pair count = 4
iavf 0000:37:02.0: MAC address: e6:69:28:15:d5:55
iavf 0000:37:02.0: GRO is enabled
iavf 0000:37:02.1: Invalid MAC address 00:00:00:00:00:00, using random
iavf 0000:37:02.1: irq 319 for MSI/MSI-X
iavf 0000:37:02.1: irq 320 for MSI/MSI-X
iavf 0000:37:02.1: irq 321 for MSI/MSI-X
iavf 0000:37:02.1: irq 322 for MSI/MSI-X
iavf 0000:37:02.1: irq 323 for MSI/MSI-X
iavf 0000:37:02.1: Multiqueue Enabled: Queue pair count = 4
iavf 0000:37:02.1: MAC address: 52:de:da:00:7a:88
iavf 0000:37:02.1: GRO is enabled
IPv6: ADDRCONF(NETDEV_UP): enp55s2: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s2: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s2f1: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s2f1: link is not ready
iavf 0000:37:02.0 enp55s2: NIC Link is Up 10 Gbps Full Duplex
IPv6: ADDRCONF(NETDEV_CHANGE): enp55s2: link becomes ready
iavf 0000:37:02.1 enp55s2f1: NIC Link is Up 10 Gbps Full Duplex
IPv6: ADDRCONF(NETDEV_CHANGE): enp55s2f1: link becomes ready
i40e 0000:37:00.1: VF BW shares not restored
i40e 0000:37:00.1: FW LLDP is enabled
i40e 0000:37:00.1: Allocating 2 VFs.
pci 0000:37:0a.0: [8086:154c] type 00 class 0x020000
iommu: Adding device 0000:37:0a.0 to group 85
iavf 0000:37:0a.0: enabling device (0000 -> 0002)
pci 0000:37:0a.1: [8086:154c] type 00 class 0x020000
iommu: Adding device 0000:37:0a.1 to group 86
iavf 0000:37:0a.1: enabling device (0000 -> 0002)
i40e 0000:37:00.1: created sriov sysfs
i40e 0000:37:00.1: created VF 0 sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF 1 sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
i40e 0000:37:00.1: created VF qos sysfs
iavf 0000:37:0a.0: Device is still in reset (-16), retrying
iavf 0000:37:0a.1: Device is still in reset (-16), retrying
iavf 0000:37:0a.0: Invalid MAC address 00:00:00:00:00:00, using random
iavf 0000:37:0a.0: irq 324 for MSI/MSI-X
iavf 0000:37:0a.0: irq 325 for MSI/MSI-X
iavf 0000:37:0a.0: irq 326 for MSI/MSI-X
iavf 0000:37:0a.0: irq 327 for MSI/MSI-X
iavf 0000:37:0a.0: irq 328 for MSI/MSI-X
iavf 0000:37:0a.0: Multiqueue Enabled: Queue pair count = 4
iavf 0000:37:0a.0: MAC address: ae:bb:70:3d:3f:5a
iavf 0000:37:0a.0: GRO is enabled
iavf 0000:37:0a.1: Invalid MAC address 00:00:00:00:00:00, using random
iavf 0000:37:0a.1: irq 329 for MSI/MSI-X
iavf 0000:37:0a.1: irq 330 for MSI/MSI-X
iavf 0000:37:0a.1: irq 331 for MSI/MSI-X
iavf 0000:37:0a.1: irq 332 for MSI/MSI-X
iavf 0000:37:0a.1: irq 333 for MSI/MSI-X
iavf 0000:37:0a.1: Multiqueue Enabled: Queue pair count = 4
iavf 0000:37:0a.1: MAC address: 7e:65:87:0b:2f:4f
iavf 0000:37:0a.1: GRO is enabled
IPv6: ADDRCONF(NETDEV_UP): enp55s10: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s10: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s10f1: link is not ready
IPv6: ADDRCONF(NETDEV_UP): enp55s10f1: link is not ready
iavf 0000:37:0a.0 enp55s10: NIC Link is Up 10 Gbps Full Duplex
IPv6: ADDRCONF(NETDEV_CHANGE): enp55s10: link becomes ready
iavf 0000:37:0a.1 enp55s10f1: NIC Link is Up 10 Gbps Full Duplex
IPv6: ADDRCONF(NETDEV_CHANGE): enp55s10f1: link becomes ready
------------[ cut here ]------------
WARNING: CPU: 25 PID: 0 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: enp55s10f1 (iavf): transmit queue 2 timed out
Modules linked in: iavf xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ipmi_ssif skx_edac nfit libnvdimm intel_powerclamp coretemp intel_rapl mei_me hpilo iosf_mbi mei kvm_intel ses enclosure kvm irqbypass crc32_pclmul sg ghash_clmulni_intel aesni_intel wmi lrw gf128mul glue_helper ablk_helper cryptd pcspkr hpwdt lpc_ich ipmi_si ipmi_devintf
ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic uas usb_storage mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm crct10dif_pclmul crct10dif_common crc32c_intel igb serio_raw smartpqi i40e(OE) scsi_transport_sas mdio ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 25 PID: 0 Comm: swapper/25 Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 07/16/2020
Call Trace:
<IRQ> [<ffffffffae38311a>] dump_stack+0x19/0x1b
[<ffffffffadc9b1b8>] __warn+0xd8/0x100
[<ffffffffadc9b23f>] warn_slowpath_fmt+0x5f/0x80
[<ffffffffadcbbd1c>] ? __queue_work+0x13c/0x3f0
[<ffffffffae283f78>] dev_watchdog+0x248/0x260
[<ffffffffae283d30>] ? dev_deactivate_queue.constprop.27+0x60/0x60
[<ffffffffadcabcf8>] call_timer_fn+0x38/0x110
[<ffffffffae283d30>] ? dev_deactivate_queue.constprop.27+0x60/0x60
[<ffffffffadcae30d>] run_timer_softirq+0x25d/0x340
[<ffffffffadca4b35>] __do_softirq+0xf5/0x280
[<ffffffffae3994ec>] call_softirq+0x1c/0x30
[<ffffffffadc2f715>] do_softirq+0x65/0xa0
[<ffffffffadca4eb5>] irq_exit+0x105/0x110
[<ffffffffae39aa88>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffffae396fba>] apic_timer_interrupt+0x16a/0x170
<EOI> [<ffffffffae1c8467>] ? cpuidle_enter_state+0x57/0xd0
[<ffffffffae1c85be>] cpuidle_idle_call+0xde/0x230
[<ffffffffadc37c8e>] arch_cpu_idle+0xe/0xc0
[<ffffffffadd014ca>] cpu_startup_entry+0x14a/0x1e0
[<ffffffffadc5a827>] start_secondary+0x1f7/0x270
[<ffffffffadc000d5>] start_cpu+0x5/0x14
---[ end trace 718f4305018e4b5a ]---

Appreciate any help on this.  

Thanks in advance. 

[BaconPackets]

Did you ever find a resolution on this?

[NikhilA]

yes, upgrading to latest firmware solved it for me. 

[BaconPackets]

Out of curiosity, what is the latest firmware you see?

I have 10.54.7 and we are still seeing the issue.