Kernel Panic on i40e using last drivers and last NVM (7.1)

[PNade2]

We are experiencing kernel panic and system freeze when we try to detach PCI from KVM machine. For do this we execute this orders.

ip link set pf0 vf 3 mac 00:00:00:00:00:00

ip link set pf0 vf 3 vlan 0

ip link set pf0 vf 3 trust off

ip link set pf0 vf 3 spoofchk on

Trace:

[ 878.355774] CPU: 34 PID: 45959 Comm: ip Tainted: P      O   5.3.13-1-pve #1

[ 878.355825] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.9.1 12/04/2018

[ 878.355898] RIP: 0010:i40e_config_vf_promiscuous_mode+0x177/0x330 [i40e]

[ 878.355946] Code: 8b 00 83 d1 00 48 85 c0 75 ef 48 83 c6 08 48 39 75 c8 75 dd 85 c9 74 79 0f b6 45 b8 45 31 d2 89 45 d4 4d 8b 2f 4d 85 ed 74 58 <41> 0f b7 4d 16 66 81 f9 ff 0f 77 43 0f b7 b3 ea 0c 00 00 8b 55 d4

[ 878.356061] RSP: 0018:ffff9a7046a97560 EFLAGS: 00010202

[ 878.356100] RAX: 0000000000000000 RBX: ffff889cd7f0d000 RCX: 0000000000000000

[ 878.356147] RDX: 0000000000000000 RSI: 0000000006000000 RDI: 0000000000000000

[ 878.356194] RBP: ffff9a7046a975b8 R08: 0000000000000000 R09: 0000000000000022

[ 878.356240] R10: 0000000000000000 R11: ffff9a7046a97500 R12: ffff889dcfcfc008

[ 878.356287] R13: 4d2d4b7e20d769c2 R14: 0000000000000000 R15: ffff889cd7f0d338

[ 878.356335] FS: 00007f67256f8e40(0000) GS:ffff889ddf840000(0000) knlGS:0000000000000000

[ 878.356388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 878.356427] CR2: 00007f6725a4c730 CR3: 00000006eea66003 CR4: 00000000001626e0

[ 878.356475] Call Trace:

[ 878.356518]  i40e_ndo_set_vf_port_vlan+0x1d1/0x2d0 [i40e]

[ 878.356561]  do_setlink+0x572/0xe40

[ 878.356593]  __rtnl_newlink+0x538/0x900

[ 878.356627]  ? __nla_validate_parse+0x50/0x7f0

[ 878.356665]  ? mem_cgroup_try_charge+0x71/0x190

[ 878.356700]  ? __alloc_pages_nodemask+0x16a/0x330

[ 878.356735]  ? page_counter_try_charge+0x5b/0xd0

[ 878.356775]  ? __mutex_lock.isra.10+0x3c0/0x4c0

[ 878.356810]  ? _cond_resched+0x19/0x30

[ 878.356840]  ? kmem_cache_alloc_trace+0x15e/0x230

[ 878.356875]  rtnl_newlink+0x48/0x70

[ 878.356903]  rtnetlink_rcv_msg+0x2b6/0x370

[ 878.356936]  ? _cond_resched+0x19/0x30

[ 878.356965]  ? rtnl_calcit.isra.35+0x100/0x100

[ 878.357001]  netlink_rcv_skb+0x4f/0x120

[ 878.357031]  rtnetlink_rcv+0x15/0x20

[ 878.357061]  netlink_unicast+0x197/0x220

[ 878.357093]  netlink_sendmsg+0x213/0x3c0

[ 878.357125]  sock_sendmsg+0x63/0x70

[ 878.357153]  ___sys_sendmsg+0x29f/0x320

[ 878.357186]  ? call_rcu+0x10/0x20

[ 878.358471]  ? destroy_inode+0x65/0x80

[ 878.359747]  ? evict+0x139/0x1a0

[ 878.360996]  ? _cond_resched+0x19/0x30

[ 878.362225]  ? __dentry_kill+0x124/0x170

[ 878.363433]  __sys_sendmsg+0x5c/0xa0

[ 878.364620]  __x64_sys_sendmsg+0x1f/0x30

[ 878.365805]  do_syscall_64+0x5a/0x130

[ 878.366993]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

[ 878.368196] RIP: 0033:0x7f6725aa9914

[ 878.369379] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41 89 d4 55 48 89 f5 53

[ 878.371882] RSP: 002b:00007ffd55473218 EFLAGS: 00000246 ORIG_RAX: 000000000000002e

[ 878.373128] RAX: ffffffffffffffda RBX: 000000005df26cc1 RCX: 00007f6725aa9914

[ 878.374367] RDX: 0000000000000000 RSI: 00007ffd55473280 RDI: 0000000000000005

[ 878.375599] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000010

[ 878.376790] R10: fffffffffffff466 R11: 0000000000000246 R12: 0000000000000001

[ 878.377942] R13: 0000555ce4170560 R14: 0000000000000000 R15: 00007ffd55473a58

[ 878.379077] Modules linked in: rbd libceph ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp iptable_filter bpfilter openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel mgag200 aes_x86_64 drm_vram_helper crypto_simd cryptd ttm glue_helper dcdbas drm_kms_helper intel_cstate intel_rapl_perf drm pcspkr joydev input_leds fb_sys_fops syscopyarea sysfillrect sysimgblt mxm_wmi mei_me mei ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor

[ 878.379139]  zstd_compress raid6_pq libcrc32c hid_generic usbkbd usbmouse usbhid hid i40e(O) ixgbe igb lpc_ich i2c_algo_bit xfrm_algo ahci dca libahci mdio megaraid_sas wmi

[ 878.392543] ---[ end trace f48e2fe75d2d811a ]---

[ 878.431337] RIP: 0010:i40e_config_vf_promiscuous_mode+0x177/0x330 [i40e]

[ 878.433138] Code: 8b 00 83 d1 00 48 85 c0 75 ef 48 83 c6 08 48 39 75 c8 75 dd 85 c9 74 79 0f b6 45 b8 45 31 d2 89 45 d4 4d 8b 2f 4d 85 ed 74 58 <41> 0f b7 4d 16 66 81 f9 ff 0f 77 43 0f b7 b3 ea 0c 00 00 8b 55 d4

[ 878.436303] RSP: 0018:ffff9a7046a97560 EFLAGS: 00010202

[ 878.437873] RAX: 0000000000000000 RBX: ffff889cd7f0d000 RCX: 0000000000000000

[ 878.439483] RDX: 0000000000000000 RSI: 0000000006000000 RDI: 0000000000000000

[ 878.441073] RBP: ffff9a7046a975b8 R08: 0000000000000000 R09: 0000000000000022

[ 878.442666] R10: 0000000000000000 R11: ffff9a7046a97500 R12: ffff889dcfcfc008

[ 878.444292] R13: 4d2d4b7e20d769c2 R14: 0000000000000000 R15: ffff889cd7f0d338

[ 878.445931] FS: 00007f67256f8e40(0000) GS:ffff889ddf840000(0000) knlGS:0000000000000000

[ 878.447665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 878.449311] CR2: 00007f6725a4c730 CR3: 00000006eea66003 CR4: 00000000001626e0

Kindly,

Pau

[PNade2]

Rollback to old kernel 5.0.21-5 fixed this. Can someone check ?  If hardware is needed we can give remote access.

Kindly

[Caguicla_Intel]

Hello Pau,

Thank you for posting in Intel Ethernet Communities. Please provide the following details for us to check on your request.

1.) Exact model of the adapter as well as the PBA number. This would help us identify if you are using an OEM or retail version of Intel Ethernet adapter. Please refer to the link below in where to locate the PBA. It consists of 6-digit number located at the last part of the serial number.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-io/ethernet-products.html

2.) What is the operating system used?

3.) Please share the kernel information.

4.) Please share the link where the driver and NVM was downloaded.

Looking forward to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[PNade2]

Hello,

1) this NIC is bought in FS.com, X710 BM2, servers are in production, so I must poweroff and unrack to check this. Can we check remotely? We are at 80km far of servers.

2) Using Proxmox VE 6.1 (Debian Buster 10)

3) Kernel 5.3.13-pve

4) NVM:

https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

Driver:

https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-

kindly,

Pau

[Caguicla_Intel]

Hello Pau,

Thank you for the reply.

Looking at the link you have provided, this doesn't seem to be a Retail version of Intel Ethernet Adapter. With this, we would suggest to contact your manufacturer for the appropriate firmware and driver as this is an OEM adapter. For OEM adapters, the manufacturers might have customized drivers and firmware to adapt to the changes made on their NIC, thus they are in best position to provide the firmware so that you don’t lose features or customizations. Please also be informed that the NVM link you have provided is intended for Intel Retail adapters only. 

Additionally, the X710 BM2 is not mentioned on the List of Supported Retail Intel® Ethernet Adapters

https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-io/ethernet-products.html

Let us know if you have additional questions and clarifications on this request.

Awaiting to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Pau,

Good day!

We'd like to check if you were able to request assistance from your manufacturer. In case you need any other assistance from Intel Ethernet team, please do not hesitate to let us know.

Looking forward to hear from you.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Mike_Intel]

Hello PNade2,

I am just sending another follow up to check if you were able to contact the manufacturer regarding this inquiry. 

Since we have not heard back from you, I will close this inquiry now. 

If you need further assistance, please post a new question. 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel