Intel E810-XXVDA4T: Transmit Queue Timed Out

[KylePC]

All,

I am experiencing an issue where a port on my E810-XXVDA4T is experiencing a "transmit queue 96 timed out". I am currently using ice and iavf drivers from the Complete Driver Pack 27_6. 

Below is the dmesg output. I'm trying to understand what causes this issue?

[ 1014.835270] pci 0000:4b:19.0: Removing from iommu group 371
[ 1015.868805] ice 0000:4b:00.3: Enabling 1 VFs
[ 1015.976713] pci 0000:4b:19.0: [8086:1889] type 00 class 0x020000
[ 1015.976736] pci 0000:4b:19.0: enabling Extended Tags
[ 1015.977056] pci 0000:4b:19.0: Adding to iommu group 371
[ 1015.977325] iavf 0000:4b:19.0: enabling device (0000 -> 0002)
[ 1015.977540] ice 0000:4b:00.3: Enabling 1 VFs with 65 vectors and 16 queues per VF
[ 1015.997113] ice 0000:4b:00.3 ens2f3: Setting MAC aa:bb:cc:dd:ee:14 on VF 0. VF driver will be reinitialized
[ 1016.025486] ice 0000:4b:00.3: Setting VLAN 3, QoS 0, TPID 0x8100 on VF 0
[ 1016.115462] ice 0000:4b:00.3: VF 0 is now trusted
[ 1016.123458] iavf 0000:4b:19.0: Removing device
[ 1377.195752] step4-startCU.s (14535): drop_caches: 1
[ 1377.329624] step4-startCU.s (14535): drop_caches: 2
[ 1377.371654] step4-startCU.s (14535): drop_caches: 3
[ 1388.193160] igb_uio 0000:17:00.0: enabling device (0140 -> 0142)
[ 1388.194846] igb_uio 0000:17:00.0: mapping 1K dma=0x16133a000 host=000000009a952fb7
[ 1388.194852] igb_uio 0000:17:00.0: unmapping 1K dma=0x16133a000 host=000000009a952fb7
[ 1390.726324] igb_uio 0000:b3:01.0: uio device registered with irq 2928
[ 1390.930815] igb_uio 0000:b3:01.1: uio device registered with irq 2929
[ 1393.005158] sctp: Hash tables configured (bind 1024/1638)
[ 1397.180714] igb_uio 0000:17:00.0: uio device registered with irq 2930
[ 1397.366730] vfio-pci 0000:4b:19.0: enabling device (0000 -> 0002)
[ 1533.272386] ------------[ cut here ]------------
[ 1533.272392] NETDEV WATCHDOG: ens2f3 (ice): transmit queue 96 timed out
[ 1533.272424] WARNING: CPU: 32 PID: 310 at net/sched/sch_generic.c:477 dev_watchdog+0x2cd/0x2e0
[ 1533.272438] Modules linked in: sctp ip6_udp_tunnel udp_tunnel iavf(OE) igb_uio(OE) vtsspp(OE) intel_rapl_msr intel_rapl_common sep5(OE) i10nm_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp socperf3(OE) msr kvm_intel usdm_drv(OE) pax(OE) snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio ipmi_ssif binfmt_misc kvm rapl nls_iso8859_1 snd_hda_intel intel_cstate snd_intel_dspcfg qat_c62x(OE) snd_intel_sdw_acpi intel_qat(OE) snd_hda_codec uio isst_if_mbox_pci joydev snd_hda_core input_leds snd_hwdep snd_pcm snd_timer i40e snd isst_if_mmio isst_if_common ib_uverbs soundcore ib_core intel_pch_thermal mei_me mei ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid ses enclosure
[ 1533.272553] scsi_transport_sas ast drm_vram_helper crct10dif_pclmul drm_ttm_helper crc32_pclmul ttm drm_kms_helper syscopyarea ghash_clmulni_intel sysfillrect sysimgblt fb_sys_fops aesni_intel cec crypto_simd cryptd rc_core ahci megaraid_sas ice(OE) igb i2c_i801 xhci_pci dca drm i2c_smbus i2c_algo_bit libahci xhci_pci_renesas intel_pmt wmi
[ 1533.272872] CPU: 32 PID: 310 Comm: ksoftirqd/32 Tainted: G OE 5.15.0-1009-realtime #9-Ubuntu
[ 1533.272875] Hardware name: Insyde Whitley/Type2 - Board Product Name1, BIOS 05.42.23.0063 01/05/2022
[ 1533.272876] RIP: 0010:dev_watchdog+0x2cd/0x2e0
[ 1533.272880] Code: ff ff 48 8b 5d d0 c6 05 f3 b9 6e 01 01 48 89 df e8 68 a5 f9 ff 44 89 e1 48 89 de 48 c7 c7 08 2b 4a 8c 48 89 c2 e8 6c 9a 18 00 <0f> 0b e9 27 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44
[ 1533.272882] RSP: 0018:ff3525f90d597d58 EFLAGS: 00010282
[ 1533.272884] RAX: 0000000000000000 RBX: ff2411c0e6e3f000 RCX: 0000000000000027
[ 1533.272885] RDX: 0000000000000001 RSI: 80000000ffff8c11 RDI: ff2411efcee20a88
[ 1533.272886] RBP: ff3525f90d597d90 R08: ff2411efcee20a80 R09: 0000000000000001
[ 1533.272887] R10: ff3525f90d597c58 R11: ff2411efced33622 R12: 0000000000000060
[ 1533.272888] R13: ff2411c0e9c3ef40 R14: 00000000000000c0 R15: ff2411c0e6e3f4e0
[ 1533.272889] FS: 0000000000000000(0000) GS:ff2411efcee00000(0000) knlGS:0000000000000000
[ 1533.272890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1533.272891] CR2: 00007fdc96dc9a20 CR3: 00000018b8792003 CR4: 0000000000771ea0
[ 1533.272892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1533.272893] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1533.272894] PKRU: 55555554
[ 1533.272895] Call Trace:
[ 1533.272896] <TASK>
[ 1533.272898] ? pfifo_fast_reset+0x130/0x130
[ 1533.272900] call_timer_fn+0x29/0x190
[ 1533.272904] __run_timers.part.0+0x23f/0x340
[ 1533.272905] ? pfifo_fast_reset+0x130/0x130
[ 1533.272907] ? psi_task_switch+0x1d3/0x1f0
[ 1533.272910] ? sched_clock+0x9/0x10
[ 1533.272913] run_timer_softirq+0x2a/0x50
[ 1533.272914] __do_softirq+0xc6/0x2c8
[ 1533.272918] run_ksoftirqd+0x30/0x80
[ 1533.272922] smpboot_thread_fn+0x1d0/0x2d0
[ 1533.272924] kthread+0x18e/0x1b0
[ 1533.272927] ? smpboot_register_percpu_thread+0xe0/0xe0
[ 1533.272929] ? set_kthread_struct+0x50/0x50
[ 1533.272931] ret_from_fork+0x1f/0x30
[ 1533.272936] </TASK>
[ 1533.272936] ---[ end trace 0000000000000002 ]---
[ 1534.279092] ice 0000:4b:00.3 ens2f3: tx_timeout: VSI_num: 18, Q 96, NTC: 0x4, HW_HEAD: 0x6, NTU: 0x7, INT: 0x0
[ 1534.279096] ice 0000:4b:00.3 ens2f3: tx_timeout recovery level 1, txqueue 96

[Irwan_Intel]

Hi KylePC,

Thank you for posting in Intel Ethernet Communities. 

For us to further check and investigate the issue, please provide the following details.

1. Ubuntu and kernel version?

2. What is the brand and model of your board or system?

Please try updating the driver to the latest and see if the error still persist.

https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html

In case we do not hear from you, we will make a follow up after 3 workings days.

Regards,

Irwan_Intel

Intel Customer Support

[Irwan_Intel]

Hi KylePC,

Just a follow up to see if you have had a chance to gather all the information as needed or do you have any concerns or further questions regarding the case?

Feel free to contact us if you need any further assistance, and we will be more than glad to continue assisting you.

Regards,

Irwan_Intel

Intel Customer Support

[Irwan_Intel]

Hi KylePC,

We would like to inform you that due to a lack of response to our previous follow-ups, we will be closing this request. If you have any further inquiries in the future, please don't hesitate to post a new question, as this thread will no longer be monitored.

Best regards,

Irwan_Intel

Intel® Customer Support