X550-T2: "Detected Tx Unit Hang" when trying to run DPDK Hello World

[AyaanAnwar]

I have three Intel X550-T2 NICs (https://ark.intel.com/content/www/us/en/ark/products/88209/intel-ethernet-converged-network-adapter-x550-t2.html). I am able to successfully setup and run DPDK sample applications (https://doc.dpdk.org/guides/sample_app_ug/hello_world.html) on only one of them.

For the other two I am seeing the following dmesg prints when the application is run with a connected link partner:

[ 460.459718] ixgbe 0000:01:00.0 eth1: Detected Tx Unit Hang
Tx Queue <0>
TDH, TDT <4>, <4>
next_to_use <4>
next_to_clean <0>
tx_buffer_info[next_to_clean]
time_stamp <100009941>
jiffies <100009b38>
[ 460.500143] ixgbe 0000:01:00.0 eth1: tx hang 7 detected on queue 0, resetting adapter
[ 460.500147] ixgbe 0000:01:00.0 eth1: initiating reset due to tx timeout
[ 460.500160] ixgbe 0000:01:00.0 eth1: Reset adapter
[ 460.500175] ixgbe 0000:01:00.0 eth1: Detected Tx Unit Hang
Tx Queue <2>
TDH, TDT <a>, <a>
next_to_use <a>
next_to_clean <0>
tx_buffer_info[next_to_clean]
time_stamp <10000993e>
jiffies <100009b42>
[ 460.546211] ixgbe 0000:01:00.0 eth1: tx hang 8 detected on queue 2, resetting adapter

This results in interface reset:

[ 467.253702] ixgbe 0000:01:00.0 eth1: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[ 468.819971] ixgbe 0000:01:00.0 eth1: VF Reset msg received from vf 0
[ 468.830477] ixgbe 0000:01:00.0 eth1: VF 0 requested invalid api version 6
[ 469.836568] ixgbevf 0000:02:10.0 eth4: Could not remove VLAN 0
[ 469.843292] ixgbevf 0000:02:10.0 eth4: failed to kill vid 0081/0

Shortly after that, seeing the following stack trace (no kernel panic) on dmesg:

[ 483.275366] ------------[ cut here ]------------
[ 483.275382] NETDEV WATCHDOG: eth1 (ixgbe): transmit queue 2 timed out 8676 ms
[ 483.275398] WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x29c/0x2b4
[ 483.275409] Modules linked in: usb_f_uvc videobuf2_vmalloc uvc videobuf2_dma_sg videobuf2_memops bnep tz_log hci_uart cfg80211 btqca bluetooth qcom_smcinvoke 8021q ecdh_generic ecc rfkill garp mrp stp llc overlay iris_vpu(O) v4l2_mem2mem msm_drm(O) videobuf2_v4l2 fastrpc pmic_glink_altmode videodev anx7625 pmic_glink qrtr_smd videobuf2_common drm_display_helper rpmsg_ctrl drm_dp_aux_bus drm_kms_helper rpmsg_char mc stm_p_ost typec marvell qcom_spmi_mbg_tm qcom_spmi_adc5_gen3 rtc_pm8xxx stm_heartbeat nvmem_qcom_spmi_sdam videocc_sa8775p camcc_sa8775p dispcc0_sa8775p spi_geni_qcom phy_qcom_sgmii_eth i2c_qcom_geni stm_ftrace icc_bwmon stm_console msm_kgsl(O) nt36xxx_i2c(O) panel_event_notifier dwmac_qcom_ethqos stmmac_platform ipa qcom_q6v5_pas stmmac qcom_pil_info qcrypto llcc_qcom qcom_q6v5 gpucc_sa8775p pci_pwrctl_pwrseq sha256_generic pcs_xpcs authenc qcom_sysmon pinctrl_sa8775p_lpass_lpi pci_pwrctl_core phylink libdes pinctrl_lpass_lpi qcom_common dispcc1_sa8775p qcom_rng qcom_glink_smem pwrseq_qcom_wcn
[ 483.275488] snd_soc_qcm6490 snd_soc_qcom_sdw mdt_loader pwrseq_core socinfo snd_soc_qcom_common nvmem_reboot_mode coresight_tpdm coresight_tpda coresight_tmc usb_f_qdss coresight_tgu coresight_stm stm_core coresight_replicator qrtr coresight_remote_etm coresight_funnel coresight_etm4x coresight_csr coresight_dummy coresight_cti coresight zram bt_fm_slim(O) zsmalloc slimbus at803x drm fuse ipv6 qcom_pon
[ 483.275519] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W O 6.6.33-debug #1
[ 483.275522] Hardware name: Qualcomm SA8775P Ride (DT)
[ 483.275523] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 483.275526] pc : dev_watchdog+0x29c/0x2b4
[ 483.275528] lr : dev_watchdog+0x29c/0x2b4
[ 483.275530] sp : ffffffc08002bde0
[ 483.275531] x29: ffffffc08002bde0 x28: ffffffda688489c0 x27: ffffffda682b5008
[ 483.275534] x26: 0000000000000000 x25: 00000000000021e4 x24: ffffffda68848000
[ 483.275537] x23: 0000000000000002 x22: ffffff80a89f03e0 x21: ffffff80a89f0000
[ 483.275540] x20: ffffff80b9368280 x19: ffffff80a89f04a0 x18: ffffffffffffffff
[ 483.275542] x17: 6d20363736382074 x16: 756f2064656d6974 x15: 2032206575657571
[ 483.275545] x14: 2074696d736e6172 x13: 00000000000016b9 x12: 00000000ffffffea
[ 483.275548] x11: 00000000ffffdfff x10: 00000000ffffdfff x9 : ffffffda6891eec0
[ 483.275551] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 0000000005000000
[ 483.275554] x5 : 0000000000000006 x4 : 0000000000000040 x3 : 0000000000000001
[ 483.275556] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff80a427b000
[ 483.275559] Call trace:
[ 483.275560] dev_watchdog+0x29c/0x2b4
[ 483.275562] call_timer_fn+0x34/0x1c0
[ 483.275567] __run_timers.part.0+0x224/0x2f0
[ 483.275569] run_timer_softirq+0x3c/0x74
[ 483.275571] handle_softirqs+0x118/0x36c
[ 483.275574] __do_softirq+0x14/0x20
[ 483.275576] ____do_softirq+0x10/0x1c
[ 483.275579] call_on_irq_stack+0x24/0x4c
[ 483.275581] do_softirq_own_stack+0x1c/0x28
[ 483.275583] irq_exit_rcu+0xbc/0xd8
[ 483.275585] el1_interrupt+0x38/0x64
[ 483.275588] el1h_64_irq_handler+0x18/0x24
[ 483.275590] el1h_64_irq+0x68/0x6c
[ 483.275591] default_idle_call+0x5c/0x160
[ 483.275594] do_idle+0x220/0x27c
[ 483.275597] cpu_startup_entry+0x34/0x3c
[ 483.275598] secondary_start_kernel+0x12c/0x14c
[ 483.275602] __secondary_switched+0xb8/0xbc
[ 483.275605] ---[ end trace 0000000000000000 ]---
[ 483.275615] ixgbe 0000:01:00.0 eth1: initiating reset due to tx timeout
[ 483.275637] ixgbe 0000:01:00.0 eth1: Reset adapter

Also, the application prints out the following error:

EAL: Probe PCI driver: net_ixgbe_vf (8086:1565) device: 0000:02:10.0 (socket -1)
eth_ixgbevf_dev_init(): VF Initialization Failure: -15,: ../drivers/net/ixgbe/ixgbe_ethdev.c: 1683

(The error code is not always -15, I have seen other values during the same procedure as well).

If it helps, I have tried finding out the difference between the working and non-working NICs. Here's what I have found till now:

1. The working NIC has a Yottamark hologram on the back while the 2 non-working ones don't have one.

2. I also captured the firmware versions of the working and non-working cards using ethtool -i <interface>:

[Working NIC] firmware-version: 0x80000492
[Non-Working NIC] firmware-version: 0x80000dd2, 1.1276.0

Is the issue happening due to the NICs being very old or having non-compatible firmware? I would appreciate if I could get some pointers to troubleshoot this issue.

[Devi_Intel]

Hello AyaanAnwar

Greetings ~

Please share the details of your system drivers and ensure they are updated to the latest versions.

For further support with DPDK, kindly refer to Home - DPDK or open an IPS case at  IPS Support

After registration, you will be able to open a case on the Intel® Premier Support (IPS) platform, and your request will be handled by one of our engineers as soon as possible.

Thank you & Best Regards,

Devi

[Devi_Intel]

Hello AyaanAnwar

Greetings ~

we are following up to find out if you were able to complete the actions we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. We will follow up after 3 business days in case we don't receive a response we shall proceed to close this case.

Thank you & Best Regards

Devi

[Devi_Intel]

Hello AyaanAnwar

Greetings ~

We are writing to inform you that we will proceed to close your case.

If you have any more questions in the future, please don't hesitate to post a new question, as this thread will no longer be monitored.

Thank you & Best Regards,

Devi