Re:Intel X710-4 NETDEV WATCHDOG: eth5 (i40e): transmit queue 4 timed out

AndriiV · ‎02-07-2023

Hello! I have a problem with X710-4 ethernet card. When I add 2 ports to existing bond or make new bond, nothing works. I tried latest firmware 9.10 and latest driver 2.22.8, then rollback to 6.8 firmware and driver 2.7.12 but nothing works too. What to do?

Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Converged 6.128(6.80) 1572 00:059 Up to date
Network Adapter X710-4

Feb 6 15:44:18 152224 kernel: [81835.868123] bond0: Adding slave eth4
Feb 6 15:44:18 152224 kernel: [81835.868131] i40e 0000:3b:00.2 eth4: set new mac address a0:36:9f:13:f4:2c
Feb 6 15:44:18 152224 kernel: [81835.880536] bond0: Enslaving eth4 as a backup interface with an up link
Feb 6 15:44:18 152224 kernel: [81836.025903] bond0: Adding slave eth5
Feb 6 15:44:18 152224 kernel: [81836.025915] i40e 0000:3b:00.3 eth5: set new mac address a0:36:9f:13:f4:2c
Feb 6 15:44:18 152224 kernel: [81836.108958] bond0: Enslaving eth5 as a backup interface with an up link
Feb 6 15:44:18 152224 kernel: [81836.145694] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb 6 15:44:18 152224 kernel: [81836.353706] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb 6 15:44:30 152224 kernel: [81847.697538] ------------[ cut here ]------------
Feb 6 15:44:30 152224 kernel: [81847.697547] WARNING: CPU: 30 PID: 0 at /build/linux-dqnRSc/linux-4.9.228/net/sched/sch_generic.c:316 dev_watchdog+0x233/0x240
Feb 6 15:44:30 152224 kernel: [81847.697548] NETDEV WATCHDOG: eth5 (i40e): transmit queue 4 timed out
Feb 6 15:44:30 152224 kernel: [81847.697549] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport iptable_filter sch_htb ipmi_devintf iTCO_wdt iTCO_vendor_support intel_rapl skx_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i40e(O) intel_uncore raid0 intel_rapl_perf ast ixgbe ttm pcspkr mdio drm_kms_helper intel_auxiliary(O) drm ptp pps_core i2c_algo_bit i2c_i801 mei_me lpc_ich joydev evdev ioatdma sg i2c_smbus mfd_core shpchp mei dca ipmi_si ipmi_msghandler acpi_power_meter acpi_pad acpi_cpufreq button tcp_bbr sch_fq bonding ip_tables x_tables ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache raid1 hid_generic usbhid md_mod hid sd_mod crc32c_intel aes_x86_64 nvme nvme_core xhci_pci
Feb 6 15:44:30 152224 kernel: [81847.697594] ahci libahci xhci_hcd libata usbcore scsi_mod usb_common autofs4
Feb 6 15:44:30 152224 kernel: [81847.697602] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G O 4.9.0-13-amd64 #1 Debian 4.9.228-1
Feb 6 15:44:30 152224 kernel: [81847.697603] Hardware name: Supermicro SYS-229U-TN24R4T-2-IN001/X11DPU, BIOS 3.8a 10/28/2022
Feb 6 15:44:30 152224 kernel: [81847.697604] 0000000000000000 ffffffffb49376de ffff8fd6bd583e20 0000000000000000
Feb 6 15:44:30 152224 kernel: [81847.697607] ffffffffb467b99b 0000000000000004 ffff8fd6bd583e78 ffff8fb6adfc3000
Feb 6 15:44:30 152224 kernel: [81847.697608] 000000000000001e ffff8fb69628cf40 0000000000000040 ffffffffb467ba1f
Feb 6 15:44:30 152224 kernel: [81847.697610] Call Trace:
Feb 6 15:44:30 152224 kernel: [81847.697612] <IRQ>
Feb 6 15:44:30 152224 kernel: [81847.697618] [<ffffffffb49376de>] ? dump_stack+0x66/0x88
Feb 6 15:44:30 152224 kernel: [81847.697622] [<ffffffffb467b99b>] ? __warn+0xcb/0xf0
Feb 6 15:44:30 152224 kernel: [81847.697623] [<ffffffffb467ba1f>] ? warn_slowpath_fmt+0x5f/0x80
Feb 6 15:44:30 152224 kernel: [81847.697626] [<ffffffffb4b3f983>] ? dev_watchdog+0x233/0x240
Feb 6 15:44:30 152224 kernel: [81847.697627] [<ffffffffb4b3f750>] ? dev_deactivate_queue.constprop.26+0x60/0x60
Feb 6 15:44:30 152224 kernel: [81847.697630] [<ffffffffb46ea672>] ? call_timer_fn+0x32/0x120
Feb 6 15:44:30 152224 kernel: [81847.697632] [<ffffffffb46ea9e3>] ? run_timer_softirq+0x1d3/0x410
Feb 6 15:44:30 152224 kernel: [81847.697635] [<ffffffffb46fbb20>] ? tick_sched_do_timer+0x30/0x30
Feb 6 15:44:30 152224 kernel: [81847.697637] [<ffffffffb4940894>] ? timerqueue_add+0x54/0xa0
Feb 6 15:44:30 152224 kernel: [81847.697638] [<ffffffffb46ec6b8>] ? enqueue_hrtimer+0x38/0x80
Feb 6 15:44:30 152224 kernel: [81847.697641] [<ffffffffb4c22f6d>] ? __do_softirq+0x10d/0x2b0
Feb 6 15:44:30 152224 kernel: [81847.697643] [<ffffffffb4681fb2>] ? irq_exit+0xc2/0xd0
Feb 6 15:44:30 152224 kernel: [81847.697644] [<ffffffffb4c229ec>] ? smp_apic_timer_interrupt+0x4c/0x60
Feb 6 15:44:30 152224 kernel: [81847.697645] [<ffffffffb4c2111e>] ? apic_timer_interrupt+0x9e/0xb0
Feb 6 15:44:30 152224 kernel: [81847.697645] <EOI>
Feb 6 15:44:30 152224 kernel: [81847.697650] [<ffffffffb4aded02>] ? cpuidle_enter_state+0xa2/0x2d0
Feb 6 15:44:30 152224 kernel: [81847.697651] [<ffffffffb4adecf0>] ? cpuidle_enter_state+0x90/0x2d0
Feb 6 15:44:30 152224 kernel: [81847.697654] [<ffffffffb46bf784>] ? cpu_startup_entry+0x154/0x240
Feb 6 15:44:30 152224 kernel: [81847.697658] [<ffffffffb464af00>] ? start_secondary+0x170/0x1b0
Feb 6 15:44:30 152224 kernel: [81847.697659] ---[ end trace dbc12ce096b7c36f ]---
Feb 6 15:44:30 152224 kernel: [81847.697665] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 4, NTC: 0x57, HWB: 0x57, NTU: 0x86, TAIL: 0x86, INT: 0x1
Feb 6 15:44:30 152224 kernel: [81847.697666] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 1, hung_queue 4
Feb 6 15:44:30 152224 kernel: [81847.705687] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb 6 15:44:30 152224 kernel: [81847.709534] bond0: link status down for interface eth5, disabling it in 200 ms
Feb 6 15:44:30 152224 kernel: [81847.825288] i40e 0000:3b:00.3: VSI seid 402 Tx ring 383 disable timeout
Feb 6 15:44:30 152224 kernel: [81847.957122] i40e 0000:3b:00.3: VF BW shares not restored
Feb 6 15:44:30 152224 kernel: [81847.961533] bond0: link status up again after 200 ms for interface eth5
Feb 6 15:44:35 152224 kernel: [81852.817560] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 7, NTC: 0xaf, HWB: 0xaf, NTU: 0xb5, TAIL: 0xb5, INT: 0x1
Feb 6 15:44:35 152224 kernel: [81852.817567] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 1, hung_queue 7
Feb 6 15:44:35 152224 kernel: [81852.835690] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb 6 15:44:35 152224 kernel: [81852.853527] bond0: link status down for interface eth4, disabling it in 200 ms
Feb 6 15:44:35 152224 kernel: [81852.904562] i40e 0000:3b:00.2: VSI seid 401 Tx ring 383 disable timeout
Feb 6 15:44:35 152224 kernel: [81852.983163] i40e 0000:3b:00.2: VF BW shares not restored
Feb 6 15:44:35 152224 kernel: [81853.061533] bond0: link status up again after 200 ms for interface eth4
Feb 6 15:44:41 152224 kernel: [81858.705556] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 0, NTC: 0x81, HWB: 0x81, NTU: 0x8c, TAIL: 0x8c, INT: 0x0
Feb 6 15:44:41 152224 kernel: [81858.705565] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 2, hung_queue 0
Feb 6 15:44:41 152224 kernel: [81858.705635] i40e 0000:3b:00.1: unhandled interrupt icr0=0x00000001
Feb 6 15:44:41 152224 kernel: [81858.936668] i40e 0000:3b:00.0: VF BW shares not restored
Feb 6 15:44:41 152224 kernel: [81858.990495] i40e 0000:3b:00.1: VF BW shares not restored
Feb 6 15:44:46 152224 kernel: [81863.825532] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 0, NTC: 0x19, HWB: 0x19, NTU: 0x78, TAIL: 0x0, INT: 0x1
Feb 6 15:44:46 152224 kernel: [81863.825534] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 2, hung_queue 0
Feb 6 15:47:06 152224 kernel: [82003.903578] bond0: Removing slave eth4
Feb 6 15:47:06 152224 kernel: [82003.903959] bond0: Releasing backup interface eth4
Feb 6 15:47:06 152224 kernel: [82004.184317] i40e 0000:3b:00.2: ignoring delete macvlan error on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:06 152224 kernel: [82004.250612] i40e 0000:3b:00.2 eth4: returning to hw mac address 3c:fd:fe:9d:2f:12
Feb 6 15:47:07 152224 kernel: [82004.501640] i40e 0000:3b:00.2 eth4: Ignoring error from firmware on LAA update, status I40E_ERR_ADMIN_QUEUE_TIMEOUT, AQ ret OK
Feb 6 15:47:07 152224 kernel: [82004.508788] bond0: Removing slave eth5
Feb 6 15:47:07 152224 kernel: [82004.755160] i40e 0000:3b:00.2: ignoring delete macvlan error on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:07 152224 kernel: [82004.755185] bond0: Releasing backup interface eth5
Feb 6 15:47:07 152224 kernel: [82005.008909] i40e 0000:3b:00.2: Error OK adding RX filters on PF, promiscuous mode forced on
Feb 6 15:47:07 152224 kernel: [82005.262100] i40e 0000:3b:00.2: Set default VSI failed, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:07 152224 kernel: [82005.262104] i40e 0000:3b:00.2: Setting promiscuous on failed on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT aq_err OK
Feb 6 15:47:08 152224 kernel: [82005.546867] i40e 0000:3b:00.3: ignoring delete macvlan error on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:08 152224 kernel: [82005.678454] i40e 0000:3b:00.3 eth5: returning to hw mac address 3c:fd:fe:9d:2f:13
Feb 6 15:47:08 152224 kernel: [82005.929872] i40e 0000:3b:00.3 eth5: Ignoring error from firmware on LAA update, status I40E_ERR_ADMIN_QUEUE_TIMEOUT, AQ ret OK
Feb 6 15:47:08 152224 kernel: [82006.183826] i40e 0000:3b:00.3: ignoring delete macvlan error on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:08 152224 kernel: [82006.438235] i40e 0000:3b:00.3: Error OK adding RX filters on PF, promiscuous mode forced on
Feb 6 15:47:09 152224 kernel: [82006.692105] i40e 0000:3b:00.3: Set default VSI failed, err I40E_ERR_ADMIN_QUEUE_TIMEOUT, aq_err OK
Feb 6 15:47:09 152224 kernel: [82006.692108] i40e 0000:3b:00.3: Setting promiscuous on failed on PF, err I40E_ERR_ADMIN_QUEUE_TIMEOUT aq_err OK

IntelSupport · ‎02-07-2023

Hi AndriiV,

Thank you for posting in Intel Ethernet Communities.

In order to better assist you, we recommend for you to provide below information:

1.) What operating system are you using for NIC bonding?

2.) Does the issue occurs with the bonding with onboard NICs?

Best regards,

Fariz_Intel

AndriiV · ‎02-08-2023

Debian 9.13

Other 2 NIC's are Intel(R) Ethernet 10G 2P X520 Adapter 154D 00:216
For connection with switch using QSFP+ to 4 x 10G SFP+ Passive Direct Attach Copper Breakout Cable

# ethtool -m eth0
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x21 (Copper pigtail)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
Transceiver type : Passive Cable
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 3m
Length (OM3) : 0m
Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]
Vendor name : FS
Vendor OUI : 64:9d:99
Vendor PN : Q-4SPC03
Vendor rev : D
Option values : 0x00 0x12
Option : RX_LOS implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : C2301344553-1
Date code : 230201

# ethtool -m eth4
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x21 (Copper pigtail)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
Transceiver type : Passive Cable
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 3m
Length (OM3) : 0m
Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]
Vendor name : FS
Vendor OUI : 64:9d:99
Vendor PN : Q-4SPC03
Vendor rev : D
Option values : 0x00 0x12
Option : RX_LOS implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : C2301344553-4
Date code : 230201

When I add 2 ports of X710 NIC to bonding

# ifenslave bond0 eth4 eth5

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: a0:36:9f:13:f4:2c
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 13
Partner Key: 8
Partner Mac Address: 88:e6:4b:6d:d5:24

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a0:36:9f:13:f4:2c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: a0:36:9f:13:f4:2c
port key: 13
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 127
system mac address: 88:e6:4b:6d:d5:24
oper key: 8
port priority: 127
port number: 53
port state: 63

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: a0:36:9f:13:f4:2e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: a0:36:9f:13:f4:2c
port key: 13
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 127
system mac address: 88:e6:4b:6d:d5:24
oper key: 8
port priority: 127
port number: 52
port state: 63

Slave Interface: eth4
MII Status: down
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 7
Permanent HW addr: 3c:fd:fe:9d:2f:12
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: a0:36:9f:13:f4:2c
port key: 0
port priority: 255
port number: 3
port state: 63
details partner lacp pdu:
system priority: 127
system mac address: 88:e6:4b:6d:d5:24
oper key: 8
port priority: 127
port number: 45
port state: 63

Slave Interface: eth5
MII Status: down
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 5
Permanent HW addr: 3c:fd:fe:9d:2f:13
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: a0:36:9f:13:f4:2c
port key: 0
port priority: 255
port number: 4
port state: 63
details partner lacp pdu:
system priority: 127
system mac address: 88:e6:4b:6d:d5:24
oper key: 8
port priority: 127
port number: 44
port state: 63

IntelSupport · ‎02-08-2023

Hi AndriiV,

Thank you for the response.

1) Have you try remove bonding, restart system and recreate the bond?

2)As per ethool information that you share, is the port status that show down is only at X710 NIC port?

3)Are using ethtool version 1.6 as per below link?

http://sourceforge.net/projects/gkernel.

We have notice that in Intel® Ethernet Product Software Release Notes, its show that Debian 11 is supported but not tested (SNT). It does not have much information for lower version than Debian 11 which might not been tested also. You may refer link below:

https://www.intel.com/content/www/us/en/download/19622/intel-ethernet-product-software-release-notes.html

We also notice similar case on redhat forum:

https://access.redhat.com/solutions/4054441

AndriiV · ‎02-09-2023

1. Yes, I tried restart system and create bonding only with this 2 ports X710 NIC - no network

2. Yes, after few seconds bond turn off X710 NIC

Feb  6 16:29:02 152224 kernel: [  625.430512] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  6 16:29:02 152224 kernel: [  625.462775] bond0: link status definitely down for interface eth4, disabling it
Feb  6 16:29:02 152224 kernel: [  625.462914] bond0: link status definitely down for interface eth5, disabling it

3. I am using Debian package ethtool

# LANG=C apt-cache policy ethtool
ethtool:
  Installed: 1:4.19-1
  Candidate: 1:4.19-1
  Version table:
 *** 1:4.19-1 500
        500 http://ftp.us.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

And the similar case is https://bugzilla.kernel.org/show_bug.cgi?id=197325

AndriiV · ‎02-09-2023

Now I tried with recomendation

ethtool -K eth4 tso off
ethtool -K eth5 tso off

in atop such picture:

NET | eth1 20%  | pcki   91779  | pcko  356873  | sp   10 Gbps  | si  202 Mbps  | so 2078 Mbps  | | coll 0  | mlti 2  | erri 0  | erro 0  | drpi 0  | drpo 0  |
NET | eth0 20%  | pcki   86600  | pcko  354871  | sp   10 Gbps  | si  164 Mbps  | so 2067 Mbps  | | coll 0  | mlti 2  | erri 0  | erro 0  | drpi 0  | drpo 0  |
NET | bond0 10%  | pcki  347666  | pcko  715227  | sp   40 Gbps  | si  735 Mbps  | so 4159 Mbps  | | coll 0  | mlti 8  | erri 0  | erro 0  | drpi 0  | drpo 0  |
NET | eth5 2%  | pcki   90581  | pcko    1846  | sp   10 Gbps  | si  235 Mbps  | so 6433 Kbps  | | coll 0  | mlti 2  | erri 0  | erro 0  | drpi 0  | drpo 0  |
NET | eth4 1%  | pcki   78709  | pcko    1625  | sp   10 Gbps  | si  133 Mbps  | so 6461 Kbps  | | coll 0  | mlti 2  | erri 0  | erro 0  | drpi 0  | drpo 0  |

In messages:

Feb  9 11:50:12 152224 kernel: [243095.138262] bond0: Adding slave eth4
Feb  9 11:50:12 152224 kernel: [243095.138275] i40e 0000:3b:00.2 eth4: set new mac address a0:36:9f:13:f4:2c
Feb  9 11:50:12 152224 kernel: [243095.161365] bond0: Enslaving eth4 as a backup interface with an up link
Feb  9 11:50:12 152224 kernel: [243095.162716] bond0: Adding slave eth5
Feb  9 11:50:12 152224 kernel: [243095.162721] i40e 0000:3b:00.3 eth5: set new mac address a0:36:9f:13:f4:2c
Feb  9 11:50:12 152224 kernel: [243095.187859] bond0: Enslaving eth5 as a backup interface with an up link
Feb  9 11:50:13 152224 kernel: [243095.381086] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 11:50:13 152224 kernel: [243095.485137] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 11:50:34 152224 kernel: [243116.772905] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 0, NTC: 0xc0, HWB: 0xc0, NTU: 0xc2, TAIL: 0xc2, INT: 0x0
Feb  9 11:50:34 152224 kernel: [243116.772912] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 1, hung_queue 0
Feb  9 11:50:34 152224 kernel: [243116.773338] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 11:50:34 152224 kernel: [243116.824896] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 11:50:34 152224 kernel: [243116.833665] i40e 0000:3b:00.3: VSI seid 403 Tx ring 64 disable timeout
Feb  9 11:50:34 152224 kernel: [243117.032899] bond0: link status up again after 200 ms for interface eth5
Feb  9 11:50:44 152224 kernel: [243126.756903] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 0, NTC: 0xaf, HWB: 0xaf, NTU: 0xec, TAIL: 0xec, INT: 0x1
Feb  9 11:50:44 152224 kernel: [243126.756905] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 2, hung_queue 0
Feb  9 11:50:44 152224 kernel: [243126.808053] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 11:50:44 152224 kernel: [243126.812948] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 11:50:44 152224 kernel: [243126.870883] i40e 0000:3b:00.2: VSI seid 402 Tx ring 64 disable timeout
Feb  9 11:50:44 152224 kernel: [243126.920891] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 11:50:44 152224 kernel: [243126.981431] i40e 0000:3b:00.3: VSI seid 403 Tx ring 64 disable timeout
Feb  9 11:50:44 152224 kernel: [243127.020886] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 11:50:44 152224 kernel: [243127.089055] bond0: link status definitely down for interface eth4, disabling it
Feb  9 11:50:44 152224 kernel: [243127.089227] bond0: link status definitely down for interface eth5, disabling it
Feb  9 11:50:47 152224 kernel: [243130.012885] bond0: link status up for interface eth5, enabling it in 200 ms
Feb  9 11:50:47 152224 kernel: [243130.120887] bond0: link status up for interface eth4, enabling it in 200 ms
Feb  9 11:50:47 152224 kernel: [243130.225083] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 11:50:48 152224 kernel: [243130.329016] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 11:50:58 152224 kernel: [243140.836872] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 0, NTC: 0x9f, HWB: 0x9f, NTU: 0xad, TAIL: 0xad, INT: 0x1
Feb  9 11:50:58 152224 kernel: [243140.836874] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 3, hung_queue 0
Feb  9 11:50:58 152224 kernel: [243140.836887] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 0, NTC: 0xa0, HWB: 0xa0, NTU: 0xc2, TAIL: 0xc2, INT: 0x1
Feb  9 11:50:58 152224 kernel: [243140.836888] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 1, hung_queue 0
Feb  9 11:50:58 152224 kernel: [243140.887037] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 11:50:58 152224 kernel: [243140.946638] i40e 0000:3b:00.3: VSI seid 402 Tx ring 64 disable timeout
Feb  9 11:50:58 152224 kernel: [243140.952871] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 11:50:58 152224 kernel: [243140.996782] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 11:50:58 152224 kernel: [243141.056872] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 11:50:58 152224 kernel: [243141.057397] i40e 0000:3b:00.2: VSI seid 403 Tx ring 64 disable timeout
Feb  9 11:50:58 152224 kernel: [243141.161041] bond0: link status definitely down for interface eth5, disabling it
Feb  9 11:50:58 152224 kernel: [243141.265055] bond0: link status definitely down for interface eth4, disabling it
Feb  9 11:51:01 152224 kernel: [243144.129708] i40e 0000:3b:00.3 eth5: NIC Link is Down
Feb  9 11:51:01 152224 kernel: [243144.167682] i40e 0000:3b:00.2 eth4: NIC Link is Down
Feb  9 11:51:02 152224 kernel: [243144.756472] i40e 0000:3b:00.2 eth4: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
Feb  9 11:51:02 152224 kernel: [243144.812868] bond0: link status up for interface eth4, enabling it in 200 ms
Feb  9 11:51:02 152224 kernel: [243144.860314] i40e 0000:3b:00.3 eth5: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
Feb  9 11:51:02 152224 kernel: [243144.920868] bond0: link status up for interface eth5, enabling it in 200 ms
Feb  9 11:51:02 152224 kernel: [243145.024996] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 11:51:02 152224 kernel: [243145.133017] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 11:51:15 152224 kernel: [243157.732856] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 5, NTC: 0xb8, HWB: 0xb8, NTU: 0xbc, TAIL: 0xbc, INT: 0x1
Feb  9 11:51:15 152224 kernel: [243157.732857] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 2, hung_queue 5
Feb  9 11:51:15 152224 kernel: [243157.788665] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 11:51:15 152224 kernel: [243157.840852] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 11:51:15 152224 kernel: [243157.848826] i40e 0000:3b:00.2: VSI seid 403 Tx ring 64 disable timeout
Feb  9 11:51:15 152224 kernel: [243157.898861] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 11:51:15 152224 kernel: [243157.944853] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 11:51:15 152224 kernel: [243157.959431] i40e 0000:3b:00.3: VSI seid 402 Tx ring 64 disable timeout
Feb  9 11:51:15 152224 kernel: [243158.072998] bond0: link status definitely down for interface eth4, disabling it
Feb  9 11:51:15 152224 kernel: [243158.073158] bond0: link status definitely down for interface eth5, disabling it
Feb  9 11:51:18 152224 kernel: [243160.992850] bond0: link status up for interface eth4, enabling it in 200 ms
Feb  9 11:51:18 152224 kernel: [243161.100853] bond0: link status up for interface eth5, enabling it in 200 ms
Feb  9 11:51:18 152224 kernel: [243161.205021] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 11:51:19 152224 kernel: [243161.309000] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 11:51:26 152224 kernel: [243168.673742] bond0: Removing slave eth4
Feb  9 11:51:26 152224 kernel: [243168.674055] bond0: Releasing backup interface eth4
Feb  9 11:51:26 152224 kernel: [243168.714398] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 11:51:26 152224 kernel: [243168.774914] i40e 0000:3b:00.2 eth4: returning to hw mac address 3c:fd:fe:9d:2f:12
Feb  9 11:51:26 152224 kernel: [243168.779226] bond0: Removing slave eth5
Feb  9 11:51:26 152224 kernel: [243168.779543] bond0: Releasing backup interface eth5
Feb  9 11:51:26 152224 kernel: [243168.791516] i40e 0000:3b:00.2: TX driver issue detected, PF reset issued
Feb  9 11:51:26 152224 kernel: [243168.878459] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 11:51:26 152224 kernel: [243168.942154] i40e 0000:3b:00.3 eth5: returning to hw mac address 3c:fd:fe:9d:2f:13
Feb  9 11:51:26 152224 kernel: [243168.942644] i40e 0000:3b:00.2: VSI seid 401 Tx ring 64 disable timeout
Feb  9 11:51:26 152224 kernel: [243169.034222] i40e 0000:3b:00.3: TX driver issue detected, PF reset issued
Feb  9 11:51:26 152224 kernel: [243169.034476] i40e 0000:3b:00.3: VSI seid 402 Tx ring 64 disable timeout

AndriiV · ‎02-09-2023

I just installed new kernel and updated for latest driver and firmware:

Feb  9 17:17:31 152224 kernel: [ 1320.401110] i40e 0000:3b:00.0: PTP external clock not supported.
Feb  9 17:17:31 152224 kernel: [ 1320.401176] i40e 0000:3b:00.1: PTP external clock not supported.
Feb  9 17:17:31 152224 kernel: [ 1320.401233] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:31 152224 kernel: [ 1320.401317] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.891177] i40e 0000:3b:00.2 eth4: set new mac address a0:36:9f:13:f4:2c
Feb  9 17:17:42 152224 kernel: [ 1331.906512] 8021q: adding VLAN 0 to HW filter on device eth4
Feb  9 17:17:42 152224 kernel: [ 1331.906896] bond0: Enslaving eth4 as a backup interface with an up link
Feb  9 17:17:42 152224 kernel: [ 1331.907025] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.907202] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.907256] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.907304] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.907352] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.908953] i40e 0000:3b:00.3 eth5: set new mac address a0:36:9f:13:f4:2c
Feb  9 17:17:42 152224 kernel: [ 1331.924151] 8021q: adding VLAN 0 to HW filter on device eth5
Feb  9 17:17:42 152224 kernel: [ 1331.924769] bond0: Enslaving eth5 as a backup interface with an up link
Feb  9 17:17:42 152224 kernel: [ 1331.924961] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.925133] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.925201] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.925297] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1331.925376] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1332.189346] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 17:17:42 152224 kernel: [ 1332.189402] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 17:17:42 152224 kernel: [ 1332.189520] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:42 152224 kernel: [ 1332.193083] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:43 152224 kernel: [ 1332.933484] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.908791] i40e 0000:3b:00.0: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.908884] i40e 0000:3b:00.1: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.908970] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.909256] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.926392] i40e 0000:3b:00.0: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.926486] i40e 0000:3b:00.1: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.926573] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:47 152224 kernel: [ 1336.926824] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:48 152224 kernel: [ 1337.797244] ------------[ cut here ]------------
Feb  9 17:17:48 152224 kernel: [ 1337.797247] NETDEV WATCHDOG: eth4 (i40e): transmit queue 6 timed out
Feb  9 17:17:48 152224 kernel: [ 1337.797267] WARNING: CPU: 17 PID: 0 at /build/linux-4.19-7d6em1/linux-4.19-4.19.232/net/sched/sch_generic.c:466 dev_watchdog+0x20e/0x220
Feb  9 17:17:48 152224 kernel: [ 1337.797267] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink 8021q garp stp mrp llc intel_rapl skx_edac nfit ipmi_ssif libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate i40e(O) ast ttm iTCO_wdt iTCO_vendor_support drm_kms_helper raid0 ixgbe(O) intel_uncore drm intel_rapl_perf pcspkr evdev joydev intel_auxiliary(O) i2c_algo_bit mei_me sg ioatdma lpc_ich pcc_cpufreq mei i2c_i801 mfd_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad button tcp_bbr sch_fq ip_tables x_tables hid_generic usbhid hid ext4 crc16 mbcache jbd2 crc32c_generic fscrypto raid1 md_mod sd_mod crc32c_intel nvme
Feb  9 17:17:48 152224 kernel: [ 1337.797302]  ahci xhci_pci nvme_core libahci xhci_hcd libata usbcore scsi_mod usb_common bonding autofs4
Feb  9 17:17:48 152224 kernel: [ 1337.797309] CPU: 17 PID: 0 Comm: swapper/17 Tainted: G           O      4.19.0-0.bpo.19-amd64 #1 Debian 4.19.232-1~deb9u1
Feb  9 17:17:48 152224 kernel: [ 1337.797309] Hardware name: Supermicro SYS-229U-TN24R4T-2-IN001/X11DPU, BIOS 3.8a 10/28/2022
Feb  9 17:17:48 152224 kernel: [ 1337.797311] RIP: 0010:dev_watchdog+0x20e/0x220
Feb  9 17:17:48 152224 kernel: [ 1337.797313] Code: 00 49 63 4e e0 eb 92 4c 89 e7 c6 05 1d ff ac 00 01 e8 46 a4 fc ff 89 d9 48 89 c2 4c 89 e6 48 c7 c7 70 5e 8d b8 e8 e2 22 a5 ff <0f> 0b eb c0 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
Feb  9 17:17:48 152224 kernel: [ 1337.797314] RSP: 0018:ffff96177f443e88 EFLAGS: 00010286
Feb  9 17:17:48 152224 kernel: [ 1337.797315] RAX: 0000000000000000 RBX: 0000000000000006 RCX: 0000000000000006
Feb  9 17:17:48 152224 kernel: [ 1337.797316] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff96177f45b6b0
Feb  9 17:17:48 152224 kernel: [ 1337.797317] RBP: ffff95f7618f645c R08: 0000000000000048 R09: ffff9617fff5c000
Feb  9 17:17:48 152224 kernel: [ 1337.797318] R10: ffff96177f45f830 R11: 0000000000000000 R12: ffff95f7618f6000
Feb  9 17:17:48 152224 kernel: [ 1337.797318] R13: 0000000000000011 R14: ffff95f7618f6480 R15: 0000000000000040
Feb  9 17:17:48 152224 kernel: [ 1337.797320] FS:  0000000000000000(0000) GS:ffff96177f440000(0000) knlGS:0000000000000000
Feb  9 17:17:48 152224 kernel: [ 1337.797320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb  9 17:17:48 152224 kernel: [ 1337.797321] CR2: 000056374c12d000 CR3: 0000003f8480a003 CR4: 00000000007606e0
Feb  9 17:17:48 152224 kernel: [ 1337.797322] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb  9 17:17:48 152224 kernel: [ 1337.797323] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb  9 17:17:48 152224 kernel: [ 1337.797324] PKRU: 55555554
Feb  9 17:17:48 152224 kernel: [ 1337.797324] Call Trace:
Feb  9 17:17:48 152224 kernel: [ 1337.797326]  <IRQ>
Feb  9 17:17:48 152224 kernel: [ 1337.797329]  ? pfifo_fast_reset+0x100/0x100
Feb  9 17:17:48 152224 kernel: [ 1337.797333]  call_timer_fn+0x2b/0x130
Feb  9 17:17:48 152224 kernel: [ 1337.797335]  run_timer_softirq+0x1cf/0x400
Feb  9 17:17:48 152224 kernel: [ 1337.797337]  ? enqueue_hrtimer+0x3a/0x90
Feb  9 17:17:48 152224 kernel: [ 1337.797340]  __do_softirq+0x10d/0x2c3
Feb  9 17:17:48 152224 kernel: [ 1337.797344]  irq_exit+0xc2/0xd0
Feb  9 17:17:48 152224 kernel: [ 1337.797346]  smp_apic_timer_interrupt+0x74/0x130
Feb  9 17:17:48 152224 kernel: [ 1337.797348]  apic_timer_interrupt+0xf/0x20
Feb  9 17:17:48 152224 kernel: [ 1337.797349]  </IRQ>
Feb  9 17:17:48 152224 kernel: [ 1337.797353] RIP: 0010:cpuidle_enter_state+0xb2/0x330
Feb  9 17:17:48 152224 kernel: [ 1337.797354] Code: f4 40 06 48 e8 6f 09 b0 ff 49 89 c7 0f 1f 44 00 00 31 ff e8 70 14 b0 ff 80 7c 24 17 00 0f 85 1c 02 00 00 fb 66 0f 1f 44 00 00 <4c> 2b 7c 24 08 48 ba cf f7 53 e3 a5 9b c4 20 4c 89 f8 49 c1 ff 3f
Feb  9 17:17:48 152224 kernel: [ 1337.797355] RSP: 0018:ffffb43a402c3e70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Feb  9 17:17:48 152224 kernel: [ 1337.797356] RAX: ffff96177f467140 RBX: ffff96177f46fd28 RCX: 000000000000001f
Feb  9 17:17:48 152224 kernel: [ 1337.797357] RDX: 000001377aed9c20 RSI: 00000000294a5294 RDI: 0000000000000000
Feb  9 17:17:48 152224 kernel: [ 1337.797358] RBP: ffffffffb8ab6f60 R08: 0000000000000002 R09: 0000000000026a00
Feb  9 17:17:48 152224 kernel: [ 1337.797359] R10: 00000000ffffffff R11: 000000000000128b R12: ffff961778712c00
Feb  9 17:17:48 152224 kernel: [ 1337.797359] R13: 0000000000000003 R14: 0000000000000003 R15: 000001377aed9c20
Feb  9 17:17:48 152224 kernel: [ 1337.797363]  do_idle+0x1f6/0x270
Feb  9 17:17:48 152224 kernel: [ 1337.797364]  cpu_startup_entry+0x6f/0x80
Feb  9 17:17:48 152224 kernel: [ 1337.797367]  start_secondary+0x1a4/0x200
Feb  9 17:17:48 152224 kernel: [ 1337.797370]  secondary_startup_64+0xa4/0xb0
Feb  9 17:17:48 152224 kernel: [ 1337.797372] ---[ end trace 00ce5ac2cd805844 ]---
Feb  9 17:17:48 152224 kernel: [ 1337.797378] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 6, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
Feb  9 17:17:48 152224 kernel: [ 1337.797380] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 1, hung_queue 6
Feb  9 17:17:48 152224 kernel: [ 1337.798419] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:48 152224 kernel: [ 1337.798448] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 17:17:48 152224 kernel: [ 1337.849274] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:48 152224 kernel: [ 1337.888193] i40e 0000:3b:00.2: VF BW shares not restored
Feb  9 17:17:48 152224 kernel: [ 1337.897429] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:48 152224 kernel: [ 1337.953273] bond0: link status up again after 100 ms for interface eth4
Feb  9 17:17:48 152224 kernel: [ 1337.953370] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 17:17:48 152224 kernel: [ 1337.953478] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:49 152224 kernel: [ 1338.821243] i40e 0000:3b:00.3 eth5: tx_timeout: VSI_seid: 397, Q 6, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
Feb  9 17:17:49 152224 kernel: [ 1338.821246] i40e 0000:3b:00.3 eth5: tx_timeout recovery level 1, hung_queue 6
Feb  9 17:17:49 152224 kernel: [ 1338.822393] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 17:17:49 152224 kernel: [ 1338.822452] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:49 152224 kernel: [ 1338.893252] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:49 152224 kernel: [ 1338.912724] i40e 0000:3b:00.3: VF BW shares not restored
Feb  9 17:17:49 152224 kernel: [ 1338.914070] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:49 152224 kernel: [ 1338.997257] bond0: link status up again after 100 ms for interface eth5
Feb  9 17:17:49 152224 kernel: [ 1338.997350] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 17:17:49 152224 kernel: [ 1338.997434] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:54 152224 kernel: [ 1343.685239] i40e 0000:3b:00.2 eth4: tx_timeout: VSI_seid: 396, Q 17, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
Feb  9 17:17:54 152224 kernel: [ 1343.685242] i40e 0000:3b:00.2 eth4: tx_timeout recovery level 2, hung_queue 17
Feb  9 17:17:54 152224 kernel: [ 1343.685890] i40e 0000:3b:00.3: VSI seid 397 Tx ring 0 disable timeout
Feb  9 17:17:54 152224 kernel: [ 1343.685893] i40e 0000:3b:00.2: VSI seid 396 Tx ring 0 disable timeout
Feb  9 17:17:54 152224 kernel: [ 1343.689253] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.689261] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.697233] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.697236] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.705231] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.705234] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.713237] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.713240] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.721231] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.721233] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.729233] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.729236] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.737232] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.737234] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.745242] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.745244] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.750833] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:54 152224 kernel: [ 1343.753228] bond0: link status down for interface eth4, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.753229] bond0: link status down for interface eth5, disabling it in 200 ms
Feb  9 17:17:54 152224 kernel: [ 1343.797440] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:54 152224 kernel: [ 1343.965385] bond0: link status definitely down for interface eth4, disabling it
Feb  9 17:17:54 152224 kernel: [ 1343.965444] bond0: link status definitely down for interface eth5, disabling it
Feb  9 17:17:54 152224 kernel: [ 1343.965543] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:54 152224 kernel: [ 1343.965749] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:57 152224 kernel: [ 1346.878618] i40e 0000:3b:00.1: VF BW shares not restored
Feb  9 17:17:57 152224 kernel: [ 1346.882935] i40e 0000:3b:00.0: VF BW shares not restored
Feb  9 17:17:57 152224 kernel: [ 1346.915138] i40e 0000:3b:00.3: VF BW shares not restored
Feb  9 17:17:57 152224 kernel: [ 1346.926951] i40e 0000:3b:00.2: VF BW shares not restored
Feb  9 17:17:57 152224 kernel: [ 1346.927023] i40e 0000:3b:00.3: PTP external clock not supported.
Feb  9 17:17:57 152224 kernel: [ 1346.931179] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:57 152224 kernel: [ 1346.993231] bond0: link status up for interface eth4, enabling it in 200 ms
Feb  9 17:17:57 152224 kernel: [ 1346.993234] bond0: link status up for interface eth5, enabling it in 200 ms
Feb  9 17:17:57 152224 kernel: [ 1347.201295] bond0: link status definitely up for interface eth4, 10000 Mbps full duplex
Feb  9 17:17:57 152224 kernel: [ 1347.201342] bond0: link status definitely up for interface eth5, 10000 Mbps full duplex
Feb  9 17:17:57 152224 kernel: [ 1347.201440] i40e 0000:3b:00.2: PTP external clock not supported.
Feb  9 17:17:57 152224 kernel: [ 1347.201909] i40e 0000:3b:00.3: PTP external clock not supported.

IntelSupport · ‎02-19-2023

Hi AndriiV,

We are sorry for the late response. Can share with us what kernel version that you are currently use?

You also can refer to below link which have similar case of bonding using Debian.

https://community.intel.com/t5/Ethernet-Products/Intel-10G-NIC-no-longer-works-in-Debian-11-after-Firmware-update/m-p/1411805

Best regards,

Fariz_Intel

AndriiV · ‎02-20-2023

@IntelSupport wrote:

Can share with us what kernel version that you are currently use?

# uname -r
4.19.0-0.bpo.19-amd64

# cat /etc/debian_version
10.13

X710-4 NIC changed by XL710-Q1 NIC

# lspci | grep Eth
3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
d8:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
d8:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

IntelSupport · ‎02-20-2023

Hi AndriiV,

Seem like you have replace X710-4 NIC with XL710-Q1 NIC. Have you try bonding using XL710-Q1 NIC?

May I know what network switch model that you connect with? Is it possible for you to test with different model of network switch?

Since the driver for Debian 10 and lower is mostly not fully tested or compatible, we might need to do some further testing only if available.

Best regards,

Fariz_Intel

AndriiV · ‎02-23-2023

@IntelSupport wrote:

May I know what network switch model that you connect with? Is it possible for you to test with different model of network switch?

QFX5200-32C No, it`s colocation.

Seem like you have replace X710-4 NIC with XL710-Q1 NIC. Have you try bonding using XL710-Q1 NIC?

No, we need only 40Gbps link

IntelSupport · ‎02-20-2023

Hi AndriiV,

You may also want to check on below link and see if it can further help with your issue.

Intel® Ethernet Adapter Complete Driver Pack (large file)

https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html?cache=1663009402

Intel® System Support Utility for the Linux* Operating System

https://www.intel.com/content/www/us/en/download/18895/intel-system-support-utility-for-the-linux-operating-system.html

Best regards,

Fariz_Intel

AndriiV · ‎02-24-2023

./ssu.sh log in attachment

RalphC · ‎03-03-2023

Hello AndriiV,

This Intel document recommends disabling LLDP when using LACP.

https://www.intel.com/content/www/us/en/content-details/337361/lacp-configuration-guide-using-intel-ethernet-500-and-700-series-network-adapters-and-various-server-operating-systems-technical-brief.html?wapkw=lacp%20guide

Have you tried disabling the Intel LLDP firmware agent? You should be able to find this setting in the X710 adapter settings via System UEFI.

Fariz_Intel · ‎07-13-2023

Hi AndriiV

Through the further checking recently we have found that Debian 11 is only supported for the ice driver for E800 series. Unfortunately at the moment there is no current support for i40e or ixgbe drivers.

If you need any additional information, please submit a new question as this thread will no longer be monitored.

Best regards,

Fariz_Intel