XL710 too many irq

[AndriiV]

Hello!

I have a problem with Intel XL710,  after 22 Gbps of traffic some cores of processor loads 100% IRQ and traffic goes down.

There are latast firmware and driver:

# ethtool -i ens2
driver: i40e
version: 2.24.6
firmware-version: 9.40 0x8000ecc0 1.3429.0
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I read this guide

And made all possible configurations.  But nothing changes

382  set_irq_affinity local ens2
  384  set_irq_affinity all ens2
  387  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
  389  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 5000 tx-usecs 20000
  391  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 10000 tx-usecs 20000
  393  ethtool -g ens2
  394  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 84 tx-usecs 84
  396  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 62 tx-usecs 62
  398   ethtool -S ens2 | grep drop
  399   ethtool -S ens2 | grep drop
  400   ethtool -S ens2 | grep drop
  401   ethtool -S ens2 | grep drop
  402  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 336 tx-usecs 84
  403   ethtool -S ens2 | grep drop
  404   ethtool -S ens2 | grep drop
  406   ethtool -S ens2 | grep drop
  407  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 672 tx-usecs 84
  408   ethtool -S ens2 | grep drop
  409   ethtool -S ens2 | grep drop
  411   ethtool -S ens2 | grep drop
  412   ethtool -S ens2 | grep drop
  425   ethtool -S ens2 | grep drop
  426   ethtool -S ens2 | grep drop
  427  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 8400 tx-usecs 840
  428  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 840
  430   ethtool -S ens2 | grep drop
  431   ethtool -S ens2 | grep drop
  432   ethtool -S ens2 | grep drop
  433   ethtool -S ens2 | grep drop
  434  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1680
  435   ethtool -S ens2 | grep drop
  436   ethtool -S ens2 | grep drop
  439  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 3200 tx-usecs 3200
  469  ethtool -a ens2
  472  ethtool ens2
  473  ethtool -i ens2
  475  ethtool -i ens2
  476  ethtool ens2
  482  ethtool -C ens2 adaptive-rx on
  484  ethtool -c ens2
  486  ethtool -C ens2 adaptive-tx on
  487  ethtool -c ens2
  492  history | grep ens2
  494  ethtool -m ens2
  499  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1600
  501  history | grep ens2

Server configuration:

80 cores  Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz

RAM 960 GB

8 x SAMSUNG MZQLB7T6HMLA-000AZ NVME disks

Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

What can be done with the settings of this network card to solve the problem?

On another server with a similar configuration, but a different network cards, everything is fine

Server configuration:

88 cores Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

RAM 512 GB

8 x SAMSUNG MZQLB7T6HMLA-00007 NVME disks

4 x 82599ES 10-Gigabit SFI/SFP+ Network Connection

[Simon-Intel]

Hi AndriiV,

Thank you for reaching out to the Intel community. We sincerely apologize for the delay in our response.

We acknowledge receipt of your concern and want to assure you that assisting you is our top priority. To effectively troubleshoot the issue you're facing, we require some additional information from your end.

Could you kindly provide the following details?

1. System Model:
2. Operating System:
3. Is the Ethernet card embedded on the board?
4. Was the Ethernet card working before? Have any changes been made recently?

Your prompt response with this information will greatly assist us in diagnosing and resolving the issue as quickly as possible.

We look forward to hearing from you soon.

Best Regards,

Simon

[AndriiV]

Hello Simon!

1.  Supermicro SuperServer 1029U-TRTP2

2.  Debian 10.13

# uname -a
Linux 15224 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux

3. No

4. No,  I am using XL710 card for a first time

[Simon-Intel]

Hi AndriiV,

Thank you for sharing the requested details.

As you mentioned, there is another server with a similar configuration, but with different network cards. Could you please share the server motherboard and the network card details?

We look forward to hearing from you soon.

Best Regards,

Simon

[AndriiV]

Hi Simon!

Supermicro SuperServer 1028U-TR4+

Motherboard: X10DRU-i+

Debian 10.13

# uname -a
Linux 157540 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08) x86_64 GNU/Linux

 4x10Gbps network cards:

87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535

[Simon-Intel]

Hi Andrii,

Good day!

Thank you for providing the requested details.

We are currently investigating this inquiry with our resources, and we will provide you with an update at the earliest opportunity.

Thank you for your patience and understanding.

Best regards,
Simon

[Simon-Intel]

Hi Andrii,

Good Day!

Thank you for your patience as we work to resolve the issue you've encountered. We at Intel are fully committed to ensuring your satisfaction.

To address your concern, please refer to the following link for detailed instructions: Intel Ethernet 700 Series Linux Performance Tuning Guide - Section 4.1.

Your understanding and cooperation are greatly appreciated.

Best regards,

Simon

[AndriiV]

Hi Simon!

As you can see my first message in this topic. I already read this guide and made all tunnings. I tried

382  set_irq_affinity local ens2
  384  set_irq_affinity all ens2

 With "local" configuration 100% IRQ on some cores at 19Gbps yet. You can see atop screen after 

set_irq_affinity local ens2

[AndriiV]

I think the problem is that the traffic is unevenly distributed across the queues

[Simon-Intel]

Hi Andrii,

Good Day!

Firstly, I'd like to extend our apologies for any inconvenience caused by our previous communication. Your detailed explanation of the issue and server configuration was immensely helpful, and we appreciate your patience and cooperation.

I'm writing to inform you that we've escalated the matter to our next level team for further investigation and resolution. Rest assured, we're prioritizing this issue to ensure a swift resolution.

We will keep you updated on the progress and provide any necessary information as soon as it becomes available.

Once again, thank you for your understanding and cooperation throughout this process.

Best regards,

Simon