XL710 too many irq

[AndriiV]

Hello!

I have a problem with Intel XL710,  after 22 Gbps of traffic some cores of processor loads 100% IRQ and traffic goes down.

There are latast firmware and driver:

# ethtool -i ens2
driver: i40e
version: 2.24.6
firmware-version: 9.40 0x8000ecc0 1.3429.0
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I read this guide

And made all possible configurations.  But nothing changes

382  set_irq_affinity local ens2
  384  set_irq_affinity all ens2
  387  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
  389  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 5000 tx-usecs 20000
  391  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 10000 tx-usecs 20000
  393  ethtool -g ens2
  394  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 84 tx-usecs 84
  396  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 62 tx-usecs 62
  398   ethtool -S ens2 | grep drop
  399   ethtool -S ens2 | grep drop
  400   ethtool -S ens2 | grep drop
  401   ethtool -S ens2 | grep drop
  402  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 336 tx-usecs 84
  403   ethtool -S ens2 | grep drop
  404   ethtool -S ens2 | grep drop
  406   ethtool -S ens2 | grep drop
  407  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 672 tx-usecs 84
  408   ethtool -S ens2 | grep drop
  409   ethtool -S ens2 | grep drop
  411   ethtool -S ens2 | grep drop
  412   ethtool -S ens2 | grep drop
  425   ethtool -S ens2 | grep drop
  426   ethtool -S ens2 | grep drop
  427  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 8400 tx-usecs 840
  428  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 840
  430   ethtool -S ens2 | grep drop
  431   ethtool -S ens2 | grep drop
  432   ethtool -S ens2 | grep drop
  433   ethtool -S ens2 | grep drop
  434  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1680
  435   ethtool -S ens2 | grep drop
  436   ethtool -S ens2 | grep drop
  439  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 3200 tx-usecs 3200
  469  ethtool -a ens2
  472  ethtool ens2
  473  ethtool -i ens2
  475  ethtool -i ens2
  476  ethtool ens2
  482  ethtool -C ens2 adaptive-rx on
  484  ethtool -c ens2
  486  ethtool -C ens2 adaptive-tx on
  487  ethtool -c ens2
  492  history | grep ens2
  494  ethtool -m ens2
  499  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1600
  501  history | grep ens2

Server configuration:

80 cores  Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz

RAM 960 GB

8 x SAMSUNG MZQLB7T6HMLA-000AZ NVME disks

Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

What can be done with the settings of this network card to solve the problem?

On another server with a similar configuration, but a different network cards, everything is fine

Server configuration:

88 cores Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

RAM 512 GB

8 x SAMSUNG MZQLB7T6HMLA-00007 NVME disks

4 x 82599ES 10-Gigabit SFI/SFP+ Network Connection

[AndriiV]

Hi Simon!

Sorry for my late answer.

We use a lot of different Intel network cards from 1000Mbit/s on the e1000e and igb drivers to 100Gbit/s on the ice driver.

Nowhere have there been problems with such large IRQ except this server with XL710

For example, a server graph with E810-C:

With 84 Gbps traffic, only 1.16k IRQ.

Immediately with a traffic of 20 Gbit/s there are more than 2k IRQ. Why?

[IntelSupport]

Hello AndriiV,

Greetings!

Thank you for writing to us. Thank you for sharing further information. We are actively working on this case with our level 2 team. We will get back to you again with an update at earliest.

Thank you for your patience.

Regards,

Subhashish.

[Hayat]

Hi AndriiV, 

Thank you for your patience.

Please let us know if you have try the following tunings that we hadrecommended? If so, have you noticed any differences in the results?

With regards to high IRQ load, Higher-end NICs typically employ more sophisticated interrupt coalescing mechanisms to aggregate multiple network events into a single interrupt. Thereby reducing the frequency of interrupts and mitigating the IRQ load. 

Interrupt affinity, interrupt moderation parameters, and CPU scaling governors, can affect how interrupts are handled and distributed across CPU cores.

Along with the tunings we had suggested in the previous email, to please try the steps below to configure CPU governance.

To view the current CPU scaling governor -> #cpupower frequency-info

To change the CPU scaling governor to a specific mode -> #cpupower frequency-set -g performance

Kindly clarify few questions as below:

+ In the link , you had shared the graph for bonding results too. Have  

 you encountered the same issue with bonding?

+ Does the issue occur across all platforms? Please try the following  

 scenarios below:

  - Test with different XL710 NICs to determine if the issue is with   

   the card or something else.

  - Try inserting the same NIC in another platform to determine if the 

   issue is platform-specific or not.

Regards,

Hayat

Intel Customer Support

[AndriiV]

Hi Hayat!

To view the current CPU scaling governor -> #cpupower frequency-info

To change the CPU scaling governor to a specific mode -> #cpupower frequency-set -g performance

# cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 800 MHz - 3.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 800 MHz and 3.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 2.80 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
# cpupower frequency-set -g performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
Setting cpu: 8
Setting cpu: 9
Setting cpu: 10
Setting cpu: 11
Setting cpu: 12
Setting cpu: 13
Setting cpu: 14
Setting cpu: 15
Setting cpu: 16
Setting cpu: 17
Setting cpu: 18
Setting cpu: 19
Setting cpu: 20
Setting cpu: 21
Setting cpu: 22
Setting cpu: 23
Setting cpu: 24
Setting cpu: 25
Setting cpu: 26
Setting cpu: 27
Setting cpu: 28
Setting cpu: 29
Setting cpu: 30
Setting cpu: 31
Setting cpu: 32
Setting cpu: 33
Setting cpu: 34
Setting cpu: 35
Setting cpu: 36
Setting cpu: 37
Setting cpu: 38
Setting cpu: 39
Setting cpu: 40
Setting cpu: 41
Setting cpu: 42
Setting cpu: 43
Setting cpu: 44
Setting cpu: 45
Setting cpu: 46
Setting cpu: 47
Setting cpu: 48
Setting cpu: 49
Setting cpu: 50
Setting cpu: 51
Setting cpu: 52
Setting cpu: 53
Setting cpu: 54
Setting cpu: 55
Setting cpu: 56
Setting cpu: 57
Setting cpu: 58
Setting cpu: 59
Setting cpu: 60
Setting cpu: 61
Setting cpu: 62
Setting cpu: 63
Setting cpu: 64
Setting cpu: 65
Setting cpu: 66
Setting cpu: 67
Setting cpu: 68
Setting cpu: 69
Setting cpu: 70
Setting cpu: 71
Setting cpu: 72
Setting cpu: 73
Setting cpu: 74
Setting cpu: 75
Setting cpu: 76
Setting cpu: 77
Setting cpu: 78
Setting cpu: 79

I already made it by startup-script:

cpucores=`cat /proc/cpuinfo | awk '/^processor/{print $3}' | tail -1`

for i in `seq 0 $cpucores`;
do
echo performance > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor
done;

+ In the link , you had shared the graph for bonding results too. Have you encountered the same issue with bonding?

No, as for me, the 82599 cards are the most problem-free.

+ Does the issue occur across all platforms? Please try the following scenarios below:

- Test with different XL710 NICs to determine if the issue is with the card or something else.

- Try inserting the same NIC in another platform to determine if the issue is platform-specific or not.

Unfortunately, we currently do not have a second server with an XL710 card or another such card in this data center.  We want to replace it with 4x82599.

[Simon-Intel]

Hi AndriiV,

Thank you for your response. Please allow us some time to check this internally. We will get back to you as soon as we have an update.

Regards,

Simon

[MACM]

Hi AndriiV,

Greetings from Intel.

Hope you are doing great.

We noticed one small mistake in the startup-script which is listed in below email , we hope the highlighted is typo in the email, not in the script.

 echo performance ] /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor 

Correct syntax : echo performance > sudo tee /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor

Regards,

Ali

[AndriiV]

Hello, MACM!

I see my message like this:

[Simon-Intel]

Hello AndriiV,

Good Day!

Please allow us sometime while we check this internally with our team and will get back to you as soon as we have an update.

Best regards,

Simon

[Irwan_Intel]

Hello AndriiV,

We have emailed you to request remote access. Please share if there's any way for us to access the system.

Regards,

Irwan_Intel

[AndriiV]

Hello Irwan!

I can grant root access via SSH to this server. How can I send you credentials?

[Fikri_Intel]

Hi AndriiV,

We have emailed you separately for you to share the credential access.

Regards,

Fikri

[AndriiV]

Hello all!

We decided to try another network card on this server.

NIC: Intel XXV710-DA2 2x25GbE

Cable: Intel XXV4DACBL2M passive 100GBASE-CR4 QSFP28
to 4x25GBASE-CR SFP28

First of all.  Advertised auto-negotiation only 10Gbps on each port.

After 6Mbps of trafic everything hung out.

After

ethtool --set-priv-flags ens1f0 disable-fw-lldp on
ethtool --set-priv-flags ens1f1 disable-fw-lldp on

almost all was fine but traffic did not rise above 9 Gbit/s on each port. Also was disabled DCB on switch.

As it turned out, this cable is not in the compatibility list of Juniper QFX5200-32C 

It was replaced by  H3C QSFP28-4SFP28-CU-3M Compatible 100G QSFP28 to 4 x 25G SFP28 Passive Direct Attach Copper Breakout Cable.

Advertised auto-negotiation 25Gbps on each port.

But after 20 Gbps on bonding interface (of this two ports) exactly the same problems as with the XL710 were observed.

Judging by the graphs, we rest on some ceiling of 20 Gbps.

We have almost the same server with Intel XXV710-DA2 2x25Gb NIC

Server configuration:

72 cores  Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz

RAM 768 GB

8 x SAMSUNG MZQLB7T6HMLA-00007 NVME disks

Ethernet Controller Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

It conected to Arista DCS-7060CX-32S switch by Intel® Ethernet QSFP28 to SFP28 Twinaxial Breakout XXV4DACBL2M 952303 NDAQGF-I202 100GbE/4x25GbE/4x10GbE Direct Attach

There no such problem. And there is LLDP enabled (by default):

# ethtool --show-priv-flags ens2f0
Private flags for ens2f0:
MFP                     : off
total-port-shutdown     : off
LinkPolling             : off
flow-director-atr       : on
veb-stats               : off
hw-atr-eviction         : off
link-down-on-close      : off
legacy-rx               : off
disable-source-pruning  : off
mac-source-pruning      : on
disable-fw-lldp         : off
rs-fec                  : on
base-r-fec              : on
multiple-traffic-classes: off
vf-vlan-pruning         : off
vf-source-pruning       : on
mdd-auto-reset-vf       : off
vf-true-promisc-support : off

After all these tests I am increasingly inclined to the conclusion that the problem lies in the incompatibility of Intel and Juniper and there are clearly some problems with LDDP/DCB.

[AndriiV]

Hello all! A few days ago we connected this server using a new scheme.

NIC: 2 ports X710 

         2 ports XXV710

Cable: Intel XXV4DACBL2M passive 100GBASE-CR4 QSFP28

Switch: Juniper QFX5200-32C 

I used Linux software bonding to set up a 4x 10Gbps connection:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 3c:fd:fe:a8:62:04
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 4
        Actor Key: 15
        Partner Key: 5
        Partner Mac Address: 88:e6:4b:6d:d5:24

Slave Interface: ens1f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:fd:fe:a8:62:04
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 3c:fd:fe:a8:62:04
    port key: 15
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: 88:e6:4b:6d:d5:24
    oper key: 5
    port priority: 127
    port number: 41
    port state: 63

Slave Interface: ens1f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:fd:fe:a8:62:05
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 3c:fd:fe:a8:62:04
    port key: 15
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: 88:e6:4b:6d:d5:24
    oper key: 5
    port priority: 127
    port number: 40
    port state: 63

Slave Interface: eno3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:1f:6b:59:c9:7e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 3c:fd:fe:a8:62:04
    port key: 15
    port priority: 255
    port number: 3
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: 88:e6:4b:6d:d5:24
    oper key: 5
    port priority: 127
    port number: 38
    port state: 63

Slave Interface: eno4
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:1f:6b:59:c9:7f
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: 3c:fd:fe:a8:62:04
    port key: 15
    port priority: 255
    port number: 4
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: 88:e6:4b:6d:d5:24
    oper key: 5
    port priority: 127
    port number: 39
    port state: 63

Also LLDP was disabled:

# ethtool --set-priv-flags ens1f1 disable-fw-lldp on
# ethtool --set-priv-flags ens1f0 disable-fw-lldp on
# ethtool --set-priv-flags eno3 disable-fw-lldp on
# ethtool --set-priv-flags eno4 disable-fw-lldp on

Also set irq affinity:

# set_irq_affinity -x local ens1f0 ens1f1 eno3 eno4

Results:

We've hit a ceiling of 23Gbps.
Interrupts are still eating up the entire CPU

When I tried to enable LLDP on two cards, traffic stopped flowing through them and errors started pouring in:

Aug  9 11:02:49 15224 kernel: [245313.055731] i40e 0000:19:00.1: VSI seid 390 Tx ring 0 disable timeout
Aug  9 11:02:49 15224 kernel: [245313.062170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.070168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.078169] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.086168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.094173] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.102171] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.114168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.122168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.134170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.146189] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.154167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.162170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.174168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.182167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.194183] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.202168] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.210167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.221934] i40e 0000:19:00.1: VSI seid 0 Tx ring 767 disable timeout
Aug  9 11:02:49 15224 kernel: [245313.222170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.230167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.238165] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.246166] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.258167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.270170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.278189] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.284384] i40e 0000:19:00.1: FW LLDP is disabled, attempting SW DCB
Aug  9 11:02:49 15224 kernel: [245313.286167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.294167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.295276] i40e 0000:19:00.1: SW DCB initialization succeeded.
Aug  9 11:02:49 15224 kernel: [245313.302170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.303844] i40e 0000:19:00.1: MAC source pruning enabled on all VFs
Aug  9 11:02:49 15224 kernel: [245313.310167] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.318240] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.330170] bond0: link status down for interface eno4, disabling it in 200 ms
Aug  9 11:02:49 15224 kernel: [245313.333712] i40e 0000:19:00.1: Set default VSI failed, err I40E_ERR_ADMIN_QUEUE_ERROR, aq_err I40E_AQ_RC_EINVAL
Aug  9 11:02:49 15224 kernel: [245313.333715] i40e 0000:19:00.1: Failed to restore promiscuous setting: off, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
Aug  9 11:02:49 15224 kernel: [245313.333717] i40e 0000:19:00.1: VF BW shares not restored
Aug  9 11:02:49 15224 kernel: [245313.333772] i40e 0000:19:00.1: FW LLDP is disabled
Aug  9 11:02:54 15224 kernel: [245318.023464] i40e 0000:19:00.0: VSI seid 391 Tx ring 0 disable timeout
Aug  9 11:02:54 15224 kernel: [245318.078151] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.086148] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.089907] i40e 0000:19:00.0: VSI seid 0 Tx ring 767 disable timeout
Aug  9 11:02:54 15224 kernel: [245318.098147] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.106153] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.114146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.122145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.130146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.138144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.146148] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.154154] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.162156] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.174161] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.182145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.190154] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.198146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.206152] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.218154] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.226148] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.234235] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.242286] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.250159] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.262150] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.274212] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.386149] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.394147] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.402150] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.410151] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.418153] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.426144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.434148] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.442160] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.454147] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.462146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.463467] i40e 0000:19:00.0: VSI seid 391 Tx ring 0 disable timeout
Aug  9 11:02:54 15224 kernel: [245318.470149] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.478153] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.486144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.494144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.502143] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.510143] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.522155] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.534344] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.546147] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.554145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.562145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.570154] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.582151] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.590144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.598152] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.606144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.614149] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.619470] i40e 0000:19:00.0: VSI seid 0 Tx ring 767 disable timeout
Aug  9 11:02:54 15224 kernel: [245318.622143] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.630145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.638146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.646144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.654145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.662142] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.670146] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.678147] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.682065] i40e 0000:19:00.0: FW LLDP is disabled, attempting SW DCB
Aug  9 11:02:54 15224 kernel: [245318.686145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.692934] i40e 0000:19:00.0: SW DCB initialization succeeded.
Aug  9 11:02:54 15224 kernel: [245318.698145] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:54 15224 kernel: [245318.701717] i40e 0000:19:00.0: MAC source pruning enabled on all VFs
Aug  9 11:02:55 15224 kernel: [245318.710161] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:55 15224 kernel: [245318.718144] bond0: link status down for interface eno3, disabling it in 200 ms
Aug  9 11:02:55 15224 kernel: [245318.724405] i40e 0000:19:00.0: Set default VSI failed, err I40E_ERR_ADMIN_QUEUE_ERROR, aq_err I40E_AQ_RC_EINVAL
Aug  9 11:02:55 15224 kernel: [245318.724408] i40e 0000:19:00.0: Failed to restore promiscuous setting: off, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
Aug  9 11:02:55 15224 kernel: [245318.724410] i40e 0000:19:00.0: VF BW shares not restored
Aug  9 11:02:55 15224 kernel: [245318.724466] i40e 0000:19:00.0: FW LLDP is disabled