Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

i40e + almalinux 8.6 performance issue

NicolasB827
Beginner
869 Views

Hello,

I have a Ethernet Controller X710 for 10GbE backplane on a FC640 Dell server.

I have installed AlmaLinux 8.6.

The network is running, but I have huge performance issues with it. I tried with a Centos 7.9 earlier with the same issues. I compared with other servers with (almost) the same settings, and the only think which is not the same is the network card, and the blade chassis.

I activated SR-IOV settings in bios, but no VF. The OS was installed with nothing more installed on it.

Using the stock driver, I have some issues.

Now, I have a more recent driver (not the latest), with the same issues.

Can you help me resolving my problem ?

Tests

Using iperf on this host to another server:

 

[root@host i40e-2.17.15]# iperf3 -c 172.16.12.51 -i 1s -b 0 -N
Connecting to host 172.16.12.51, port 5201
[  5] local 172.16.12.37 port 39790 connected to 172.16.12.51 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   641 KBytes  5.25 Mbits/sec    1   29.7 KBytes
[  5]   1.00-2.00   sec   764 KBytes  6.26 Mbits/sec    0   32.5 KBytes
[  5]   2.00-3.00   sec   764 KBytes  6.26 Mbits/sec    0   33.9 KBytes
[  5]   3.00-4.00   sec   764 KBytes  6.26 Mbits/sec    0   33.9 KBytes
[  5]   4.00-5.00   sec   891 KBytes  7.30 Mbits/sec    0   49.5 KBytes
[  5]   5.00-6.00   sec   764 KBytes  6.26 Mbits/sec    0   49.5 KBytes
[  5]   6.00-7.00   sec   636 KBytes  5.21 Mbits/sec    0   49.5 KBytes
[  5]   7.00-8.00   sec   764 KBytes  6.26 Mbits/sec    0   49.5 KBytes
[  5]   8.00-9.00   sec   764 KBytes  6.26 Mbits/sec    0   49.5 KBytes
[  5]   9.00-10.00  sec   764 KBytes  6.26 Mbits/sec    0   49.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.34 MBytes  6.15 Mbits/sec    1             sender
[  5]   0.00-10.00  sec  7.13 MBytes  5.98 Mbits/sec                  receiver

iperf Done.

 

I tried also without bonding Interface, using eno3:

 

[root@gimli07 i40e-2.17.15]# iperf3 -c 172.16.2.51 -i 1s -b 1G -N -u
Connecting to host 172.16.2.51, port 5201
[  5] local 172.16.2.37 port 60987 connected to 172.16.2.51 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.06 MBytes  8.88 Mbits/sec  124
[  5]   1.00-2.00   sec  1.41 MBytes  11.8 Mbits/sec  165
[  5]   2.00-3.00   sec  1.70 MBytes  14.2 Mbits/sec  199
[  5]   3.00-4.00   sec  1.43 MBytes  12.0 Mbits/sec  168
[  5]   4.00-5.00   sec  1.79 MBytes  15.0 Mbits/sec  210
[  5]   5.00-6.00   sec  1.27 MBytes  10.7 Mbits/sec  149
[  5]   6.00-7.00   sec  1.45 MBytes  12.2 Mbits/sec  170
[  5]   7.00-8.00   sec  1.78 MBytes  15.0 Mbits/sec  209
[  5]   8.00-9.00   sec  1.48 MBytes  12.5 Mbits/sec  174
[  5]   9.00-10.00  sec  1.75 MBytes  14.7 Mbits/sec  205
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  15.1 MBytes  12.7 Mbits/sec  0.000 ms  0/1773 (0%)  sender
[  5]   0.00-10.04  sec  7.62 MBytes  6.36 Mbits/sec  4.907 ms  868/1761 (49%)  receiver

iperf Done.

 

The same command, using a F430 with another NIC (not a wonderfull BW, but quite different )

 

[root@merry04 ~]# iperf3 -c 172.16.12.51 -i 1s -b 0 -N
Connecting to host 172.16.12.51, port 5201
[  5] local 172.16.12.63 port 53912 connected to 172.16.12.51 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  18.3 MBytes   153 Mbits/sec  336   1.41 KBytes
[  5]   1.00-2.00   sec  36.6 MBytes   307 Mbits/sec  556   25.5 KBytes
[  5]   2.00-3.00   sec  62.6 MBytes   525 Mbits/sec  1336   1.41 KBytes
[  5]   3.00-4.00   sec  39.3 MBytes   330 Mbits/sec  569    141 KBytes
[  5]   4.00-5.00   sec  64.8 MBytes   544 Mbits/sec  1017   84.8 KBytes
[  5]   5.00-6.00   sec  28.8 MBytes   241 Mbits/sec  796   96.2 KBytes
[  5]   6.00-7.00   sec  24.4 MBytes   204 Mbits/sec  657   28.3 KBytes
[  5]   7.00-8.00   sec  34.2 MBytes   287 Mbits/sec  373   22.6 KBytes
[  5]   8.00-9.00   sec  47.5 MBytes   398 Mbits/sec  1014   77.8 KBytes
[  5]   9.00-10.00  sec  65.5 MBytes   549 Mbits/sec  638   42.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   422 MBytes   354 Mbits/sec  7292             sender
[  5]   0.00-10.00  sec   420 MBytes   352 Mbits/sec                  receiver

iperf Done.

 

 

Please find some technical informations:

  • uname -r
[root@host i40e-2.17.15]# uname -r
4.18.0-372.16.1.el8_6.x86_64
  • modinfo i40e
filename:       /lib/modules/4.18.0-372.16.1.el8_6.x86_64/updates/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.17.15
license:        GPL
description:    Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
rhelversion:    8.6
srcversion:     9B949E0F5207219EC83A942
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d0000101Fsv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:
name:           i40e
vermagic:       4.18.0-372.16.1.el8_6.x86_64 SMP mod_unload modversions
parm:           debug:Debug level (0=none,...,16=all) (int)
parm:           l4mode:L4 cloud filter mode: 0=UDP,1=TCP,2=Both,-1=Disabled(default) (int)
  • modinfo iavf
filename:       /lib/modules/4.18.0-372.16.1.el8_6.x86_64/updates/drivers/net/ethernet/intel/iavf/iavf.ko
version:        4.5.3
license:        GPL
description:    Intel(R) Ethernet Adaptive Virtual Function Driver
author:         Intel Corporation, <linux.nics@intel.com>
alias:          i40evf
rhelversion:    8.6
srcversion:     099202B0B2ABC75FD9DC570
alias:          pci:v00008086d00000DD5sv*sd*bc*sc*i*
alias:          pci:v00008086d00001889sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CDsv*sd*bc*sc*i*
alias:          pci:v00008086d00001571sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Csv*sd*bc*sc*i*
depends:
name:           iavf
vermagic:       4.18.0-372.16.1.el8_6.x86_64 SMP mod_unload modversions
  • SR-IOV numvfs
[root@host i40e-2.17.15]# for i in 1 2 3 4; do echo "=== eno$i ===" ; cat /sys/class/net/eno$i/device/sriov_numvfs ; done
=== eno1 ===
0
=== eno2 ===
0
=== eno3 ===
0
=== eno4 ===
0
  • lspci
[root@host i40e-2.17.15]# lspci | grep Ether
18:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE backplane (rev 01)
18:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE backplane (rev 01)
18:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE backplane (rev 01)
18:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE backplane (rev 01)
  • lshw (2 of 4 cards, all the same)
*-network:0
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE backplane
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:18:00.0
logical name: eno1
version: 01
serial: e4:43:4b:d6:88:a0
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical autonegotiation
configuration: autonegotiation=on broadcast=yes driver=i40e driverversion=2.17.15 duplex=full firmware=8.40 0x8000af8b 20.5.13 latency=0 link=yes multicast=yes slave=yes speed=10Gbit/s
resources: iomemory:38400-383ff iomemory:38400-383ff irq:37 memory:384000000000-384000ffffff memory:384004800000-384004807fff memory:9d800000-9d87ffff memory:384004000000-3840041fffff memory:384004820000-38400489ffff
*-network:1
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE backplane
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:18:00.1
logical name: eno2
version: 01
serial: e4:43:4b:d6:88:a0
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical autonegotiation
configuration: autonegotiation=on broadcast=yes driver=i40e driverversion=2.17.15 duplex=full firmware=8.40 0x8000af8b 20.5.13 latency=0 link=yes multicast=yes slave=yes speed=10Gbit/s
resources: iomemory:38400-383ff iomemory:38400-383ff irq:37 memory:384001000000-384001ffffff memory:384004808000-38400480ffff memory:9d880000-9d8fffff memory:384004200000-3840043fffff memory:3840048a0000-38400491ffff
  • nmcli
vlan112: connected to vlan112
        "vlan112"
        vlan, E4:43:4B:D6:88:A0, sw, mtu 9000
        ip4 default, ip6 default
        inet4 172.16.12.37/24
        route4 default via 172.16.12.254 metric 400
        route4 172.16.12.0/24 metric 400
        inet6 fc00:16:93:12:457:21a8:f7ec:4355/64
        inet6 fe80::f452:5d16:2e1e:9642/64
        route6 fc00:16:93:12::/64 metric 400
        route6 fe80::/64 metric 1024
        route6 default via fe80::200:5eff:fe00:270 metric 400

eno3: connected to eno3
        "Intel X710"
        ethernet (i40e), E4:43:4B:D6:88:A2, hw, port e4434bd688a2, mtu 9000
        inet4 172.16.3.37/24
        inet4 172.16.2.37/24
        route4 172.16.2.0/24 metric 102
        route4 172.16.3.0/24 metric 102

eno4: connected to eno4
        "Intel X710"
        ethernet (i40e), E4:43:4B:D6:88:A3, hw, port e4434bd688a3, mtu 9000
        inet4 172.16.4.137/24
        inet4 172.16.2.137/24
        route4 172.16.2.0/24 metric 103
        route4 172.16.4.0/24 metric 103

bond0: connected to bond0
        "bond0"
        bond, E4:43:4B:D6:88:A0, sw, mtu 9000

eno1: connected to eth0
        "Intel X710"
        ethernet (i40e), E4:43:4B:D6:88:A0, hw, port e4434bd688a0, mtu 9000
        master bond0

eno2: connected to eth1
        "Intel X710"
        ethernet (i40e), E4:43:4B:D6:88:A0, hw, port e4434bd688a1, mtu 9000
        master bond0

idrac: disconnected
        "Dell iDRAC Virtual"
        1 connection available
        ethernet (cdc_ether), 6C:2B:59:BD:48:2E, hw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 8.8.8.8
        interface: vlan112
  • dmesg on driver load
[ 9360.992283] i40e: Intel(R) 40-10 Gigabit Ethernet Connection Network Driver - version 2.17.15
[ 9360.995809] i40e: Copyright(c) 2013 - 2021 Intel Corporation.
[ 9361.011858] i40e 0000:18:00.0: fw 8.84.66032 api 1.14 nvm 8.40 0x8000af8b 20.5.13
[ 9361.253524] i40e 0000:18:00.0: MAC address: e4:43:4b:d6:88:a0
[ 9361.257798] i40e 0000:18:00.0: FW LLDP is enabled
[ 9361.262796] Non-contiguous TC - Disabling DCB
[ 9361.270619] i40e 0000:18:00.0: couldn't add VEB, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
[ 9361.274578] i40e 0000:18:00.0: couldn't add VEB
[ 9361.275158] i40e 0000:18:00.0 eno1: renamed from eth0
[ 9361.279609] i40e 0000:18:00.0: Couldn't create FDir VSI
[ 9361.280482] i40e 0000:18:00.0 eno1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 9361.295860] i40e 0000:18:00.0: PCI-Express: Speed 8.0GT/s Width x8
[ 9361.302590] i40e 0000:18:00.0: Features: PF-id[0] VFs: 32 VSIs: 34 QP: 64 RSS FD_ATR CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[ 9361.321483] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[ 9361.323187] i40e 0000:18:00.1: fw 8.84.66032 api 1.14 nvm 8.40 0x8000af8b 20.5.13
[ 9361.340779] 8021q: adding VLAN 0 to HW filter on device eno1
[ 9361.348342] i40e 0000:18:00.0 eno1: Changing settings is not supported on backplane.
[ 9361.355321] i40e 0000:18:00.0 eno1: changing MTU from 1500 to 9000
[ 9361.367208] i40e 0000:18:00.0: VSI seid 397 Tx ring 0 disable timeout
[ 9361.569611] i40e 0000:18:00.1: MAC address: e4:43:4b:d6:88:a1
[ 9361.578053] i40e 0000:18:00.1: FW LLDP is enabled
[ 9361.584306] Non-contiguous TC - Disabling DCB
[ 9361.679960] i40e 0000:18:00.0: VSI seid 397 Tx ring 0 disable timeout
[ 9361.743593] i40e 0000:18:00.0 eno1: already using mac address e4:43:4b:d6:88:a0
[ 9361.743639] i40e 0000:18:00.1: couldn't add VEB, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
[ 9361.756839] irq 41: Affinity broken due to vector space exhaustion.
[ 9361.777214] i40e 0000:18:00.1: couldn't add VEB
[ 9361.777235] irq 43: Affinity broken due to vector space exhaustion.
[ 9361.788961] i40e 0000:18:00.1: Couldn't create FDir VSI
[ 9361.798858] irq 45: Affinity broken due to vector space exhaustion.
[ 9361.809198] i40e 0000:18:00.1 eth0: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 9361.817890] irq 47: Affinity broken due to vector space exhaustion.
[ 9361.835318] irq 49: Affinity broken due to vector space exhaustion.
[ 9361.835338] i40e 0000:18:00.1: PCI-Express: Speed 8.0GT/s Width x8
[ 9361.845701] irq 51: Affinity broken due to vector space exhaustion.
[ 9361.855452] i40e 0000:18:00.1: Features: PF-id[1] VFs: 32 VSIs: 34 QP: 64 RSS FD_ATR CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[ 9361.872930] irq 53: Affinity broken due to vector space exhaustion.
[ 9361.895973] irq 55: Affinity broken due to vector space exhaustion.
[ 9361.897916] i40e 0000:18:00.2: fw 8.84.66032 api 1.14 nvm 8.40 0x8000af8b 20.5.13
[ 9361.906531] irq 57: Affinity broken due to vector space exhaustion.
[ 9361.928088] irq 59: Affinity broken due to vector space exhaustion.
[ 9361.938286] irq 61: Affinity broken due to vector space exhaustion.
[ 9361.948975] irq 63: Affinity broken due to vector space exhaustion.
[ 9361.959496] irq 65: Affinity broken due to vector space exhaustion.
[ 9361.969527] irq 67: Affinity broken due to vector space exhaustion.
[ 9361.978687] irq 69: Affinity broken due to vector space exhaustion.
[ 9361.987616] irq 71: Affinity broken due to vector space exhaustion.
[ 9361.995238] irq 73: Affinity broken due to vector space exhaustion.
[ 9362.002072] irq 75: Affinity broken due to vector space exhaustion.
[ 9362.010309] irq 77: Affinity broken due to vector space exhaustion.
[ 9362.023175] irq 79: Affinity broken due to vector space exhaustion.
[ 9362.034130] irq 81: Affinity broken due to vector space exhaustion.
[ 9362.043564] irq 83: Affinity broken due to vector space exhaustion.
[ 9362.052298] irq 85: Affinity broken due to vector space exhaustion.
[ 9362.061028] irq 87: Affinity broken due to vector space exhaustion.
[ 9362.069519] irq 89: Affinity broken due to vector space exhaustion.
[ 9362.076285] irq 91: Affinity broken due to vector space exhaustion.
[ 9362.082833] irq 93: Affinity broken due to vector space exhaustion.
[ 9362.090376] irq 95: Affinity broken due to vector space exhaustion.
[ 9362.099385] irq 97: Affinity broken due to vector space exhaustion.
[ 9362.106210] irq 99: Affinity broken due to vector space exhaustion.
[ 9362.110978] irq 101: Affinity broken due to vector space exhaustion.
[ 9362.116313] irq 103: Affinity broken due to vector space exhaustion.
[ 9362.157245] i40e 0000:18:00.2: MAC address: e4:43:4b:d6:88:a2
[ 9362.164947] i40e 0000:18:00.2: FW LLDP is enabled
[ 9362.172644] Non-contiguous TC - Disabling DCB
[ 9362.224272] 8021q: adding VLAN 0 to HW filter on device eno1
[ 9362.231441] bond0: (slave eno1): Enslaving as a backup interface with an up link
[ 9362.238975] i40e 0000:18:00.1 eno2: renamed from eth0
[ 9362.244791] i40e 0000:18:00.2: couldn't add VEB, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
[ 9362.249822] i40e 0000:18:00.2: couldn't add VEB
[ 9362.255894] i40e 0000:18:00.2: Couldn't create FDir VSI
[ 9362.267080] i40e 0000:18:00.2 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 9362.273065] i40e 0000:18:00.2: PCI-Express: Speed 8.0GT/s Width x8
[ 9362.281047] i40e 0000:18:00.2: Features: PF-id[2] VFs: 32 VSIs: 34 QP: 64 RSS FD_ATR CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[ 9362.293274] i40e 0000:18:00.2 eno3: renamed from eth1
[ 9362.302532] i40e 0000:18:00.3: fw 8.84.66032 api 1.14 nvm 8.40 0x8000af8b 20.5.13
[ 9362.547527] i40e 0000:18:00.3: MAC address: e4:43:4b:d6:88:a3
[ 9362.553715] i40e 0000:18:00.3: FW LLDP is enabled
[ 9362.559862] Non-contiguous TC - Disabling DCB
[ 9362.573128] i40e 0000:18:00.3: couldn't add VEB, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EINVAL
[ 9362.576426] i40e 0000:18:00.3 eno4: renamed from eth0
[ 9362.578311] i40e 0000:18:00.3: couldn't add VEB
[ 9362.578314] i40e 0000:18:00.3: Couldn't create FDir VSI
[ 9362.596808] i40e 0000:18:00.3 eno4: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 9362.603353] i40e 0000:18:00.3: PCI-Express: Speed 8.0GT/s Width x8
[ 9362.610067] i40e 0000:18:00.3: Features: PF-id[3] VFs: 32 VSIs: 34 QP: 64 RSS FD_ATR CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[ 9362.623914] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[ 9362.640254] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[ 9362.646601] 8021q: adding VLAN 0 to HW filter on device eno2
[ 9362.657300] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[ 9362.664050] IPv6: ADDRCONF(NETDEV_UP): eno3: link is not ready
[ 9362.679972] 8021q: adding VLAN 0 to HW filter on device eno3
[ 9362.691793] IPv6: ADDRCONF(NETDEV_UP): eno4: link is not ready
[ 9362.706336] 8021q: adding VLAN 0 to HW filter on device eno4
[ 9362.715480] i40e 0000:18:00.1 eno2: Changing settings is not supported on backplane.
[ 9362.720936] i40e 0000:18:00.1 eno2: changing MTU from 1500 to 9000
[ 9362.727365] i40e 0000:18:00.1: VSI seid 396 Tx ring 0 disable timeout
[ 9362.998697] i40e 0000:18:00.1: VSI seid 396 Tx ring 0 disable timeout
[ 9363.059029] i40e 0000:18:00.1 eno2: set new mac address e4:43:4b:d6:88:a0
[ 9363.068747] irq 123: Affinity broken due to vector space exhaustion.
[ 9363.074544] irq 125: Affinity broken due to vector space exhaustion.
[ 9363.080450] irq 127: Affinity broken due to vector space exhaustion.
[ 9363.086106] irq 129: Affinity broken due to vector space exhaustion.
[ 9363.092518] irq 131: Affinity broken due to vector space exhaustion.
[ 9363.100790] irq 133: Affinity broken due to vector space exhaustion.
[ 9363.113052] irq 135: Affinity broken due to vector space exhaustion.
[ 9363.124566] irq 137: Affinity broken due to vector space exhaustion.
[ 9363.132924] irq 139: Affinity broken due to vector space exhaustion.
[ 9363.140423] irq 141: Affinity broken due to vector space exhaustion.
[ 9363.148555] irq 143: Affinity broken due to vector space exhaustion.
[ 9363.157013] irq 145: Affinity broken due to vector space exhaustion.
[ 9363.164536] irq 147: Affinity broken due to vector space exhaustion.
[ 9363.171070] irq 149: Affinity broken due to vector space exhaustion.
[ 9363.176325] irq 151: Affinity broken due to vector space exhaustion.
[ 9363.182656] irq 153: Affinity broken due to vector space exhaustion.
[ 9363.189655] irq 155: Affinity broken due to vector space exhaustion.
[ 9363.196317] irq 157: Affinity broken due to vector space exhaustion.
[ 9363.202274] irq 159: Affinity broken due to vector space exhaustion.
[ 9363.207554] irq 161: Affinity broken due to vector space exhaustion.
[ 9363.212065] irq 163: Affinity broken due to vector space exhaustion.
[ 9363.217442] irq 165: Affinity broken due to vector space exhaustion.
[ 9363.223601] irq 167: Affinity broken due to vector space exhaustion.
[ 9363.231686] irq 169: Affinity broken due to vector space exhaustion.
[ 9363.237975] irq 171: Affinity broken due to vector space exhaustion.
[ 9363.242444] irq 173: Affinity broken due to vector space exhaustion.
[ 9363.247450] irq 175: Affinity broken due to vector space exhaustion.
[ 9363.251831] irq 177: Affinity broken due to vector space exhaustion.
[ 9363.255331] irq 179: Affinity broken due to vector space exhaustion.
[ 9363.258281] irq 181: Affinity broken due to vector space exhaustion.
[ 9363.261518] irq 183: Affinity broken due to vector space exhaustion.
[ 9363.267130] irq 185: Affinity broken due to vector space exhaustion.
[ 9363.377135] 8021q: adding VLAN 0 to HW filter on device eno2
[ 9363.382728] bond0: (slave eno2): Enslaving as a backup interface with an up link
[ 9363.387602] i40e 0000:18:00.2 eno3: Changing settings is not supported on backplane.
[ 9363.390489] i40e 0000:18:00.2 eno3: changing MTU from 1500 to 9000
[ 9363.393586] i40e 0000:18:00.2: VSI seid 398 Tx ring 0 disable timeout
[ 9363.558062] i40e 0000:18:00.3 eno4: Changing settings is not supported on backplane.
[ 9363.561002] i40e 0000:18:00.3 eno4: changing MTU from 1500 to 9000
[ 9363.564139] i40e 0000:18:00.3: VSI seid 399 Tx ring 0 disable timeout
[ 9367.948999] i40e 0000:18:00.0 eno1: tx_timeout: VSI_seid: 397, Q 40, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 9367.952328] i40e 0000:18:00.0 eno1: tx_timeout recovery level 1, hung_queue 40
[ 9367.957074] bond0: (slave eno1): link status definitely down, disabling slave
[ 9368.056993] i40e 0000:18:00.0: VSI seid 397 Tx ring 0 disable timeout
[ 9368.130993] Non-contiguous TC - Disabling DCB
[ 9368.140146] irq 41: Affinity broken due to vector space exhaustion.
[ 9368.143588] irq 43: Affinity broken due to vector space exhaustion.
[ 9368.146626] irq 45: Affinity broken due to vector space exhaustion.
[ 9368.149629] irq 47: Affinity broken due to vector space exhaustion.
[ 9368.152875] irq 49: Affinity broken due to vector space exhaustion.
[ 9368.156364] irq 51: Affinity broken due to vector space exhaustion.
[ 9368.159422] irq 53: Affinity broken due to vector space exhaustion.
[ 9368.162739] irq 55: Affinity broken due to vector space exhaustion.
[ 9368.166070] irq 57: Affinity broken due to vector space exhaustion.
[ 9368.169639] irq 59: Affinity broken due to vector space exhaustion.
[ 9368.172599] irq 61: Affinity broken due to vector space exhaustion.
[ 9368.175565] irq 63: Affinity broken due to vector space exhaustion.
[ 9368.178432] irq 65: Affinity broken due to vector space exhaustion.
[ 9368.181476] irq 67: Affinity broken due to vector space exhaustion.
[ 9368.185635] irq 69: Affinity broken due to vector space exhaustion.
[ 9368.189681] irq 71: Affinity broken due to vector space exhaustion.
[ 9368.196722] irq 73: Affinity broken due to vector space exhaustion.
[ 9368.200706] irq 75: Affinity broken due to vector space exhaustion.
[ 9368.204319] irq 77: Affinity broken due to vector space exhaustion.
[ 9368.207927] irq 79: Affinity broken due to vector space exhaustion.
[ 9368.211545] irq 81: Affinity broken due to vector space exhaustion.
[ 9368.215164] irq 83: Affinity broken due to vector space exhaustion.
[ 9368.218679] irq 85: Affinity broken due to vector space exhaustion.
[ 9368.221838] irq 87: Affinity broken due to vector space exhaustion.
[ 9368.225524] irq 89: Affinity broken due to vector space exhaustion.
[ 9368.229111] irq 91: Affinity broken due to vector space exhaustion.
[ 9368.232113] irq 93: Affinity broken due to vector space exhaustion.
[ 9368.234999] irq 95: Affinity broken due to vector space exhaustion.
[ 9368.237855] irq 97: Affinity broken due to vector space exhaustion.
[ 9368.240756] irq 99: Affinity broken due to vector space exhaustion.
[ 9368.243753] irq 101: Affinity broken due to vector space exhaustion.
[ 9368.249062] irq 103: Affinity broken due to vector space exhaustion.
[ 9368.254647] i40e 0000:18:00.0: VF BW shares not restored
[ 9368.255078] bond0: (slave eno1): link status definitely up, 10000 Mbps full duplex
[ 9368.264135] bond0: active interface up!
[ 9368.655389] i40e 0000:18:00.0: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
[ 9368.973001] i40e 0000:18:00.1 eno2: tx_timeout: VSI_seid: 396, Q 29, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 9368.976459] i40e 0000:18:00.1 eno2: tx_timeout recovery level 1, hung_queue 29
[ 9368.980003] bond0: (slave eno2): link status definitely down, disabling slave
[ 9369.082784] i40e 0000:18:00.1: VSI seid 396 Tx ring 0 disable timeout
[ 9369.155858] Non-contiguous TC - Disabling DCB
[ 9369.166283] irq 123: Affinity broken due to vector space exhaustion.
[ 9369.169903] irq 125: Affinity broken due to vector space exhaustion.
[ 9369.173502] irq 127: Affinity broken due to vector space exhaustion.
[ 9369.176927] irq 129: Affinity broken due to vector space exhaustion.
[ 9369.180082] irq 131: Affinity broken due to vector space exhaustion.
[ 9369.184140] irq 133: Affinity broken due to vector space exhaustion.
[ 9369.187255] irq 135: Affinity broken due to vector space exhaustion.
[ 9369.190298] irq 137: Affinity broken due to vector space exhaustion.
[ 9369.194028] irq 139: Affinity broken due to vector space exhaustion.
[ 9369.199852] irq 141: Affinity broken due to vector space exhaustion.
[ 9369.205360] irq 143: Affinity broken due to vector space exhaustion.
[ 9369.212332] irq 145: Affinity broken due to vector space exhaustion.
[ 9369.216205] irq 147: Affinity broken due to vector space exhaustion.
[ 9369.219440] irq 149: Affinity broken due to vector space exhaustion.
[ 9369.223262] irq 151: Affinity broken due to vector space exhaustion.
[ 9369.226831] irq 153: Affinity broken due to vector space exhaustion.
[ 9369.230199] irq 155: Affinity broken due to vector space exhaustion.
[ 9369.233033] irq 157: Affinity broken due to vector space exhaustion.
[ 9369.236053] irq 159: Affinity broken due to vector space exhaustion.
[ 9369.239895] irq 161: Affinity broken due to vector space exhaustion.
[ 9369.243609] irq 163: Affinity broken due to vector space exhaustion.
[ 9369.249981] irq 165: Affinity broken due to vector space exhaustion.
[ 9369.257339] irq 167: Affinity broken due to vector space exhaustion.
[ 9369.261566] irq 169: Affinity broken due to vector space exhaustion.
[ 9369.264720] irq 171: Affinity broken due to vector space exhaustion.
[ 9369.268388] irq 173: Affinity broken due to vector space exhaustion.
[ 9369.272458] irq 175: Affinity broken due to vector space exhaustion.
[ 9369.276208] irq 177: Affinity broken due to vector space exhaustion.
[ 9369.280523] irq 179: Affinity broken due to vector space exhaustion.
[ 9369.284731] irq 181: Affinity broken due to vector space exhaustion.
[ 9369.288832] irq 183: Affinity broken due to vector space exhaustion.
[ 9369.292024] irq 185: Affinity broken due to vector space exhaustion.
[ 9369.295036] bond0: (slave eno2): link status definitely up, 10000 Mbps full duplex
[ 9369.295326] i40e 0000:18:00.1: VF BW shares not restored
[ 9369.997012] i40e 0000:18:00.3 eno4: tx_timeout: VSI_seid: 399, Q 1, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 9370.000504] i40e 0000:18:00.3 eno4: tx_timeout recovery level 1, hung_queue 1
[ 9370.003878] i40e 0000:18:00.2 eno3: tx_timeout: VSI_seid: 398, Q 1, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 9370.007592] i40e 0000:18:00.2 eno3: tx_timeout recovery level 1, hung_queue 1
[ 9370.113844] i40e 0000:18:00.3: VSI seid 399 Tx ring 0 disable timeout
[ 9370.117432] i40e 0000:18:00.2: VSI seid 398 Tx ring 0 disable timeout
[ 9370.229396] Non-contiguous TC - Disabling DCB
[ 9370.238564] irq 272: Affinity broken due to vector space exhaustion.
[ 9370.242011] irq 274: Affinity broken due to vector space exhaustion.
[ 9370.245237] irq 276: Affinity broken due to vector space exhaustion.
[ 9370.248259] irq 278: Affinity broken due to vector space exhaustion.
[ 9370.251159] irq 280: Affinity broken due to vector space exhaustion.
[ 9370.253833] irq 282: Affinity broken due to vector space exhaustion.
[ 9370.256455] irq 284: Affinity broken due to vector space exhaustion.
[ 9370.258835] irq 286: Affinity broken due to vector space exhaustion.
[ 9370.265333] irq 288: Affinity broken due to vector space exhaustion.
[ 9370.268578] irq 290: Affinity broken due to vector space exhaustion.
[ 9370.271315] irq 292: Affinity broken due to vector space exhaustion.
[ 9370.273851] irq 294: Affinity broken due to vector space exhaustion.
[ 9370.276801] irq 296: Affinity broken due to vector space exhaustion.
[ 9370.282307] Non-contiguous TC - Disabling DCB
[ 9370.285533] irq 298: Affinity broken due to vector space exhaustion.
[ 9370.288795] irq 300: Affinity broken due to vector space exhaustion.
[ 9370.292016] irq 302: Affinity broken due to vector space exhaustion.
[ 9370.295321] irq 304: Affinity broken due to vector space exhaustion.
[ 9370.297969] irq 306: Affinity broken due to vector space exhaustion.
[ 9370.300674] irq 308: Affinity broken due to vector space exhaustion.
[ 9370.303397] irq 310: Affinity broken due to vector space exhaustion.
[ 9370.306099] irq 312: Affinity broken due to vector space exhaustion.
[ 9370.308754] irq 314: Affinity broken due to vector space exhaustion.
[ 9370.311691] irq 316: Affinity broken due to vector space exhaustion.
[ 9370.314502] irq 318: Affinity broken due to vector space exhaustion.
[ 9370.319058] irq 320: Affinity broken due to vector space exhaustion.
[ 9370.323560] irq 322: Affinity broken due to vector space exhaustion.
[ 9370.328378] irq 324: Affinity broken due to vector space exhaustion.
[ 9370.334073] irq 326: Affinity broken due to vector space exhaustion.
[ 9370.340334] irq 328: Affinity broken due to vector space exhaustion.
[ 9370.343290] irq 330: Affinity broken due to vector space exhaustion.
[ 9370.346271] irq 332: Affinity broken due to vector space exhaustion.
[ 9370.348959] irq 334: Affinity broken due to vector space exhaustion.
[ 9370.352191] i40e 0000:18:00.3: VF BW shares not restored
[ 9370.360853] irq 198: Affinity broken due to vector space exhaustion.
[ 9370.364296] irq 200: Affinity broken due to vector space exhaustion.
[ 9370.368294] irq 202: Affinity broken due to vector space exhaustion.
[ 9370.371851] irq 204: Affinity broken due to vector space exhaustion.
[ 9370.377380] irq 206: Affinity broken due to vector space exhaustion.
[ 9370.381025] irq 208: Affinity broken due to vector space exhaustion.
[ 9370.386165] irq 210: Affinity broken due to vector space exhaustion.
[ 9370.390945] irq 212: Affinity broken due to vector space exhaustion.
[ 9370.394652] irq 214: Affinity broken due to vector space exhaustion.
[ 9370.397339] irq 216: Affinity broken due to vector space exhaustion.
[ 9370.399804] irq 218: Affinity broken due to vector space exhaustion.
[ 9370.402115] irq 220: Affinity broken due to vector space exhaustion.
[ 9370.404543] irq 222: Affinity broken due to vector space exhaustion.
[ 9370.406972] irq 224: Affinity broken due to vector space exhaustion.
[ 9370.408866] irq 226: Affinity broken due to vector space exhaustion.
[ 9370.411069] irq 228: Affinity broken due to vector space exhaustion.
[ 9370.412884] irq 230: Affinity broken due to vector space exhaustion.
[ 9370.414672] irq 232: Affinity broken due to vector space exhaustion.
[ 9370.416524] irq 234: Affinity broken due to vector space exhaustion.
[ 9370.418135] irq 236: Affinity broken due to vector space exhaustion.
[ 9370.419747] irq 238: Affinity broken due to vector space exhaustion.
[ 9370.421440] irq 240: Affinity broken due to vector space exhaustion.
[ 9370.423274] irq 242: Affinity broken due to vector space exhaustion.
[ 9370.425319] irq 244: Affinity broken due to vector space exhaustion.
[ 9370.427525] irq 246: Affinity broken due to vector space exhaustion.
[ 9370.429377] irq 248: Affinity broken due to vector space exhaustion.
[ 9370.431167] irq 250: Affinity broken due to vector space exhaustion.
[ 9370.432736] irq 252: Affinity broken due to vector space exhaustion.
[ 9370.434211] irq 254: Affinity broken due to vector space exhaustion.
[ 9370.435931] irq 256: Affinity broken due to vector space exhaustion.
[ 9370.437967] irq 258: Affinity broken due to vector space exhaustion.
[ 9370.440452] irq 260: Affinity broken due to vector space exhaustion.
[ 9370.443233] i40e 0000:18:00.2: VF BW shares not restored
[ 9374.093068] i40e 0000:18:00.0 eno1: tx_timeout: VSI_seid: 397, Q 37, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 9374.094823] i40e 0000:18:00.0 eno1: tx_timeout recovery level 2, hung_queue 37
[ 9374.097121] i40e 0000:18:00.1: VSI seid 396 Tx ring 0 disable timeout
[ 9374.097202] i40e 0000:18:00.2: VSI seid 398 Tx ring 0 disable timeout
[ 9374.097202] i40e 0000:18:00.3: VSI seid 399 Tx ring 0 disable timeout
[ 9374.099274] i40e 0000:18:00.0: VSI seid 397 Tx ring 0 disable timeout
[ 9374.193093] bond0: (slave eno1): link status definitely down, disabling slave
[ 9374.195226] bond0: now running without any active interface!
[ 9374.197605] bond0: (slave eno2): link status definitely down, disabling slave
[ 9377.279925] Non-contiguous TC - Disabling DCB
[ 9377.289877] i40e 0000:18:00.1: VF BW shares not restored
[ 9377.291144] bond0: (slave eno2): link status definitely up, 10000 Mbps full duplex
[ 9377.295927] bond0: active interface up!
[ 9377.305541] Non-contiguous TC - Disabling DCB
[ 9377.312154] Non-contiguous TC - Disabling DCB
[ 9377.314410] irq 272: Affinity broken due to vector space exhaustion.
[ 9377.322243] irq 274: Affinity broken due to vector space exhaustion.
[ 9377.328842] irq 276: Affinity broken due to vector space exhaustion.
[ 9377.332151] irq 278: Affinity broken due to vector space exhaustion.
[ 9377.334862] Non-contiguous TC - Disabling DCB
[ 9377.336123] irq 280: Affinity broken due to vector space exhaustion.
[ 9377.343149] irq 282: Affinity broken due to vector space exhaustion.
[ 9377.347168] irq 284: Affinity broken due to vector space exhaustion.
[ 9377.350949] irq 286: Affinity broken due to vector space exhaustion.
[ 9377.354467] irq 288: Affinity broken due to vector space exhaustion.
[ 9377.358198] irq 290: Affinity broken due to vector space exhaustion.
[ 9377.361400] irq 292: Affinity broken due to vector space exhaustion.
[ 9377.364543] irq 294: Affinity broken due to vector space exhaustion.
[ 9377.367611] irq 296: Affinity broken due to vector space exhaustion.
[ 9377.370626] irq 298: Affinity broken due to vector space exhaustion.
[ 9377.375911] irq 300: Affinity broken due to vector space exhaustion.
[ 9377.381046] irq 302: Affinity broken due to vector space exhaustion.
[ 9377.387629] irq 304: Affinity broken due to vector space exhaustion.
[ 9377.395577] irq 306: Affinity broken due to vector space exhaustion.
[ 9377.398647] irq 308: Affinity broken due to vector space exhaustion.
[ 9377.401694] irq 310: Affinity broken due to vector space exhaustion.
[ 9377.404647] irq 312: Affinity broken due to vector space exhaustion.
[ 9377.407683] irq 314: Affinity broken due to vector space exhaustion.
[ 9377.410793] irq 316: Affinity broken due to vector space exhaustion.
[ 9377.413933] irq 318: Affinity broken due to vector space exhaustion.
[ 9377.417131] irq 320: Affinity broken due to vector space exhaustion.
[ 9377.420116] irq 322: Affinity broken due to vector space exhaustion.
[ 9377.423478] irq 324: Affinity broken due to vector space exhaustion.
[ 9377.427308] irq 326: Affinity broken due to vector space exhaustion.
[ 9377.431034] irq 328: Affinity broken due to vector space exhaustion.
[ 9377.434595] irq 330: Affinity broken due to vector space exhaustion.
[ 9377.438424] irq 332: Affinity broken due to vector space exhaustion.
[ 9377.441549] irq 334: Affinity broken due to vector space exhaustion.
[ 9377.445292] i40e 0000:18:00.3: VF BW shares not restored
[ 9377.453323] i40e 0000:18:00.0: VF BW shares not restored
[ 9377.459666] irq 198: Affinity broken due to vector space exhaustion.
[ 9377.462751] irq 200: Affinity broken due to vector space exhaustion.
[ 9377.467383] irq 202: Affinity broken due to vector space exhaustion.
[ 9377.471952] irq 204: Affinity broken due to vector space exhaustion.
[ 9377.476872] irq 206: Affinity broken due to vector space exhaustion.
[ 9377.479991] irq 208: Affinity broken due to vector space exhaustion.
[ 9377.482927] irq 210: Affinity broken due to vector space exhaustion.
[ 9377.485834] irq 212: Affinity broken due to vector space exhaustion.
[ 9377.488392] irq 214: Affinity broken due to vector space exhaustion.
[ 9377.490292] irq 216: Affinity broken due to vector space exhaustion.
[ 9377.492168] irq 218: Affinity broken due to vector space exhaustion.
[ 9377.493854] irq 220: Affinity broken due to vector space exhaustion.
[ 9377.495682] irq 222: Affinity broken due to vector space exhaustion.
[ 9377.497282] irq 224: Affinity broken due to vector space exhaustion.
[ 9377.498848] irq 226: Affinity broken due to vector space exhaustion.
[ 9377.500448] irq 228: Affinity broken due to vector space exhaustion.
[ 9377.502082] irq 230: Affinity broken due to vector space exhaustion.
[ 9377.503750] irq 232: Affinity broken due to vector space exhaustion.
[ 9377.505391] irq 234: Affinity broken due to vector space exhaustion.
[ 9377.506953] irq 236: Affinity broken due to vector space exhaustion.
[ 9377.508430] irq 238: Affinity broken due to vector space exhaustion.
[ 9377.509948] irq 240: Affinity broken due to vector space exhaustion.
[ 9377.511631] irq 242: Affinity broken due to vector space exhaustion.
[ 9377.513148] irq 244: Affinity broken due to vector space exhaustion.
[ 9377.514648] irq 246: Affinity broken due to vector space exhaustion.
[ 9377.516471] irq 248: Affinity broken due to vector space exhaustion.
[ 9377.517938] irq 250: Affinity broken due to vector space exhaustion.
[ 9377.519293] irq 252: Affinity broken due to vector space exhaustion.
[ 9377.520738] irq 254: Affinity broken due to vector space exhaustion.
[ 9377.522144] irq 256: Affinity broken due to vector space exhaustion.
[ 9377.523495] irq 258: Affinity broken due to vector space exhaustion.
[ 9377.524846] irq 260: Affinity broken due to vector space exhaustion.
[ 9377.526810] i40e 0000:18:00.2: VF BW shares not restored
[ 9377.528138] bond0: (slave eno1): link status definitely up, 10000 Mbps full duplex
[ 9377.529094] i40e 0000:18:00.0: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on

 

Labels (1)
0 Kudos
0 Replies
Reply