Hello,
We have encountered an issue related to the Intel E810-C card (100Gx2). We have conducted tests on Debian 12.5 and Debian 11.10, downloading both operating systems from the official Debian site: debian-11.10.0-amd64-netinst.iso and debian-12.5.0-amd64-netinst.iso.
We noticed that, when installing the E810-C drivers: ice 1.14.11, we encountered warnings during compilation (in both Debian 11 and Debian 12) due to numerous errors in the symbolic links: depmod: WARNING unknown symbol.
The kernel version of Debian 11 is: 5.10.0-30-amd64.
According to Intel's official site, Debian 11 is compatible with the E810-C card.
The driver ends up installing, but we are not sure of achieving its maximum performance due to these warning messages in the symbolic links, as can be seen in ethtool: 1.14.11
ethtool -i enp3s0f0
driver: ice
version: 1.14.11
firmware-version: 4.50 0x8001d8b6 1.3597.0
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
When using the generic drivers on Debian 12, version: 6.1.0-21-amd64, we observed improved performance in forwarding traffic, reaching 40 Gbps/4 million packets.
However, we noticed degradation beginning, specifically packet loss in the kernel, when one or two IRQs rose to 100%, recording `rx_dropped: 99286`. In this scenario, with the generic drivers, we have an XDP code to mitigate DoS/DDoS attacks injected into the board in `xdpdrv` mode. By disabling XDP on the interface, we noticed that the problem was resolved.
We thought it might be because we were using the generic Debian drivers: 6.1.0-21-amd64, so we decided to install the Intel drivers, ice 1.14.11.
However, this worsened the performance, even without the injection of the XDP code to protect the network from DoS/DDoS attacks, achieving only 15/20 Gbps of forwarding traffic through the kernel stack.
ethtool -i enp3s0f0
driver: ice
version: 6.1.0-21-amd64
firmware-version: 4.50 0x8001d8b6 1.3597.0
expansion-rom-version:
bus-info: 0000:81:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
We have read all of Intel's optimization instructions: 636781_800 Series Linux Performance Tuning Guide, applying adjustments in the BIOS, turning off virtualization, and disabling HT technology so that only logical cores are used.
We have also adjusted the settings of the E810-C interface, including RX/TX Combined, IRQ settings for local numa, remote numa, and all, as well as the configuration of interrupts with rx-usecs and tx-usecs. However, we were not successful in running the ice drivers; we did not exceed 20 Gbps without XDP. With the generic drivers, we surpassed 40 Gbps of forwarding traffic with XDP active to mitigate DoS/DDoS attacks, but upon reaching 4 million packets, the issue recurs.
- ./set_irq_affinity all enp3s0f0 (I also tried configuring it on local numa and remote numa)
- ethtool -L enp3s0f0 combined 36
- ethtool -G enp3s0f0 rx 1024 tx 1024 (I also tried pushing it to the maximum: 8160)
- ethtool -C enp3s0f0 adaptive-rx off adaptive-tx off ethtool -C ethX rx-usecs 64 tx-usecs 64 (I tried various values of rx-usecs and tx-usecs)
- ethtool -X ens6f1 weight 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (I tried adjusting each flow manually, even isolating specific flows with flow-type)
I verified that the E810-C card is on numa 0 to avoid potential degradation.
cat /sys/class/net/enp3s0f0/device/numa_node
0
However, I would like to clarify that to mitigate DoS and DDoS attacks, meaning traffic that is dropped with XDP_DROP without passing through the kernel stack, we achieved rates of 70/90 Gbps without issues.
I also configured the PCI bifurcation of the E810-C card to 8x8x, although I am currently only using port 0, with a single 100G cable connected.
ethtool enp3s0f0
Settings for enp3s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 25000baseCR/Full
25000baseSR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
50000baseSR2/Full
50000baseSR/Full
50000baseCR/Full
50000baseLR_ER_FR/Full
100000baseSR2/Full
100000baseCR2/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: None
Advertised link modes: 25000baseSR/Full
100000baseSR4/Full
50000baseSR2/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: None
Speed: 100000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
The processor is a dual: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
The motherboard is: Supermicro X10DRi
Attached is a photo of the E810-C card:
We do not have a LAG configuration.
RSS (Receive Side Scaling):
ethtool -x enp3s0f0
RX flow hash indirection table for enp3s0f0 with 36 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 16 17 18 19 20 21 22 23
24: 24 25 26 27 28 29 30 31
32: 32 33 34 35 0 1 2 3
40: 4 5 6 7 8 9 10 11
48: 12 13 14 15 16 17 18 19
56: 20 21 22 23 24 25 26 27
64: 28 29 30 31 32 33 34 35
72: 0 1 2 3 4 5 6 7
80: 8 9 10 11 12 13 14 15
88: 16 17 18 19 20 21 22 23
96: 24 25 26 27 28 29 30 31
104: 32 33 34 35 0 1 2 3
112: 4 5 6 7 8 9 10 11
120: 12 13 14 15 16 17 18 19
128: 20 21 22 23 24 25 26 27
136: 28 29 30 31 32 33 34 35
144: 0 1 2 3 4 5 6 7
152: 8 9 10 11 12 13 14 15
160: 16 17 18 19 20 21 22 23
168: 24 25 26 27 28 29 30 31
176: 32 33 34 35 0 1 2 3
184: 4 5 6 7 8 9 10 11
192: 12 13 14 15 16 17 18 19
200: 20 21 22 23 24 25 26 27
208: 28 29 30 31 32 33 34 35
216: 0 1 2 3 4 5 6 7
224: 8 9 10 11 12 13 14 15
232: 16 17 18 19 20 21 22 23
240: 24 25 26 27 28 29 30 31
248: 32 33 34 35 0 1 2 3
256: 4 5 6 7 8 9 10 11
264: 12 13 14 15 16 17 18 19
272: 20 21 22 23 24 25 26 27
280: 28 29 30 31 32 33 34 35
288: 0 1 2 3 4 5 6 7
296: 8 9 10 11 12 13 14 15
304: 16 17 18 19 20 21 22 23
312: 24 25 26 27 28 29 30 31
320: 32 33 34 35 0 1 2 3
328: 4 5 6 7 8 9 10 11
336: 12 13 14 15 16 17 18 19
344: 20 21 22 23 24 25 26 27
352: 28 29 30 31 32 33 34 35
360: 0 1 2 3 4 5 6 7
368: 8 9 10 11 12 13 14 15
376: 16 17 18 19 20 21 22 23
384: 24 25 26 27 28 29 30 31
392: 32 33 34 35 0 1 2 3
400: 4 5 6 7 8 9 10 11
408: 12 13 14 15 16 17 18 19
416: 20 21 22 23 24 25 26 27
424: 28 29 30 31 32 33 34 35
432: 0 1 2 3 4 5 6 7
440: 8 9 10 11 12 13 14 15
448: 16 17 18 19 20 21 22 23
456: 24 25 26 27 28 29 30 31
464: 32 33 34 35 0 1 2 3
472: 4 5 6 7 8 9 10 11
480: 12 13 14 15 16 17 18 19
488: 20 21 22 23 24 25 26 27
496: 28 29 30 31 32 33 34 35
504: 0 1 2 3 4 5 6 7
512: 8 9 10 11 12 13 14 15
520: 16 17 18 19 20 21 22 23
528: 24 25 26 27 28 29 30 31
536: 32 33 34 35 0 1 2 3
544: 4 5 6 7 8 9 10 11
552: 12 13 14 15 16 17 18 19
560: 20 21 22 23 24 25 26 27
568: 28 29 30 31 32 33 34 35
576: 0 1 2 3 4 5 6 7
584: 8 9 10 11 12 13 14 15
592: 16 17 18 19 20 21 22 23
600: 24 25 26 27 28 29 30 31
608: 32 33 34 35 0 1 2 3
616: 4 5 6 7 8 9 10 11
624: 12 13 14 15 16 17 18 19
632: 20 21 22 23 24 25 26 27
640: 28 29 30 31 32 33 34 35
648: 0 1 2 3 4 5 6 7
656: 8 9 10 11 12 13 14 15
664: 16 17 18 19 20 21 22 23
672: 24 25 26 27 28 29 30 31
680: 32 33 34 35 0 1 2 3
688: 4 5 6 7 8 9 10 11
696: 12 13 14 15 16 17 18 19
704: 20 21 22 23 24 25 26 27
712: 28 29 30 31 32 33 34 35
720: 0 1 2 3 4 5 6 7
728: 8 9 10 11 12 13 14 15
736: 16 17 18 19 20 21 22 23
744: 24 25 26 27 28 29 30 31
752: 32 33 34 35 0 1 2 3
760: 4 5 6 7 8 9 10 11
768: 12 13 14 15 16 17 18 19
776: 20 21 22 23 24 25 26 27
784: 28 29 30 31 32 33 34 35
792: 0 1 2 3 4 5 6 7
800: 8 9 10 11 12 13 14 15
808: 16 17 18 19 20 21 22 23
816: 24 25 26 27 28 29 30 31
824: 32 33 34 35 0 1 2 3
832: 4 5 6 7 8 9 10 11
840: 12 13 14 15 16 17 18 19
848: 20 21 22 23 24 25 26 27
856: 28 29 30 31 32 33 34 35
864: 0 1 2 3 4 5 6 7
872: 8 9 10 11 12 13 14 15
880: 16 17 18 19 20 21 22 23
888: 24 25 26 27 28 29 30 31
896: 32 33 34 35 0 1 2 3
904: 4 5 6 7 8 9 10 11
912: 12 13 14 15 16 17 18 19
920: 20 21 22 23 24 25 26 27
928: 28 29 30 31 32 33 34 35
936: 0 1 2 3 4 5 6 7
944: 8 9 10 11 12 13 14 15
952: 16 17 18 19 20 21 22 23
960: 24 25 26 27 28 29 30 31
968: 32 33 34 35 0 1 2 3
976: 4 5 6 7 8 9 10 11
984: 12 13 14 15 16 17 18 19
992: 20 21 22 23 24 25 26 27
1000: 28 29 30 31 32 33 34 35
1008: 0 1 2 3 4 5 6 7
1016: 8 9 10 11 12 13 14 15
1024: 16 17 18 19 20 21 22 23
1032: 24 25 26 27 28 29 30 31
1040: 32 33 34 35 0 1 2 3
1048: 4 5 6 7 8 9 10 11
1056: 12 13 14 15 16 17 18 19
1064: 20 21 22 23 24 25 26 27
1072: 28 29 30 31 32 33 34 35
1080: 0 1 2 3 4 5 6 7
1088: 8 9 10 11 12 13 14 15
1096: 16 17 18 19 20 21 22 23
1104: 24 25 26 27 28 29 30 31
1112: 32 33 34 35 0 1 2 3
1120: 4 5 6 7 8 9 10 11
1128: 12 13 14 15 16 17 18 19
1136: 20 21 22 23 24 25 26 27
1144: 28 29 30 31 32 33 34 35
1152: 0 1 2 3 4 5 6 7
1160: 8 9 10 11 12 13 14 15
1168: 16 17 18 19 20 21 22 23
1176: 24 25 26 27 28 29 30 31
1184: 32 33 34 35 0 1 2 3
1192: 4 5 6 7 8 9 10 11
1200: 12 13 14 15 16 17 18 19
1208: 20 21 22 23 24 25 26 27
1216: 28 29 30 31 32 33 34 35
1224: 0 1 2 3 4 5 6 7
1232: 8 9 10 11 12 13 14 15
1240: 16 17 18 19 20 21 22 23
1248: 24 25 26 27 28 29 30 31
1256: 32 33 34 35 0 1 2 3
1264: 4 5 6 7 8 9 10 11
1272: 12 13 14 15 16 17 18 19
1280: 20 21 22 23 24 25 26 27
1288: 28 29 30 31 32 33 34 35
1296: 0 1 2 3 4 5 6 7
1304: 8 9 10 11 12 13 14 15
1312: 16 17 18 19 20 21 22 23
1320: 24 25 26 27 28 29 30 31
1328: 32 33 34 35 0 1 2 3
1336: 4 5 6 7 8 9 10 11
1344: 12 13 14 15 16 17 18 19
1352: 20 21 22 23 24 25 26 27
1360: 28 29 30 31 32 33 34 35
1368: 0 1 2 3 4 5 6 7
1376: 8 9 10 11 12 13 14 15
1384: 16 17 18 19 20 21 22 23
1392: 24 25 26 27 28 29 30 31
1400: 32 33 34 35 0 1 2 3
1408: 4 5 6 7 8 9 10 11
1416: 12 13 14 15 16 17 18 19
1424: 20 21 22 23 24 25 26 27
1432: 28 29 30 31 32 33 34 35
1440: 0 1 2 3 4 5 6 7
1448: 8 9 10 11 12 13 14 15
1456: 16 17 18 19 20 21 22 23
1464: 24 25 26 27 28 29 30 31
1472: 32 33 34 35 0 1 2 3
1480: 4 5 6 7 8 9 10 11
1488: 12 13 14 15 16 17 18 19
1496: 20 21 22 23 24 25 26 27
1504: 28 29 30 31 32 33 34 35
1512: 0 1 2 3 4 5 6 7
1520: 8 9 10 11 12 13 14 15
1528: 16 17 18 19 20 21 22 23
1536: 24 25 26 27 28 29 30 31
1544: 32 33 34 35 0 1 2 3
1552: 4 5 6 7 8 9 10 11
1560: 12 13 14 15 16 17 18 19
1568: 20 21 22 23 24 25 26 27
1576: 28 29 30 31 32 33 34 35
1584: 0 1 2 3 4 5 6 7
1592: 8 9 10 11 12 13 14 15
1600: 16 17 18 19 20 21 22 23
1608: 24 25 26 27 28 29 30 31
1616: 32 33 34 35 0 1 2 3
1624: 4 5 6 7 8 9 10 11
1632: 12 13 14 15 16 17 18 19
1640: 20 21 22 23 24 25 26 27
1648: 28 29 30 31 32 33 34 35
1656: 0 1 2 3 4 5 6 7
1664: 8 9 10 11 12 13 14 15
1672: 16 17 18 19 20 21 22 23
1680: 24 25 26 27 28 29 30 31
1688: 32 33 34 35 0 1 2 3
1696: 4 5 6 7 8 9 10 11
1704: 12 13 14 15 16 17 18 19
1712: 20 21 22 23 24 25 26 27
1720: 28 29 30 31 32 33 34 35
1728: 0 1 2 3 4 5 6 7
1736: 8 9 10 11 12 13 14 15
1744: 16 17 18 19 20 21 22 23
1752: 24 25 26 27 28 29 30 31
1760: 32 33 34 35 0 1 2 3
1768: 4 5 6 7 8 9 10 11
1776: 12 13 14 15 16 17 18 19
1784: 20 21 22 23 24 25 26 27
1792: 28 29 30 31 32 33 34 35
1800: 0 1 2 3 4 5 6 7
1808: 8 9 10 11 12 13 14 15
1816: 16 17 18 19 20 21 22 23
1824: 24 25 26 27 28 29 30 31
1832: 32 33 34 35 0 1 2 3
1840: 4 5 6 7 8 9 10 11
1848: 12 13 14 15 16 17 18 19
1856: 20 21 22 23 24 25 26 27
1864: 28 29 30 31 32 33 34 35
1872: 0 1 2 3 4 5 6 7
1880: 8 9 10 11 12 13 14 15
1888: 16 17 18 19 20 21 22 23
1896: 24 25 26 27 28 29 30 31
1904: 32 33 34 35 0 1 2 3
1912: 4 5 6 7 8 9 10 11
1920: 12 13 14 15 16 17 18 19
1928: 20 21 22 23 24 25 26 27
1936: 28 29 30 31 32 33 34 35
1944: 0 1 2 3 4 5 6 7
1952: 8 9 10 11 12 13 14 15
1960: 16 17 18 19 20 21 22 23
1968: 24 25 26 27 28 29 30 31
1976: 32 33 34 35 0 1 2 3
1984: 4 5 6 7 8 9 10 11
1992: 12 13 14 15 16 17 18 19
2000: 20 21 22 23 24 25 26 27
2008: 28 29 30 31 32 33 34 35
2016: 0 1 2 3 4 5 6 7
2024: 8 9 10 11 12 13 14 15
2032: 16 17 18 19 20 21 22 23
2040: 24 25 26 27 28 29 30 31
RSS hash key:
6b:b7:9a:f2:e9:23:65:e7:50:0d:6d:26:54:fd:15:86:cc:95:a0:02:a7:e6:a1:bd:14:60:69:6a:2c:1d:a7:b7:e1:65:16:0c:44:1b:c5:ea:58:6c:dd:99:b3:72:66:d7:f7:93:c4:cb
RSS hash function:
toeplitz: on
xor: off
crc32: off
We use the equipment as a backbone, where we run FRR to establish BGP peers, multiple VLANs.
Thank you for your help.
Link Copied
Hello again,
Just because we are such geeks, I started reading all the changelogs for the Intel E810 ICE drivers and found that at one point they had a bug in XDP related to performance in the queues and they fixed it. Could I be facing the same problem?
Fix issues around XDP performance across multiple queues
We are still investigating, now testing on Ubuntu, but also analyzing everything. Hopefully, you can help us.
