Increase the number of RSS queues on the E810 above 64?

[0x4A6F686E]

Hi there,

I have an Intel E810-CQDA2 adapter on a Debian Linux system with ice-driver version 1.16.3.

The Intel CPU on that box has 128 cores in total, so I would like to increase the number of RSS queues to around 120 so I can use a separate queue with each CPU-core.
Ethtool allows me to do that, but set_irq_affinity only uses 64 cores/interrupts.

I found the following in the man-page with the driver sources:

Transmit/Receive Queue Allocation
---------------------------------

The driver allocates a number of transmit/receive queue pairs equal to
the number of local node CPU threads with the following constraints:

* The driver will allocate a minimum of 8 queue pairs, or the total
number of CPUs, whichever is lower.

* The driver will allocate a maximum of 64 queue pairs, or 256 for the
iavf driver.

It clearly states the maximum is 64.
Is there any way to go higher than 64?

Thanks in advance for your response.

John

[Fikri_Intel]

Hi 0x4A6F686E,

Thank you for contacting Intel.

We appreciate you reaching out to us regarding the reported issue.  We understand that you're aiming to scale the Receive Side Scaling (RSS) queues to 120 in order to align one queue per CPU core. We appreciate your effort in optimizing performance at the system level.

At this point, we are checking internally to confirm if there are any supported methods to increase the RSS queue limit beyond the current 64-queue cap.

In the meantime, to assist us with a more detailed analysis, kindly provide the following information:

1- The output of the command:

ethtool -l <interface>

lscpu

2- The latest SSU (System Support Utility) log:

https://www.intel.com/content/www/us/en/support/articles/000008563/ethernet-products.html

These details will allow us to better assess the configuration and explore possible options for scaling.

We look forward to your response.

Regards,

Fikri O.

[0x4A6F686E]

Hello Fikri,

Thank you for looking into this. The results of the requested output:

root@build-sl03:/# ethtool -l enp17s0f0
Channel parameters for enp17s0f0:
Pre-set maximums:
RX: 256
TX: 256
Other: 1
Combined: 256
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 120

And...

root@build-sl03:/#  lscpu

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 256
On-line CPU(s) list: 0-255
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel(R) Corporation
Model name: INTEL(R) XEON(R) PLATINUM 8592+
BIOS Model name: INTEL(R) XEON(R) PLATINUM 8592+ CPU @ 1.9GHz
BIOS CPU family: 179
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 64
Socket(s): 2
Stepping: 2
CPU(s) scaling MHz: 21%
CPU max MHz: 3900.0000
CPU min MHz: 800.0000
BogoMIPS: 3800.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs
bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movb
e popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsba
se tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1
xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg
avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig a
rch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Caches (sum of all):
L1d: 6 MiB (128 instances)
L1i: 4 MiB (128 instances)
L2: 256 MiB (128 instances)
L3: 640 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0-63,128-191
NUMA node1 CPU(s): 64-127,192-255
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Srbds: Not affected
Tsx async abort: Not affected

When we look at the /proc/interrupts we see the following:

root@build-sl03:/# grep ice-enp17s0f0-TxRx /proc/interrupts | awk '{print $NF}'|sed -E 's/[-0-9]+$//' |sort |uniq -c
64 ice-enp17s0f0-TxRx

Please let me know if you need anything else.

Regards,

John

[0x4A6F686E]

Hello Fikri,

Thank you for looking into our issue. I've collected the requested information and attached it.

Please let me know if you need more information.

Regards,

John

[Fikri_Intel]

Hi 0x4A6F686E,

Thank you for your response.

Kindly allow us some time to further check and get back to you once the info is available.

We truly appreciate your cooperation.

Regards,

Fikri O.

[Sachinks]

Hello 0x4A6F686E,

Greetings!

We have checked this further with our team and unfortunately, it's not possible to increase the maximum queue pairs allocated by the driver.

The maximum limit is what we have mentioned in the driver "read me" file which you had stated in the email.

The default number of queues allocated by the driver during initialization was changed to a maximum of 64 queues per port, or up to the maximum number of cores on the local NUMA node for the device (if local NUMA has fewer than 64 cores).

Regards,

Sachin KS

Intel Customer Support Technician

[0x4A6F686E]

Dear Sachin, Dear Fikri,

Thank you for confirming this limit.

Regards,

John

[Azeem_Intel]

Hello 0x4A6F686E,

Thank you for contacting the Intel Community.

We would like to kindly check if you require any further assistance regarding your inquiry. If everything is resolved, please let us know if we may proceed to close the case.

We appreciate your time and cooperation.

Regards,

Azeem

Intel Customer Support Technician

[Fikri_Intel]

Hi 0x4A6F686E,

Good day to you.

Since we have not seen an update, we will be closing this forum case from our end. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards,

Fikri O.