X(L)710 supports symmetric RSS hashing. See "7.1.9.3 Symmetric hash" in the "Intel® Ethernet Controller XL710 Datasheet". I want to known how to enable this feature on Linux (CentOs).

[FJime10]

I have configured 5 RSS queues and I have several IDS processes running. Each one assigned to an RSS queue. As a result I can see that the inbound packets are not balanced to the same RSS queue as the back packets and because of this my IDS detects bad connections.

[Caguicla_Intel]

Hello FJime10,

Thank you for posting in Intel Ethernet Communities. 

Please allow us to check on your query. We will give you an update as soon as there is any but no later than 3 business days.

Hoping for you kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

Good day!

We'd like to check if you have tried the commands below? Symmetric RSS hashing functionality was added in the 1.5.18 driver, so it should have been carried forward to current i40e driver 2.11.25.

You may download the latest i40e driver from the link below.

https://sourceforge.net/projects/e1000/files/i40e%20stable/

ethtool 'priv flag' for the i40e driver called 'rss-symmetric'.

When set to 'on' the RSS hashing will be symmetric.

To enable:

ethtool --set-priv-flags $DEV rss-symmetric on

To disable:

ethtool --set-priv-flags $DEV rss-symmetric off

To see the current setting:

# ethtool --show-priv-flags $DEV

We look forward to your response.

In case we don't hear from you, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

When I follow the procedure that you indicate I get the following error:

# /usr/local/sbin/ethtool --set-priv-flags p1p1 rss-symmetric on

ethtool: bad command line argument(s)

For more information run ethtool -h

#

I have tried with version 4.8 and version 5.4 of ethtool and with version 2.4.6 of the i46e driver but I always get the same error.

# /usr/local/sbin/ethtool -i p1p1

driver: i40e

version: 2.11.25

firmware-version: 7.10 0x800051a4 19.0.12

expansion-rom-version:

bus-info: 0000:03:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

Thanks

[Caguicla_Intel]

Hello FJime10,

Thank you for the update.

Please let us further check on this matter. We will give you an update within 1-3 business days.

Hoping for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

We sincerely apologize for the delay on this matter.

Sharing the DPDK option to configure symmetric RSS hashing, please refer to the details below.

DPDK:

With DPDK, you can use testpmd to configure symmetric RSS hashing

testpmd command: set_hash_global_config 0 toeplitz ipv4-other enable

         set_sym_hash_ena_per_port 0 enable

 For Simple symmetric, the same as the above test case “test_toeplitz_symmetric”. Just pay attention to set the hash function to “simple xor” you can refer to section 67.3 and 67.5 in this link for more details: https://doc.dpdk.org/dts/test_plans/pmdrss_hash_test_plan.html?highlight=rss#test-case-test-toeplitz-symmetric

In case you still wanted to go with the ethtool method, please let us know for us to further investigate the issue

We look forward to hear from you. 

If we don't receive a response, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

The use of Dpdk development suite is not applicable since we use software that is not developed to make use of dpdk. ​ We have activated IOAT dma and the performance has improved although we do not know how to activate DCA (direct cache access) either. we need to activate the symmetric RRS functionality of the x710 card since each process parses an RSS queue and therefore each queue must have the round trip packets of the same Network flow. If it can be activated with ethtool or any other tool or I40e driver parameter would be enough. Thanks El mar., 21 abr. 2020 3:09, Intel Forums <supportreplies@intel.com> escribió:

[Caguicla_Intel]

Hello FJime10,

Thank you for the reply.

Please allow us to further investigate the issue you have encountered with the ethtool while configuring symmetric RSS hashing. We will give you an update as soon as possible but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

Good day!

Please be informed that we are still actively working on this issue for you and we are involving higher level Engineers to sort this out as soon as possible.

Thank you for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

We sincerely apologize for the delay on this matter as are still working on this issue. Rest assured that we will give you an update as soon as there is any findings but no later than 2-4 business days.

Your patience is highly appreciated!

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

Good day!

Unfortunately, we have been informed that the only support to setup symmetric RSS is via DPDK.

DPDK:

With DPDK, you can use testpmd to configure symmetric RSS hashing

testpmd command: set_hash_global_config 0 toeplitz ipv4-other enable

                set_sym_hash_ena_per_port 0 enable

For Simple symmetric, the same as the above test case “test_toeplitz_symmetric”. Just pay attention to set the hash function to “simple xor” you can refer to section 67.3 and 67.5 in this link for more details: https://doc.dpdk.org/dts/test_plans/pmdrss_hash_test_plan.html?highlight=rss#test-case-test-toeplitz-symmetric

Let us know if you have additional questions and clarifications.

We hope to hear from you.

Should there be no response, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

Hello,

Is testpdm a linux shell command with which I can enable symmetric RSS for all running processes?

Please note that we are not developing software, we simply need the network device use symmetric RSS when distributing network packets between RSS queues. Each queue must have the complete flow of packets of the same communication.

Finally, I would like to know how I can verify if the network device is using DMA, DCA, IOATDMA since we are experiencing high CPU consumption (> = 80%) over all cores (12 cores) with only 1 Gbit/s on p1p1 device and 500 Mbit/s on p1p2 device.

My current kernel and modules configuration is:

#cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

# lsmod|grep -i dma

ioatdma               67809 0

dca                   15130 1 ioatdma

idma64                17483 0

virt_dma              13326 1 idma64

# cat /proc/dma

  4: cascade

# cat /sys/class/dma/dma0chan0/in_use

0

# cat /sys/class/dma/dma0chan1/in_use

0

# ethtool -i p1p1

driver: i40e

version: 2.11.25

firmware-version: 7.10 0x800075df 19.5.12

expansion-rom-version:

bus-info: 0000:03:00.0

....

# ethtool -i p1p2

driver: i40e

version: 2.11.25

firmware-version: 7.10 0x800075df 19.5.12

expansion-rom-version:

bus-info: 0000:03:00.1

...

# dmesg|grep -i mmu

[   0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

[   0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

[   0.000000] DMAR: IOMMU enabled

[   0.057933] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0

[   0.887549] iommu: Adding device 0000:03:00.0 to group 7

[   0.887558] iommu: Adding device 0000:03:00.1 to group 7

# dmesg|grep -i dma

[   0.000000] ACPI: DMAR 000000007dfd2000 00070 (v01 DELL  PE_SC3  00000002 DELL 00000001)

[   0.000000]  DMA     [mem 0x00001000-0x00ffffff]

[   0.000000]  DMA32   [mem 0x01000000-0xffffffff]

[   0.000000]  DMA zone: 64 pages used for memmap

[   0.000000]  DMA zone: 1050 pages reserved

[   0.000000]  DMA zone: 3999 pages, LIFO batch:0

[   0.000000]  DMA32 zone: 7565 pages used for memmap

[   0.000000]  DMA32 zone: 484104 pages, LIFO batch:31

[   0.000000] DMAR: IOMMU enabled

[   0.057924] DMAR: Host address width 39

[   0.057926] DMAR: DRHD base: 0x000000fed91000 flags: 0x1

[   0.057931] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da

[   0.057932] DMAR: RMRR base: 0x000000796c9000 end: 0x000000796e8fff

[   0.057933] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0

[   0.057934] DMAR-IR: HPET id 0 under DRHD base 0xfed91000

[   0.057935] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[   0.061039] DMAR-IR: Enabled IRQ remapping in x2apic mode

[   0.887182] DMAR: No ATSR found

[   0.887223] DMAR: dmar0: Using Queued invalidation

[   0.887290] DMAR: Hardware identity mapping for device 0000:03:00.0

[   0.887291] DMAR: Hardware identity mapping for device 0000:03:00.1

[   0.887297] DMAR: Setting RMRR:

[   0.887298] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x796c9000 - 0x796e8fff]

[   0.887300] DMAR: Prepare 0-16MiB unity mapping for LPC

[   0.887301] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]

[   0.887303] DMAR: Intel(R) Virtualization Technology for Directed I/O

[   2.608274] megaraid_sas 0000:02:00.0: 32 bit DMA mask and 32 bit consistent mask

[   2.608362] DMAR: 32bit 0000:02:00.0 uses non-identity mapping

[   2.622205] tg3 0000:06:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]

[   2.633395] tg3 0000:06:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]

[   2.658584] ata1: SATA max UDMA/133 abar m2048@0x94c1c000 port 0x94c1c100 irq 139

[   2.662865] [TTM] Zone  dma32: Available graphics memory: 2097152 kiB

[   2.662873] [TTM] Initializing DMA pool allocator

[   4.313443] idma64 idma64.0: Found Intel integrated DMA 64-bit

[ 103.056680] ioatdma: Intel(R) QuickData Technology Driver 4.00

# dmesg|grep -i dca

[   2.329093] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[ 103.051282] dca service started, version 1.12.1

Thanks

[Caguicla_Intel]

Hello FJime10,

How are you today?

Please let us know if there is any other assistance needed on this matter so we can further help you.

Awaiting to hear from you.

Should there be no response, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

Hello,

Is testpdm a linux shell command with which I can enable symmetric RSS for all running processes?

Please note that we are not developing software, we simply need the network device use symmetric RSS when distributing network packets between RSS queues. Each queue must have the complete flow of packets of the same communication.

Finally, I would like to know how I can verify if the network device is using DMA, DCA, IOATDMA since we are experiencing high CPU consumption (> = 80%) over all cores (12 cores) with only 1 Gbit/s on p1p1 device and 500 Mbit/s on p1p2 device.

My current kernel and modules configuration is:

#cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

# lsmod|grep -i dma

ioatdma               67809 0

dca                   15130 1 ioatdma

idma64                17483 0

virt_dma              13326 1 idma64

# cat /proc/dma

 4: cascade

# cat /sys/class/dma/dma0chan0/in_use

0

# cat /sys/class/dma/dma0chan1/in_use

0

# ethtool -i p1p1

driver: i40e

version: 2.11.25

firmware-version: 7.10 0x800075df 19.5.12

expansion-rom-version:

bus-info: 0000:03:00.0

....

# ethtool -i p1p2

driver: i40e

version: 2.11.25

firmware-version: 7.10 0x800075df 19.5.12

expansion-rom-version:

bus-info: 0000:03:00.1

...

# dmesg|grep -i mmu

[   0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

[   0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on iommu=pt crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

[   0.000000] DMAR: IOMMU enabled

[   0.057933] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0

[   0.887549] iommu: Adding device 0000:03:00.0 to group 7

[   0.887558] iommu: Adding device 0000:03:00.1 to group 7

# dmesg|grep -i dma

[   0.000000] ACPI: DMAR 000000007dfd2000 00070 (v01 DELL  PE_SC3  00000002 DELL 00000001)

[   0.000000]  DMA     [mem 0x00001000-0x00ffffff]

[   0.000000]  DMA32   [mem 0x01000000-0xffffffff]

[   0.000000]  DMA zone: 64 pages used for memmap

[   0.000000]  DMA zone: 1050 pages reserved

[   0.000000]  DMA zone: 3999 pages, LIFO batch:0

[   0.000000]  DMA32 zone: 7565 pages used for memmap

[   0.000000]  DMA32 zone: 484104 pages, LIFO batch:31

[   0.000000] DMAR: IOMMU enabled

[   0.057924] DMAR: Host address width 39

[   0.057926] DMAR: DRHD base: 0x000000fed91000 flags: 0x1

[   0.057931] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da

[   0.057932] DMAR: RMRR base: 0x000000796c9000 end: 0x000000796e8fff

[   0.057933] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0

[   0.057934] DMAR-IR: HPET id 0 under DRHD base 0xfed91000

[   0.057935] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[   0.061039] DMAR-IR: Enabled IRQ remapping in x2apic mode

[   0.887182] DMAR: No ATSR found

[   0.887223] DMAR: dmar0: Using Queued invalidation

[   0.887290] DMAR: Hardware identity mapping for device 0000:03:00.0

[   0.887291] DMAR: Hardware identity mapping for device 0000:03:00.1

[   0.887297] DMAR: Setting RMRR:

[   0.887298] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x796c9000 - 0x796e8fff]

[   0.887300] DMAR: Prepare 0-16MiB unity mapping for LPC

[   0.887301] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]

[   0.887303] DMAR: Intel(R) Virtualization Technology for Directed I/O

[   2.608274] megaraid_sas 0000:02:00.0: 32 bit DMA mask and 32 bit consistent mask

[   2.608362] DMAR: 32bit 0000:02:00.0 uses non-identity mapping

[   2.622205] tg3 0000:06:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]

[   2.633395] tg3 0000:06:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]

[   2.658584] ata1: SATA max UDMA/133 abar m2048@0x94c1c000 port 0x94c1c100 irq 139

[   2.662865] [TTM] Zone  dma32: Available graphics memory: 2097152 kiB

[   2.662873] [TTM] Initializing DMA pool allocator

[   4.313443] idma64 idma64.0: Found Intel integrated DMA 64-bit

[ 103.056680] ioatdma: Intel(R) QuickData Technology Driver 4.00

# dmesg|grep -i dca

[   2.329093] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[ 103.051282] dca service started, version 1.12.1

Thanks

[Caguicla_Intel]

Hello FJime10,

Thank you for the reply.

Please allow us to further check on your additional questions. We will give you an update as soon as there is any findings but no later than 3 business days.

Hoping for your patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

Good day!

We've sent a private message as we are requesting for some personal information. Kindly check your inbox.

We look forward to your reply.

Should there be no response, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

Please verify that you have received the information you have requested.

Thanks

[Caguicla_Intel]

Hello FJime10,

Good day! 

Thank you for providing the requested information. We will continue to check on this and get back to you as soon as there is any update but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello FJime10,

Greetings from Intel Customer Support!

We've sent a personal message, please check your inbox.

Waiting to your response.

In case we don't hear from you, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[FJime10]

I´ve sent a personal message, please check your inbox.

[Caguicla_Intel]

Hello FJime10,

Appreciate your prompt reply.

We will make sure keep you posted on the duplicate inquiry and then proceed closing this request once we have forwarded your additional question on the other request.

Thank you very much for your time and cooperation throughout the process. Should you have any other concerns or assistance needed in the future, feel free to post a new question.

May you have a great day!

Best regards,

Crisselle C

Intel® Customer Support