need to run dpdk-crypto-perf-test on Intel(R) Xeon(R) Platinum 8480+

[JCK1]

Hi Intel expert

I have Intel(R) Xeon(R) Platinum 8480+ based HW, like to run  dpdk-crypto-perf-test , since the QAT is built into the processor, no longer a PCIe device. How to specify in  dpdk-crypto-perf-test with options like -a, --devicetype etc? the -a is used to specify the VF for the QAT card, what is right option for the built-in QAT engine in 8480 processor.

Thanks in advance

JCK

[Ronny_G_Intel · Accepted solution]

JCK1,

For the built-in QAT processor it still requires specifying the correct PCIe address. The syntax hasn’t changed. It uses bus:slot:function

Note as before you will need to bind the QAT VFs to the vfio-pci driver before they can be used with DPDK.

Thanks,

Ronny G

[JCK1]

Ronny

Thanks for the confirmation.

I attached the issue I am facing here (your forum does not allow me to post directly so I put them into a text file.

Do you know what the problem I am getting?

Thanks
JCK

[Ronny_G_Intel]

Hi JCK,

The dmesg output indicates that two QAT devices (qat_dev0 and qat_dev1) have successfully started with 9 acceleration engines each. This suggests that the QAT hardware is recognized by the system and the QAT driver is loaded.

The lsmod command shows that the qat_4xxx and intel_qat modules are loaded, but there are no services currently using these modules (0 usage count). This is expected before the DPDK application binds to the devices.

Driver Bindings (ll /sys/bus/pci/drivers/4xxx), QAT PCI driver shows two devices bound to the driver, which aligns with the dmesg output.

After binding the QAT devices to the DPDK-compatible vfio-pci driver, the dpdk-devbind tool lists the devices as available for DPDK applications. However, they are marked as unused, indicating that no DPDK application is currently using them.

The output from the dpdk-test-crypto-perf application shows that the DPDK Environment Abstraction Layer (EAL) is able to detect the QAT devices and attempts to bind to them. However, it fails with the message qat_pf2vf_exch_msg(): ACK not received from remote, followed by qat_pci_device_allocate(): Cannot acquire ring configuration for QAT_0. This indicates a communication issue between the Physical Function (PF) and the Virtual Function (VF) of the QAT device.

I would assume intel_iommu is enabled because VFS are visible.

Do you have any error when running DPDK compilation and installation?

$ meson setup <options> build

$ ninja

$ meson install

How are you binding the VF to DPDK?

$ ./dpdk-devbind.py -b vfio-pci 0000:B:S:F

Thanks,

Ronny G

[JCK1]

Hi Ronny

Thanks for your reply

>>> I would assume intel_iommu is enabled because VFS are visible.

Yes, here is the kernel setting:
intel_pstate=disable console=ttyS0,115200n8 kpti=off default_hugepagesz=1G hugepagesz=1G hugepages=64 isolcpus=1-69 nohz_full=1-69 irqaffinity=0 rcu_nocbs=1-69 rcu_nocb_poll nosoftlockup intel_iommu=on iommu.passthrough=1 transparent_hugepage=always

>>> How are you binding the VF to DPDK?

I used this to bind:

DEVBIND_PATH=/home/ubuntu/dpdk/usertools/dpdk-devbind.py

sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.0
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.1
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.2
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.3
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.4
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.5
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.6
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:00.7
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.0
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.1
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.2
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.3
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.4
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.5
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.6
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:01.7
sudo ${DEVBIND_PATH} --bind=vfio-pci 6b:02.0

There are two PFs of QAT:
Crypto devices using kernel driver
==================================
0000:6b:00.0 'Device 4940' drv=4xxx unused=qat_4xxx,vfio-pci
0000:e8:00.0 'Device 4940' drv=4xxx unused=qat_4xxx,vfio-pci

Should we do anything about it?

As for the dpdk, I did not see any error in building it.

Thanks

JCK

[JCK1]

Hi Ronny

I attached my new message in attached as your message system does not allow me post it directly

thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

I am glad to hear that you were to resolve the original DPDK binding issue.

I checked your new report and I believe that in order to get a better understanding of the system configurations, how specific parameters are set, and what QAT package version you are running, we would need the icp_dump?

This script is located in the following location of the QAT SDK package: ICP_ROOT/quickassist/utilities/release-files/debug_tool/icp_dump.sh

Can you please share this information?

We look forward to hearing back from you.

Regards,

Ronny G

[JCK1]

Hi Ronny

Thanks for your reply.

I was able to run the same test as AES-CBC/SHA2-256-HMAC and AES-GCM. However AES-CBC/SHA1-HMAC gave me an error:

# Crypto Performance Application Options:
#
# cperf test: throughput
#
# cperf operation type: cipher-then-auth
#
# size of crypto op / mbuf pool: 8192
# total number of ops: 30000000
# buffer sizes: 64 128 256 512 1024 2048
# burst sizes: 32

# segment size: 2068
#
# cryptodev type: crypto_qat
#
# number of queue pairs per device: 2
# crypto operation: cipher-then-auth
# sessionless: no
# out of place: no
#
# auth algorithm: sha1-hmac
# auth operation: generate
# auth key size: 64
# auth iv size: 0
# auth digest size: 20
#
# cipher algorithm: aes-cbc
# cipher operation: encrypt
# cipher key size: 16
# cipher iv size: 16
#
USER1: Crypto device type does not support capabilities requested
USER1: Unsupported case: errno: 2

I attached the output from icp_dump.

thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

I have been trying to replicate this error message that you are getting but I haven't had any luck, I believe this is mostly on the DPDK side but it could be a combination of QAT and DPDK.

Can you tell me what are you trying to accomplish and if you have tried QAT only and have issues?

Can you also provide me with a screenshot when you run AES-CBC/SHA2-256-HMAC and AES-GCM showing the error message for AES-CBC/SHA2-256-HMAC? I will need to bring this up to the DPDK team and will need as much detail as possible.

Thanks,

Ronny G

[JCK1]

Hi Ronny

I was trying to test same crypto test we have done with QAT PCIe card on the Intel SoC QAT engin.

this is command I have done (same on PCIe QAT):

sudo /home/ubuntu/dpdk/build/app/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem -a 6b:00.1 -a 6b:00.2 -a 6b:00.3 -a 6b:00.4 -a 6b:00.5 -a 6b:00.6 -a 6b:00.7 -a 6b:01.0 -a 6b:01.1 -a 6b:01.2 -l 2,4,6,8,10,12,14,16,18,20,22 -n 10 -- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 --cipher-key-sz 16 --devtype crypto_qat --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 30000000 --digest-sz 20 --auth-algo sha1-hmac --cipher-algo aes-cbc --cipher-op encrypt
EAL: Detected CPU lcores: 24
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
CRYPTODEV: Creating cryptodev 0000:6b:00.1_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.1_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.1_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.1_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.2_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.2_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.2_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.2_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.3_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.3_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.3_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.3_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.4_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.4_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.4_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.4_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.5_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.5_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.5_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.5_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.6_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.6_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.6_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.6_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.7_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.7_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:00.7_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:00.7_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.0_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.0_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.0_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.0_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.1_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.1_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.1_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.1_qat_sym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.2_qat_asym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.2_qat_asym,socket id: 0, max queue pairs: 0
CRYPTODEV: Creating cryptodev 0000:6b:01.2_qat_sym
CRYPTODEV: Initialisation parameters - name: 0000:6b:01.2_qat_sym,socket id: 0, max queue pairs: 0
Allocated pool "sess_mp_0" on socket 0
# Crypto Performance Application Options:
#
# cperf test: throughput
#
# cperf operation type: cipher-then-auth
#
# size of crypto op / mbuf pool: 8192
# total number of ops: 30000000
# buffer sizes: 64 128 256 512 1024 2048
# burst sizes: 32

# segment size: 2068
#
# cryptodev type: crypto_qat
#
# number of queue pairs per device: 1
# crypto operation: cipher-then-auth
# sessionless: no
# out of place: no
#
# auth algorithm: sha1-hmac
# auth operation: generate
# auth key size: 64
# auth iv size: 0
# auth digest size: 20
#
# cipher algorithm: aes-cbc
# cipher operation: encrypt
# cipher key size: 16
# cipher iv size: 16
#
USER1: Crypto device type does not support capabilities requested
USER1: Unsupported case: errno: 2

If I just changed --optype cipher-then-auth in above command to --optype cipher-only, then it worked. this is AES-CBC with SHA1-HMAC. However we do not have issue with AES-CBC/SHA2-256-HMAC.

thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

I see that you have the following kernel and QAT device:

Linux 2p8480-501 6.2.0-36-generic #37~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 9 15:34:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Number of 4xxxvf devices=32

Number of 4xxx devices=2

Can you please tell me the QAT driver version that you are using?

Thanks,

Ronny G

[JCK1]

Hi Ronny

$ lsmod | grep qat
qat_4xxx 20480 0
intel_qat 270336 1 qat_4xxx
crc8 16384 1 intel_qat
authenc 16384 1 intel_qat
$ modinfo qat_4xxx
filename: /lib/modules/6.2.0-36-generic/kernel/drivers/crypto/qat/qat_4xxx/qat_4xxx.ko
softdep: pre: crypto-intel_qat
version: 0.6.0
description: Intel(R) QuickAssist Technology
firmware: qat_4xxx_mmp.bin
firmware: qat_4xxx.bin
author: Intel
license: Dual BSD/GPL
srcversion: 0AA4DF0E993F750524173E2
alias: pci:v00008086d00004942sv*sd*bc*sc*i*
alias: pci:v00008086d00004940sv*sd*bc*sc*i*
depends: intel_qat
retpoline: Y
intree: Y
name: qat_4xxx

thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

I am still trying to replicate this issue.

To confirm, are you currently trying to test against a QAT PF, or a QAT VF?

If it is a set of QAT VFs, can you also share the configuration file for the QAT PF?

Thanks,

Ronny G

[Ronny_G_Intel]

Hi JCK1,

I need to confirm if you are currently trying to test against a QAT PF, or a QAT VF?

If it is a set of QAT VFs, can you also share the configuration file for the QAT PF?

Thanks,

Ronny G

[Ronny_G_Intel]

Hi JCK1,

I hope all is well with you.

Do you have any update by any chance? 

Regards,

Ronny G