need to replicate QAT dpdk perf test on x86

[JCK1]

Hi,

Based on https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf

I am having problems to replicate Intel's test on x86, the best I can get is 7.9gbps on Xeon(R) Gold 6338N CPU @ 2.20GHz from SuperMicro. I tried to follow your instructions of configuring BIOS and kernel settings, etc on your report.

The VFs were assigned with vfio-pci driver not QAT's VF driver.

Anything I could be missing?

Thanks in advance.

JC

[Ronny_G_Intel]

Hi JCK1,

Thanks for reaching out to our Intel communities.

This particular forum is designated to provided assistance for Intel® QuickAssist Technology (Intel® QAT) related issues so I need to ask a couple of initial question to try to understand the situation, please see below:

Are you using Intel® QAT? If yes:

What Intel® QAT hardware are you using?

What Intel® QAT Driver Version are you using?

Can you also provide me with the DPDK version that you are trying?

What Operating System is installed, please include the Kernel version as well?

Note: the general system configuration would be very helpful in cases related to benchmarking or performance testing.

Regards,

Ronny G

[JCK1]

Hi Ronny

Here is the info:

Are you using Intel® QAT? If yes:

What Intel® QAT hardware are you using? Intel Corporation C62x Chipset QuickAssist Technology (rev 04)

What Intel® QAT Driver Version are you using? it comes with linux kernel 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC

Can you also provide me with the DPDK version that you are trying? dpdk-stable-21.11.4

What Operating System is installed, please include the Kernel version as well? Ubuntu 22.04.2 LTS /linux kernel 6.2.0-37-generic

It is SuperMicro server with Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz. I am also looking for a better Xeon machine to try at the moment.

please let me know if you need more info. 

thanks

JC

[Ronny_G_Intel]

Hi JCK1,

Thanks for the info provided.

There are several "Test Case" scenarios in the DPDK Intel Cryptodev Performance Report <https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf>, are you referring to anyone in particular?

I need as much detail as possible to try to replicate the same scenario using a similar system (however this may be very challenging)

Thanks,

Ronny G

[JCK1]

Hi Ronny

Here is more details:

on x86:

• HT is on, 32 Cores 64 HT
• iommu.passthrough=1 default_hugepagesz=1G hugepagesz=1G hugepages=32 disable_denylist=1 intel_iommu=on iommu=pt intel_pstate=disable isolcpus=4-7,12-15 nohz_full=4-7,12-15 rcu_nocbs=4-7,12-15
• 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC
• we run these to configure and bind vfio-pci on QAT VFs:
  • cat config-qat-vf.sh 
    sudo modprobe vfio-pci
    echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
    echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
    echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs
  • cat bind-qat-vf.sh 
    #!/bin/sh
    DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
    for device in 5 6 7; do \
      for fn in $(seq 0 7); do \
        sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
        sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
        echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
      done;
    done

then we run these steps which are copied from Intel's report with some codification:

DPDK_TEST_CRYPTO_PERF=/home/ubuntu/dpdk-stable-21.11.4/build/app/

QAT_PF0=0000:c5:01

QAT_PF1=0000:c6:01

QAT_PF2=0000:c7:01

timestamp() {

    date +"%Y-%m-%d-%H-%M-%S"

}

LOG_FILE=test-crypto-qat-$(timestamp).txt

touch ${LOG_FILE}

echo "======================================================================"

echo "================== Output to ${LOG_FILE} ============================="

echo "======================================================================"

echo "================== QAT perf for AES-CBC-128/SHA1-HMAC ================"

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "================== QAT perf for AES-CBC-128/SHA1-HMAC ================" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

#-l 9,10,66,11,67,12,68 -n 6 \

CORE_OPTION="-l 9,10,60,11,61,12,62 -n 6"

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 \

--cipher-key-sz 16 --devtype crypto_scheduler --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 20 --auth-algo sha1-hmac --cipher-algo aes-cbc --cipher-op encrypt >> ${LOG_FILE} 2>&1

echo "======================================================================"

echo "=============== QAT perf for AES-CBC-128/SHA2-256-HMAC ==============="

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "=============== QAT perf for AES-CBC-128/SHA2-256-HMAC ===============" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 \

--cipher-key-sz 16 --devtype crypto_scheduler --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 32 --auth-algo sha2-256-hmac --cipher-algo aes-cbc --cipher-op encrypt >> ${LOG_FILE} 2>&1

echo "======================================================================"

echo "===================== QAT perf for AES-GCM-128 ======================="

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "===================== QAT perf for AES-GCM-128 =======================" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --aead-key-sz 16 --buffer-sz 64,128,256,512,1024,2048 --optype aead --ptest throughput --aead-aad-sz 16 \

--devtype crypto_scheduler --aead-op encrypt --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 16 --aead-algo aes-gcm --aead-iv-sz 12 >> ${LOG_FILE} 2>&1

and the result like AES-CBC-128/SHA1-HMAC, max 8G was reached which so low comparing to Intel's report:

lcore id   Buf   Size   Burst Size   Enqueued   Dequeued   Failed Enq    Failed Deq   MOps      Gbps     Cycles/Buf

10           64              32              30000000   30000000      83952749    67111313    2.6117     1.3372   842.36
10         128              32              30000000   30000000      83167004    65221859    2.7881     2.8550   789.07
10         256              32              30000000   30000000     108999257   87092827    2.2573     4.6229    974.62
10         512              32              30000000   30000000      247764304  226160272  1.0595     4.3397    2076.47
10       1024              32              30000000   30000000      343982308  321354180   0.8867    7.2636    2481.21
10       2048              32              30000000   30000000      728382708  707317103   0.4819    7.8961    4564.86

Thanks

JC

[JCK1]

Hi, like to see if you have any idea? thanks

[Ronny_G_Intel]

Hi JCK1,

Allow me a few extra days to try to locate an equivalent platform and run the same test.

Note, the platform I will be using wont be the exactly the same so results will be different and I may want to use DPDK latest version available: dpdk-23.11.tar.xz

Regards,

Ronny G

[JCK1]

thanks Ronny. 

[JCK1]

Hello Ronny,

Hope you are doing well in New Year.

Any update?

Actually if you have a working system on that you can produce the same results as claimed in the performance data published, I would like to know that system model so we can try to purchase that system.  Also please provide details about the system setup in BIOS and Linux kernel setting, so we can quickly replicate that. Is that possible?

Thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

Happy new year for you as well! I hope you had great Holidays.

I located the following system that I believe is similar enough to try to replicate this issue, see below:

I also have QAT enabled and running:

And I have DPDK also installed and running:

...but I am having some issues when running the dpdk crypto test https://doc.dpdk.org/guides/tools/cryptoperf.html 

would it be possible that you send me a quick list of the steps that you are doing? That would help me expedite my testing.

Regards,

Ronny G

[JCK1]

Hi Ronny,

Thanks for your quick reply.

I have not enabled qat_service service (I am not aware of that service) but I just used DPDK for testing.

As I described in my very first post, we configure and bind vfio-pci on QAT VFs once only after the system reboot, i.e.  run config-qat-vf.sh and bind-qat-vf.sh:

• cat config-qat-vf.sh 
  sudo modprobe vfio-pci
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs
• cat bind-qat-vf.sh 
  #!/bin/sh
  DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
  for device in 5 6 7; do \
    for fn in $(seq 0 7); do \
      sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
      sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
      echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
    done;
  done

After that, in the DPDK 21.11.4  directory, (assume you have compiled that version of DPDK) and then you go inside the directory dpdk-stable-21.11.4/build/app/ to run the test, again this is described in my very first post.

Now this is my understanding of running your QAT DPDK Perf test as I am reading your PDF report, it could be wrong because obviously I could not reproduce the reported results. So if you could do your own way as you are expert on this subject, then you can just tell me how to run your test in correct way so I can just replicate your steps on my testbed, can you help on that?

Thanks
JCK

[JCK1]

Hi Ronny

Any update on this? Hope to get your guidance on how to reproduce the DPDK QAT performance result as published on x86 system. Anything useful information from your side would be appreciated.

Thanks

JCK

[JCK1]

Hello Ronny

Would you update me about this?

Thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

My apologies for the delay in getting back to you.

What do you get when you run service qat_service status?

Can you provide me with a quick screenshot?

Thanks,

Ronny G

[JCK1]

Hi Ronny

Nice to hear from you!

I could not find qat_service package? do you know how to install it? it does not come when install driver from ubuntu.

Thanks

JCK

[JCK1]

Hello Ronny

With current pace, we are going no where. Would you please help us how to bring up our QAT setup to match Intel's test results?

We have target to hit with our customers who are very interested in deploying QAT on their network.

Thank you

JCK

[Ronny_G_Intel]

Hi JCK1,

I will be checking your case daily in order to help you out with any question you may have, I apologize for any delay in my previous replies.

Checking on your previous notes, I see that you have the following hardware:

Intel® C62x Chipset (formerly Lewisburg) and an Intel® Xeon® Gold 6338N Processor, with this combination and using Ubuntu 22.04.2 LTS, the Intel® QuickAssist Technology Driver for Hardware Version 1.x (known as Customer Enabling Release) is the right driver for you

Here is the URL for the driver mentioned before:

https://www.intel.com/content/www/us/en/download/19734/intel-quickassist-technology-driver-for-linux-hw-version-1-x.html

I would recommend that you deploy the latest driver version available which is 4.24.0-00005

On the other hand, all necessary steps to install and configure the driver are available here:

https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html

Please download the driver, follow the installation guide and let me know if you have questions.

Regards,

Ronny G

[JCK1]

Hi Ronny

Many thanks for your detailed instruction. I followed the Getting Start Guide and got some compiling errors, not sure how to fix that, would you help to see what I am missing?

Thanks again

JCK

[Ronny_G_Intel]

Hi JCK1,

If I am reading your report correctly, you ran the following commands:

# cd QAT

# ./ configure

# make -j install 

You should be getting something like this:

While I do some more investigation, can you please make sure that you have the latest Kernel version? Do you know if it is possible that a previous QAT driver version is present on the system?

Thanks,

Ronny G

[JCK1]

Hi Ronny

We had a QAT kernel driver came with the kernel we installed:

root@xeon-sm-01:/QAT# lsmod | grep c6
qat_c62x 12288 0
intel_qat 274432 1 qat_c62x
root@xeon-sm-01:/QAT# uname -a
Linux xeon-sm-01 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

So what is the latest kernel you support? 6.5.0 is pretty new version of kernel.

I unloaded the qat_c62x and intel_qat from the system, and did this:

./configure --enable-icp-dc-sym-only

make -j install

the errors seem to be persistent:

Building qat_direct for user space
make[2]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src'
test -d build || mkdir -p build;
make[3]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/uio'
make[3]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/common'
make[5]: *** No rule to make target 'vmlinux.o', needed by 'modpost'. Stop.
make[5]: *** Waiting for unfinished jobs....
warning: the compiler differs from the one used to build the kernel
The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
make[4]: *** [Makefile:234: __sub-make] Error 2
make[4]: Leaving directory '/usr/src/linux-headers-6.5.0-15-generic'
Copying outputs
mv: cannot stat 'OsalServices.o': No such file or directory
mv: cannot stat 'OsalSemaphore.o': No such file or directory
mv: cannot stat 'OsalSpinLock.o': No such file or directory
mv: cannot stat 'OsalThread.o': No such file or directory
mv: cannot stat 'OsalAtomic.o': No such file or directory
mv: cannot stat 'OsalMemMgmt.o': No such file or directory
mv: cannot stat 'OsalIOMMUMgmt.o': No such file or directory
mv: cannot stat 'OsalUsrKrlProxy.o': No such file or directory
mv: cannot stat 'OsalCryptoInterface.o': No such file or directory
make[3]: Leaving directory '/QAT/quickassist/utilities/osal/src/linux/kernel_space'
make[2]: Leaving directory '/QAT/quickassist/utilities/osal/src'

Copying OSAL library
cp /QAT/quickassist/utilities/osal//src/linux/kernel_space/build/linux_2.6/kernel_space/libosal.a /QAT/build/libosal_kernel.a;
cp: cannot stat '/QAT/quickassist/utilities/osal//src/linux/kernel_space/build/linux_2.6/kernel_space/libosal.a': No such file or directory
make[1]: *** [Makefile:152: libosal_kernel] Error 1
make[1]: *** Waiting for unfinished jobs....
Creating output directory
Creating output directory
AR /QAT/quickassist/utilities/libusdm_drv/linux/lib.a
ar: creating libadf_uio.a
make[3]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/uio'
ar: creating libadf_common.a
make[3]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/common'
Creating static library libadf_user.a
ar rcs libadf_user.a build/*.o
make[2]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src'

Copying qat_direct library
cp /QAT/quickassist/lookaside/access_layer/src/qat_direct//src/libadf_user.a /QAT/build/libadf.a;
LD [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.o
MODPOST /QAT/quickassist/utilities/libusdm_drv/linux/Module.symvers
CC [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.mod.o
LD [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko
BTF [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko
Skipping BTF generation for /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko due to unavailability of vmlinux
make[4]: Leaving directory '/usr/src/linux-headers-6.5.0-15-generic'
Copying outputs
make[3]: Leaving directory '/QAT/quickassist/utilities/libusdm_drv/linux'
make[2]: Leaving directory '/QAT/quickassist/utilities/libusdm_drv'
make[1]: Leaving directory '/QAT/quickassist'
make: *** [Makefile:1187: quickassist-all] Error 2

Not sure what is error above?

Thanks

JCK

[Ronny_G_Intel]

Hi JCK1,

Did you initialize the driver?

 

Notice that you are using Intel® C62x Chipset which requires: # ./configure --enable-icp-dc-sym-only 

Are you using any other installation flag?

Thanks,

Ronny G