topic Re: need to replicate QAT dpdk perf test on x86 in Intel® QuickAssist Technology (Intel® QAT)

need to replicate QAT dpdk perf test on x86

JCK1 — Mon, 18 Dec 2023 13:17:20 GMT

Hi,

Based on https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf

I am having problems to replicate Intel's test on x86, the best I can get is 7.9gbps on Xeon(R) Gold 6338N CPU @ 2.20GHz from SuperMicro. I tried to follow your instructions of configuring BIOS and kernel settings, etc on your report.

The VFs were assigned with vfio-pci driver not QAT's VF driver.

Anything I could be missing?

Thanks in advance.

Re:need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Tue, 19 Dec 2023 16:03:50 GMT

Hi JCK1,

Thanks for reaching out to our Intel communities.

This particular forum is designated to provided assistance for Intel® QuickAssist Technology (Intel® QAT) related issues so I need to ask a couple of initial question to try to understand the situation, please see below:

Are you using Intel® QAT? If yes:

What Intel® QAT hardware are you using?

What Intel® QAT Driver Version are you using?

Can you also provide me with the DPDK version that you are trying?

What Operating System is installed, please include the Kernel version as well?

Note: the general system configuration would be very helpful in cases related to benchmarking or performance testing.

Regards,

Ronny G

Re: Re:need to replicate QAT dpdk perf test on x86

JCK1 — Tue, 19 Dec 2023 16:32:50 GMT

Hi Ronny

Here is the info:

Are you using Intel® QAT? If yes:

What Intel® QAT hardware are you using? Intel Corporation C62x Chipset QuickAssist Technology (rev 04)

What Intel® QAT Driver Version are you using? it comes with linux kernel 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC

Can you also provide me with the DPDK version that you are trying? dpdk-stable-21.11.4

What Operating System is installed, please include the Kernel version as well? Ubuntu 22.04.2 LTS /linux kernel 6.2.0-37-generic

It is SuperMicro server with Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz. I am also looking for a better Xeon machine to try at the moment.

please let me know if you need more info.

thanks

Re:need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Wed, 20 Dec 2023 20:04:40 GMT

Hi JCK1,

Thanks for the info provided.

There are several "Test Case" scenarios in the DPDK Intel Cryptodev Performance Report <https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf>, are you referring to anyone in particular?

I need as much detail as possible to try to replicate the same scenario using a similar system (however this may be very challenging)

Thanks,

Ronny G

Re: Re:need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 20 Dec 2023 20:18:54 GMT

Hi Ronny

Here is more details:

on x86:

HT is on, 32 Cores 64 HT
iommu.passthrough=1 default_hugepagesz=1G hugepagesz=1G hugepages=32 disable_denylist=1 intel_iommu=on iommu=pt intel_pstate=disable isolcpus=4-7,12-15 nohz_full=4-7,12-15 rcu_nocbs=4-7,12-15
6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC
we run these to configure and bind vfio-pci on QAT VFs:
- cat config-qat-vf.sh
  sudo modprobe vfio-pci
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
  echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs
- cat bind-qat-vf.sh
  #!/bin/sh
  DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
  for device in 5 6 7; do \
  for fn in $(seq 0 7); do \
  sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
  sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
  echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
  done;
  done

then we run these steps which are copied from Intel's report with some codification:

DPDK_TEST_CRYPTO_PERF=/home/ubuntu/dpdk-stable-21.11.4/build/app/

QAT_PF0=0000:c5:01

QAT_PF1=0000:c6:01

QAT_PF2=0000:c7:01

timestamp() {

date +"%Y-%m-%d-%H-%M-%S"

}

LOG_FILE=test-crypto-qat-$(timestamp).txt

touch ${LOG_FILE}

echo "======================================================================"

echo "================== Output to ${LOG_FILE} ============================="

echo "======================================================================"

echo "================== QAT perf for AES-CBC-128/SHA1-HMAC ================"

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "================== QAT perf for AES-CBC-128/SHA1-HMAC ================" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

#-l 9,10,66,11,67,12,68 -n 6 \

CORE_OPTION="-l 9,10,60,11,61,12,62 -n 6"

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 \

--cipher-key-sz 16 --devtype crypto_scheduler --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 20 --auth-algo sha1-hmac --cipher-algo aes-cbc --cipher-op encrypt >> ${LOG_FILE} 2>&1

echo "======================================================================"

echo "=============== QAT perf for AES-CBC-128/SHA2-256-HMAC ==============="

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "=============== QAT perf for AES-CBC-128/SHA2-256-HMAC ===============" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 \

--cipher-key-sz 16 --devtype crypto_scheduler --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 32 --auth-algo sha2-256-hmac --cipher-algo aes-cbc --cipher-op encrypt >> ${LOG_FILE} 2>&1

echo "======================================================================"

echo "===================== QAT perf for AES-GCM-128 ======================="

echo "======================================================================"

echo ""

echo "======================================================================" >> ${LOG_FILE}

echo "===================== QAT perf for AES-GCM-128 =======================" >> ${LOG_FILE}

echo "======================================================================" >> ${LOG_FILE}

echo "" >> ${LOG_FILE}

sudo ${DPDK_TEST_CRYPTO_PERF}/dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem \

-a ${QAT_PF0}.0 -a ${QAT_PF1}.0 -a ${QAT_PF2}.0 \

-a ${QAT_PF0}.1 -a ${QAT_PF1}.1 -a ${QAT_PF2}.1 \

-a ${QAT_PF0}.2 -a ${QAT_PF1}.2 -a ${QAT_PF2}.2 \

-a ${QAT_PF0}.3 -a ${QAT_PF1}.3 -a ${QAT_PF2}.3 \

-a ${QAT_PF0}.4 -a ${QAT_PF1}.4 -a ${QAT_PF2}.4 \

-a ${QAT_PF0}.5 -a ${QAT_PF1}.5 -a ${QAT_PF2}.5 \

--vdev crypto_scheduler_pmd_1,worker=${QAT_PF0}.0_qat_sym,worker=${QAT_PF1}.0_qat_sym,worker=${QAT_PF2}.0_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_2,worker=${QAT_PF0}.1_qat_sym,worker=${QAT_PF1}.1_qat_sym,worker=${QAT_PF2}.1_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_3,worker=${QAT_PF0}.2_qat_sym,worker=${QAT_PF1}.2_qat_sym,worker=${QAT_PF2}.2_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_4,worker=${QAT_PF0}.3_qat_sym,worker=${QAT_PF1}.3_qat_sym,worker=${QAT_PF2}.3_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_5,worker=${QAT_PF0}.4_qat_sym,worker=${QAT_PF1}.4_qat_sym,worker=${QAT_PF2}.4_qat_sym,mode=round-robin \

--vdev crypto_scheduler_pmd_6,worker=${QAT_PF0}.5_qat_sym,worker=${QAT_PF1}.5_qat_sym,worker=${QAT_PF2}.5_qat_sym,mode=round-robin \

$CORE_OPTION \

-- --aead-key-sz 16 --buffer-sz 64,128,256,512,1024,2048 --optype aead --ptest throughput --aead-aad-sz 16 \

--devtype crypto_scheduler --aead-op encrypt --burst-sz 32 --total-ops 30000000 --silent \

--digest-sz 16 --aead-algo aes-gcm --aead-iv-sz 12 >> ${LOG_FILE} 2>&1

and the result like AES-CBC-128/SHA1-HMAC, max 8G was reached which so low comparing to Intel's report:

lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf

10 64 32 30000000 30000000 83952749 67111313 2.6117 1.3372 842.36
10 128 32 30000000 30000000 83167004 65221859 2.7881 2.8550 789.07
10 256 32 30000000 30000000 108999257 87092827 2.2573 4.6229 974.62
10 512 32 30000000 30000000 247764304 226160272 1.0595 4.3397 2076.47
10 1024 32 30000000 30000000 343982308 321354180 0.8867 7.2636 2481.21
10 2048 32 30000000 30000000 728382708 707317103 0.4819 7.8961 4564.86

Thanks

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 27 Dec 2023 15:31:12 GMT

Hi, like to see if you have any idea? thanks

Re:need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Wed, 27 Dec 2023 17:40:49 GMT

Hi JCK1,

Allow me a few extra days to try to locate an equivalent platform and run the same test.

Note, the platform I will be using wont be the exactly the same so results will be different and I may want to use DPDK latest version available: dpdk-23.11.tar.xz

Regards,

Ronny G

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 27 Dec 2023 18:24:25 GMT

thanks Ronny.

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Thu, 11 Jan 2024 15:46:23 GMT

Hello Ronny,

Hope you are doing well in New Year.

Any update?

Actually if you have a working system on that you can produce the same results as claimed in the performance data published, I would like to know that system model so we can try to purchase that system. Also please provide details about the system setup in BIOS and Linux kernel setting, so we can quickly replicate that. Is that possible?

Thanks

JCK

Re: need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Thu, 11 Jan 2024 17:00:00 GMT

Hi JCK1,

Happy new year for you as well! I hope you had great Holidays.

I located the following system that I believe is similar enough to try to replicate this issue, see below:

I also have QAT enabled and running:

And I have DPDK also installed and running:

...but I am having some issues when running the dpdk crypto test https://doc.dpdk.org/guides/tools/cryptoperf.html

would it be possible that you send me a quick list of the steps that you are doing? That would help me expedite my testing.

Regards,

Ronny G

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Thu, 11 Jan 2024 17:38:07 GMT

Hi Ronny,

Thanks for your quick reply.

I have not enabled qat_service service (I am not aware of that service) but I just used DPDK for testing.

As I described in my very first post, we configure and bind vfio-pci on QAT VFs once only after the system reboot, i.e. run config-qat-vf.sh and bind-qat-vf.sh:

cat config-qat-vf.sh
sudo modprobe vfio-pci
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs
cat bind-qat-vf.sh
#!/bin/sh
DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
for device in 5 6 7; do \
for fn in $(seq 0 7); do \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
done;
done

After that, in the DPDK 21.11.4 directory, (assume you have compiled that version of DPDK) and then you go inside the directory dpdk-stable-21.11.4/build/app/ to run the test, again this is described in my very first post.

Now this is my understanding of running your QAT DPDK Perf test as I am reading your PDF report, it could be wrong because obviously I could not reproduce the reported results. So if you could do your own way as you are expert on this subject, then you can just tell me how to run your test in correct way so I can just replicate your steps on my testbed, can you help on that?

Thanks
JCK

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 17 Jan 2024 13:28:12 GMT

Hi Ronny

Any update on this? Hope to get your guidance on how to reproduce the DPDK QAT performance result as published on x86 system. Anything useful information from your side would be appreciated.

Thanks

JCK

Re: need to replicate QAT dpdk perf test on x86

JCK1 — Fri, 19 Jan 2024 13:26:03 GMT

Hello Ronny

Would you update me about this?

Thanks

JCK

Re:need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Tue, 30 Jan 2024 22:34:44 GMT

Hi JCK1,

My apologies for the delay in getting back to you.

What do you get when you run service qat_service status?

Can you provide me with a quick screenshot?

Thanks,

Ronny G

Re: Re:need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 31 Jan 2024 00:08:49 GMT

Hi Ronny

Nice to hear from you!

I could not find qat_service package? do you know how to install it? it does not come when install driver from ubuntu.

Thanks

JCK

Re: Re:need to replicate QAT dpdk perf test on x86

JCK1 — Tue, 06 Feb 2024 20:48:45 GMT

Hello Ronny

With current pace, we are going no where. Would you please help us how to bring up our QAT setup to match Intel's test results?

We have target to hit with our customers who are very interested in deploying QAT on their network.

Thank you

JCK

Re:need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Tue, 06 Feb 2024 22:35:25 GMT

Hi JCK1,

I will be checking your case daily in order to help you out with any question you may have, I apologize for any delay in my previous replies.

Checking on your previous notes, I see that you have the following hardware:

Intel® C62x Chipset (formerly Lewisburg) and an Intel® Xeon® Gold 6338N Processor, with this combination and using Ubuntu 22.04.2 LTS, the Intel® QuickAssist Technology Driver for Hardware Version 1.x (known as Customer Enabling Release) is the right driver for you

Here is the URL for the driver mentioned before:

https://www.intel.com/content/www/us/en/download/19734/intel-quickassist-technology-driver-for-linux-hw-version-1-x.html

I would recommend that you deploy the latest driver version available which is 4.24.0-00005

On the other hand, all necessary steps to install and configure the driver are available here:

https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html

Please download the driver, follow the installation guide and let me know if you have questions.

Regards,

Ronny G

Re: Re:need to replicate QAT dpdk perf test on x86

JCK1 — Wed, 07 Feb 2024 15:43:02 GMT

Hi Ronny

Many thanks for your detailed instruction. I followed the Getting Start Guide and got some compiling errors, not sure how to fix that, would you help to see what I am missing?

Thanks again

JCK

Re: need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Wed, 07 Feb 2024 20:53:28 GMT

Hi JCK1,

If I am reading your report correctly, you ran the following commands:

# cd QAT

# ./ configure

# make -j install

You should be getting something like this:

While I do some more investigation, can you please make sure that you have the latest Kernel version? Do you know if it is possible that a previous QAT driver version is present on the system?

Thanks,

Ronny G

Re: need to replicate QAT dpdk perf test on x86

Ronny_G_Intel — Wed, 07 Feb 2024 21:01:00 GMT

Hi JCK1,

Did you initialize the driver?

Notice that you are using Intel® C62x Chipset which requires: # ./configure --enable-icp-dc-sym-only

Are you using any other installation flag?

Thanks,

Ronny G