- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Based on https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf
I am having problems to replicate Intel's test on x86, the best I can get is 7.9gbps on Xeon(R) Gold 6338N CPU @ 2.20GHz from SuperMicro. I tried to follow your instructions of configuring BIOS and kernel settings, etc on your report.
The VFs were assigned with vfio-pci driver not QAT's VF driver.
Anything I could be missing?
Thanks in advance.
JC
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
Thanks for reaching out to our Intel communities.
This particular forum is designated to provided assistance for Intel® QuickAssist Technology (Intel® QAT) related issues so I need to ask a couple of initial question to try to understand the situation, please see below:
Are you using Intel® QAT? If yes:
What Intel® QAT hardware are you using?
What Intel® QAT Driver Version are you using?
Can you also provide me with the DPDK version that you are trying?
What Operating System is installed, please include the Kernel version as well?
Note: the general system configuration would be very helpful in cases related to benchmarking or performance testing.
Regards,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
Here is the info:
Are you using Intel® QAT? If yes:
What Intel® QAT hardware are you using? Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
What Intel® QAT Driver Version are you using? it comes with linux kernel 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC
Can you also provide me with the DPDK version that you are trying? dpdk-stable-21.11.4
What Operating System is installed, please include the Kernel version as well? Ubuntu 22.04.2 LTS /linux kernel 6.2.0-37-generic
It is SuperMicro server with Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz. I am also looking for a better Xeon machine to try at the moment.
please let me know if you need more info.
thanks
JC
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
Thanks for the info provided.
There are several "Test Case" scenarios in the DPDK Intel Cryptodev Performance Report <https://fast.dpdk.org/doc/perf/DPDK_21_11_Intel_crypto_performance_report.pdf>, are you referring to anyone in particular?
I need as much detail as possible to try to replicate the same scenario using a similar system (however this may be very challenging)
Thanks,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
Here is more details:
on x86:
- HT is on, 32 Cores 64 HT
- iommu.passthrough=1 default_hugepagesz=1G hugepagesz=1G hugepages=32 disable_denylist=1 intel_iommu=on iommu=pt intel_pstate=disable isolcpus=4-7,12-15 nohz_full=4-7,12-15 rcu_nocbs=4-7,12-15
- 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC
- we run these to configure and bind vfio-pci on QAT VFs:
- cat config-qat-vf.sh
sudo modprobe vfio-pci
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs - cat bind-qat-vf.sh
#!/bin/sh
DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
for device in 5 6 7; do \
for fn in $(seq 0 7); do \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
done;
done
- cat config-qat-vf.sh
then we run these steps which are copied from Intel's report with some codification:
lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf
10 64 32 30000000 30000000 83952749 67111313 2.6117 1.3372 842.36
10 128 32 30000000 30000000 83167004 65221859 2.7881 2.8550 789.07
10 256 32 30000000 30000000 108999257 87092827 2.2573 4.6229 974.62
10 512 32 30000000 30000000 247764304 226160272 1.0595 4.3397 2076.47
10 1024 32 30000000 30000000 343982308 321354180 0.8867 7.2636 2481.21
10 2048 32 30000000 30000000 728382708 707317103 0.4819 7.8961 4564.86
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, like to see if you have any idea? thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
Allow me a few extra days to try to locate an equivalent platform and run the same test.
Note, the platform I will be using wont be the exactly the same so results will be different and I may want to use DPDK latest version available: dpdk-23.11.tar.xz
Regards,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks Ronny.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Ronny,
Hope you are doing well in New Year.
Any update?
Actually if you have a working system on that you can produce the same results as claimed in the performance data published, I would like to know that system model so we can try to purchase that system. Also please provide details about the system setup in BIOS and Linux kernel setting, so we can quickly replicate that. Is that possible?
Thanks
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
Happy new year for you as well! I hope you had great Holidays.
I located the following system that I believe is similar enough to try to replicate this issue, see below:
I also have QAT enabled and running:
And I have DPDK also installed and running:
...but I am having some issues when running the dpdk crypto test https://doc.dpdk.org/guides/tools/cryptoperf.html
would it be possible that you send me a quick list of the steps that you are doing? That would help me expedite my testing.
Regards,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny,
Thanks for your quick reply.
I have not enabled qat_service service (I am not aware of that service) but I just used DPDK for testing.
As I described in my very first post, we configure and bind vfio-pci on QAT VFs once only after the system reboot, i.e. run config-qat-vf.sh and bind-qat-vf.sh:
- cat config-qat-vf.sh
sudo modprobe vfio-pci
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c5\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c6\:00.0/sriov_numvfs
echo "16" | sudo tee /sys/bus/pci/drivers/c6xx/0000\:c7\:00.0/sriov_numvfs - cat bind-qat-vf.sh
#!/bin/sh
DEVBIND_PATH=/home/ubuntu/dpdk-stable-21.11.4/usertools/dpdk-devbind.py
for device in 5 6 7; do \
for fn in $(seq 0 7); do \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:01.${fn}; \
sudo ${DEVBIND_PATH} --bind=vfio-pci c${device}:02.${fn}; \
echo 0000:c${device}:01.${fn} 0000:c${device}:02.${fn};
done;
done
After that, in the DPDK 21.11.4 directory, (assume you have compiled that version of DPDK) and then you go inside the directory dpdk-stable-21.11.4/build/app/ to run the test, again this is described in my very first post.
Now this is my understanding of running your QAT DPDK Perf test as I am reading your PDF report, it could be wrong because obviously I could not reproduce the reported results. So if you could do your own way as you are expert on this subject, then you can just tell me how to run your test in correct way so I can just replicate your steps on my testbed, can you help on that?
Thanks
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
Any update on this? Hope to get your guidance on how to reproduce the DPDK QAT performance result as published on x86 system. Anything useful information from your side would be appreciated.
Thanks
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
My apologies for the delay in getting back to you.
What do you get when you run service qat_service status?
Can you provide me with a quick screenshot?
Thanks,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
Nice to hear from you!
I could not find qat_service package? do you know how to install it? it does not come when install driver from ubuntu.
Thanks
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Ronny
With current pace, we are going no where. Would you please help us how to bring up our QAT setup to match Intel's test results?
We have target to hit with our customers who are very interested in deploying QAT on their network.
Thank you
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
I will be checking your case daily in order to help you out with any question you may have, I apologize for any delay in my previous replies.
Checking on your previous notes, I see that you have the following hardware:
Intel® C62x Chipset (formerly Lewisburg) and an Intel® Xeon® Gold 6338N Processor, with this combination and using Ubuntu 22.04.2 LTS, the Intel® QuickAssist Technology Driver for Hardware Version 1.x (known as Customer Enabling Release) is the right driver for you
Here is the URL for the driver mentioned before:
I would recommend that you deploy the latest driver version available which is 4.24.0-00005
On the other hand, all necessary steps to install and configure the driver are available here:
Please download the driver, follow the installation guide and let me know if you have questions.
Regards,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
Many thanks for your detailed instruction. I followed the Getting Start Guide and got some compiling errors, not sure how to fix that, would you help to see what I am missing?
Thanks again
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
If I am reading your report correctly, you ran the following commands:
# cd QAT
# ./ configure
# make -j install
You should be getting something like this:
While I do some more investigation, can you please make sure that you have the latest Kernel version? Do you know if it is possible that a previous QAT driver version is present on the system?
Thanks,
Ronny G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ronny
We had a QAT kernel driver came with the kernel we installed:
root@xeon-sm-01:/QAT# lsmod | grep c6
qat_c62x 12288 0
intel_qat 274432 1 qat_c62x
root@xeon-sm-01:/QAT# uname -a
Linux xeon-sm-01 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
So what is the latest kernel you support? 6.5.0 is pretty new version of kernel.
I unloaded the qat_c62x and intel_qat from the system, and did this:
./configure --enable-icp-dc-sym-only
make -j install
the errors seem to be persistent:
Building qat_direct for user space
make[2]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src'
test -d build || mkdir -p build;
make[3]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/uio'
make[3]: Entering directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/common'
make[5]: *** No rule to make target 'vmlinux.o', needed by 'modpost'. Stop.
make[5]: *** Waiting for unfinished jobs....
warning: the compiler differs from the one used to build the kernel
The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
make[4]: *** [Makefile:234: __sub-make] Error 2
make[4]: Leaving directory '/usr/src/linux-headers-6.5.0-15-generic'
Copying outputs
mv: cannot stat 'OsalServices.o': No such file or directory
mv: cannot stat 'OsalSemaphore.o': No such file or directory
mv: cannot stat 'OsalSpinLock.o': No such file or directory
mv: cannot stat 'OsalThread.o': No such file or directory
mv: cannot stat 'OsalAtomic.o': No such file or directory
mv: cannot stat 'OsalMemMgmt.o': No such file or directory
mv: cannot stat 'OsalIOMMUMgmt.o': No such file or directory
mv: cannot stat 'OsalUsrKrlProxy.o': No such file or directory
mv: cannot stat 'OsalCryptoInterface.o': No such file or directory
make[3]: Leaving directory '/QAT/quickassist/utilities/osal/src/linux/kernel_space'
make[2]: Leaving directory '/QAT/quickassist/utilities/osal/src'
Copying OSAL library
cp /QAT/quickassist/utilities/osal//src/linux/kernel_space/build/linux_2.6/kernel_space/libosal.a /QAT/build/libosal_kernel.a;
cp: cannot stat '/QAT/quickassist/utilities/osal//src/linux/kernel_space/build/linux_2.6/kernel_space/libosal.a': No such file or directory
make[1]: *** [Makefile:152: libosal_kernel] Error 1
make[1]: *** Waiting for unfinished jobs....
Creating output directory
Creating output directory
AR /QAT/quickassist/utilities/libusdm_drv/linux/lib.a
ar: creating libadf_uio.a
make[3]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/uio'
ar: creating libadf_common.a
make[3]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src/common'
Creating static library libadf_user.a
ar rcs libadf_user.a build/*.o
make[2]: Leaving directory '/QAT/quickassist/lookaside/access_layer/src/qat_direct/src'
Copying qat_direct library
cp /QAT/quickassist/lookaside/access_layer/src/qat_direct//src/libadf_user.a /QAT/build/libadf.a;
LD [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.o
MODPOST /QAT/quickassist/utilities/libusdm_drv/linux/Module.symvers
CC [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.mod.o
LD [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko
BTF [M] /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko
Skipping BTF generation for /QAT/quickassist/utilities/libusdm_drv/linux/usdm_drv.ko due to unavailability of vmlinux
make[4]: Leaving directory '/usr/src/linux-headers-6.5.0-15-generic'
Copying outputs
make[3]: Leaving directory '/QAT/quickassist/utilities/libusdm_drv/linux'
make[2]: Leaving directory '/QAT/quickassist/utilities/libusdm_drv'
make[1]: Leaving directory '/QAT/quickassist'
make: *** [Makefile:1187: quickassist-all] Error 2
Not sure what is error above?
Thanks
JCK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JCK1,
Did you initialize the driver?
Notice that you are using Intel® C62x Chipset which requires: # ./configure --enable-icp-dc-sym-only
Are you using any other installation flag?
Thanks,
Ronny G
![](/skins/images/D2683F18326913BBA0436CB7114DD569/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page