QAT_HW device not available & QAT_SW not enabled. Using OpenSSL_SW!

[padmanabhanvg]

Hi,

This is INTEL(R) XEON(R) GOLD 6548Y+, Debian 12 (kernel 6.1.123) Physical Server.

root@10.62.130.82/home/sas/QAT_Engine#openssl engine -t -c -v qatengine
QAT_HW device not available & QAT_SW not enabled. Using OpenSSL_SW!
(qatengine) Reference implementation of QAT cr*pto engine(qat_hw) v1.9.0
[RSA, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-CCM, id-aes192-CCM, id-aes256-CCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448]
[ available ]
ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD,
GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL,
GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE,
GET_NUM_CR*PTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE,
SET_EPOLL_TIMEOUT, SET_CR*PTO_SMALL_PACKET_OFFLOAD_THRESHOLD,
ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING,
GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME,
ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP
root@10.62.130.82/home/sas/QAT_Engine#systemctl status qat
● qat.service - QAT service
Loaded: loaded (/lib/systemd/system/qat.service; enabled; preset: enabled)
Active: active (exited) since Thu 2025-05-08 15:43:28 IST; 2h 34min ago
Process: 2637 ExecStart=/etc/init.d/qat_service start (code=exited, status=0/SUCCESS)
Main PID: 2637 (code=exited, status=0/SUCCESS)
root@10.62.130.82/home/sas/QAT_Engine#lsmod | grep qat
qat_4xxx 28672 0
intel_qat 729088 2 qat_4xxx,usdm_drv
uio 28672 1 intel_qat
root@10.62.130.82/home/sas/QAT_Engine#lspci -d 8086:4942
f3:00.0 Co-processor: Intel Corporation 401xx Series QAT (rev 01)
root@10.62.130.82/home/sas/QAT_Engine#uname -a
Linux 10.62.130.82 6.1.123 #ZSIGNED SMP PREEMPT_DYNAMIC Thu Apr 17 18:38:48 IST 2025 x86_64 GNU/Linux
root@10.62.130.82/home/sas/QAT_Engine#ls -la /dev/qat*
crw-rw---- 1 root qat 242, 0 May 8 15:43 /dev/qat_adf_ctl
crw-rw---- 1 root qat 241, 0 May 8 15:43 /dev/qat_dev_processes
root@10.62.130.82/home/sas/QAT_Engine#cat /boot/config-$(uname -r) | grep -i qat
CONFIG_CR*PTO_DEV_QAT=m
CONFIG_CR*PTO_DEV_QAT_DH895xCC=m
CONFIG_CR*PTO_DEV_QAT_C3XXX=m
CONFIG_CR*PTO_DEV_QAT_C62X=m
CONFIG_CR*PTO_DEV_QAT_4XXX=m
CONFIG_CR*PTO_DEV_QAT_DH895xCCVF=m
CONFIG_CR*PTO_DEV_QAT_C3XXXVF=m
CONFIG_CR*PTO_DEV_QAT_C62XVF=m
root@10.62.130.82/home/sas/QAT_Engine#ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}.bin* 2>/dev/null
/lib/firmware/qat_402xx.bin /lib/firmware/qat_420xx.bin /lib/firmware/qat_4xxx.bin
root@10.62.130.82/home/sas/QAT_Engine#ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}_mmp.bin* 2>/dev/null
/lib/firmware/qat_402xx_mmp.bin /lib/firmware/qat_4xxx_mmp.bin
root@10.62.130.82/home/sas/QAT_Engine#echo `(lspci -d 8086:4940 && lspci -d 8086:4941 && lspci -d 8086:4942 && lspci -d 8086:4943 && lspci -d 8086:4944 && lspci -d 8086:4945 && lspci -d 8086:4946 && lspci -d 8086:4947) | wc -l` supported devices found.
1 supported devices found.
root@10.62.130.82/home/sas/QAT_Engine#echo `(lspci -vvv -d 8086:4940 && lspci -vvv -d 8086:4942 && lspci -vvv -d 8086:4944 && lspci -vvv -d 8086:4946) | grep "Kernel driver"`
Kernel driver in use: 4xxx
root@10.62.130.82/home/sas/QAT_Engine#  

In /etc/default/grub
 GRUB_CMDLINE_LINUX="crashkernel=256M,high panic=30 biosdevname=0 net.ifnames=0 intel_pstate=disable fsck.repair=yes systemd.unified_cgroup_hierarchy=0 intel_iommu=off"

I have installed intree qat kernel driver modules, so used, 
./configure --prefix=/opt/openssl/3.0.15 --with-openssl_install_dir=/opt/openssl/3.0.15 --with-openssl_dir=/home/sas/openssl-3.0.15 --host=x86_64-linux-gnu --build=x86_64-linux-gnu --enable-qat_hw

in qat engine.

Could you please advise what I might be missing?

Thank you

[Ronny_G_Intel]

Hi padmanabhanvg,

I'm looking into this issue at the moment. Meanwhile, could you please provide the icp_dump? You can generate it by executing the script found at $ICP_ROOT/quickassist/utilities/debug_tool/icp_dump.sh. This will produce a tar file that includes your complete system setup and configuration files.

Regards,

Ronny G

[padmanabhanvg]

Hi Ronny,

Please find the attached file from icp dump.


Thanks,
Padmanabhan V.

[Ronny_G_Intel]

Hi padmanabhanvg,

Thanks for providing the icp_dump.

If you are using the In-Tree driver you need to have intel_iommu=on and you have this option set to off as indicated below:

/etc/default/grub

GRUB_CMDLINE_LINUX="crashkernel=256M,high panic=30 biosdevname=0 net.ifnames=0 intel_pstate=disable fsck.repair=yes systemd.unified_cgroup_hierarchy=0 intel_iommu=off"

You may need to $sudo update-grub after enabling this option, to activate the changes.

You could also visit the following documentation for further details: https://intel.github.io/quickassist/qatlib/index.html

Please let me know the results.

Regards,

Ronny G

[Ronny_G_Intel]

Hi padmanabhanvg,

Did you try my previous recommendation? Any updates?

Regards,

Ronny G

[Ronny_G_Intel]

Hi padmanabhanvg,

I am just checking if you have any updates regarding this issue.

Please let me know.

Regards,

Ronny G

[padmanabhanvg]

Hi Ronny,

We had installed signed Debian kernel, since qat_4xxx and intel_qat are not signed by default, We have signed it separately. So it was showed as OOT kernel driver. after enable intel_iommu=on, I was getting error as Cannot use PF with IOMMU enabled and SVM off in dmesg. Also, I was getting below error message,

root@10.62.130.82/home/sas/QAT_Engine#openssl engine -t -c -v qatengine ADF_UIO_PROXY err: uio_udev_get_device_from_devid: udev_device_new_from_subsystem_sysname failed for sysname 0000:78:18.0 QAT_HW device not available & QAT_SW not enabled. Using OpenSSL_SW!

After disable intel_iommu=off, I was getting

root@10.62.130.82/etc#openssl engine -t qatengine (qatengine) Reference implementation of QAT cr*pto engine(qat_hw) v1.9.0 ADF_UIO_PROXY err: icp_adf_userProcessToStart: Error reading /dev/qat_dev_processes file QAT HW initialization Failed. [ unavailable ]

Then, I decided to do this testing without signed kernel and with Intel QAT driver(OOT), installed unsigned kernel and intel provided qat driver, after that I was still getting,

root@10.62.130.82/home/sas#openssl engine -t -c -v qatengine (qatengine) Reference implementation of QAT cr*pto engine(qat_hw) v1.9.0 [RSA, AES-256-CBC-HMAC-SHA256, id-aes128-CCM, id-aes192-CCM, id-aes256-CCM, TLS1-PRF, X25519, X448] QAT HW initialization Failed. [ unavailable ]

After adding, SHIM section in qat conf,

# Added SHIM section to /etc/4xxx_dev0.conf
[SHIM]
NumberCyInstances = 2
NumberDcInstances = 2
NumProcesses = 1
Cy0Name = "UserCY0"
Cy0IsPolled = 1
Cy0CoreAffinity = 1
Cy1Name = "UserCY1"
Cy1IsPolled = 1
Cy1CoreAffinity = 2
Dc0Name = "UserDC0"
Dc0IsPolled = 1
Dc0CoreAffinity = 3
Dc1Name = "UserDC1"
Dc1IsPolled = 1
Dc1CoreAffinity = 4

setted proper permissions on the device files
sudo chmod 666 /dev/qat_dev_processes
sudo chmod 666 /dev/qat_adf_ctl

exported,
export OPENSSL_ENGINES=/usr/lib/x86_64-linux-gnu/engines-3

and finally,
sudo systemctl restart qat

root@10.62.130.82/etc#openssl engine -t -c -v qatengine
(qatengine) Reference implementation of QAT cr*pto engine(qat_hw) v1.9.0
[RSA, AES-256-CBC-HMAC-SHA256, id-aes128-CCM, id-aes192-CCM, id-aes256-CCM, TLS1-PRF, X25519, X448]
[ available ]
ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD,
GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL,
GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE,
GET_NUM_CR*PTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE,
SET_EPOLL_TIMEOUT, SET_CRY*TO_SMALL_PACKET_OFFLOAD_THRESHOLD,
ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING,
GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME,
ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP
root@10.62.130.82/etc#

Finally my issue got resolved.


Thanks,
Padmanabhan V.

[Ronny_G_Intel]

Hi Padmanabhan 

Thank you for letting us know that you resolved the issue and for providing the detailed steps you followed.

This information will certainly help other users facing similar problems.

Regards,

Ronny G 

[yejn123]

Hello, I have encountered a similar problem. Do you have a detailed solution?

Here is my testing information

I would like to clarify if there are any issues with my driver version first

QAT22.L.1.0.0-00001.tar.gz

root@szns-p-se201-2-n2:~# lspci -nn | grep -E '8086:37c8|8086:19e2|8086:0435|8086:6f54|8086:4940|8086:4942'
f3:00.0 Co-processor [0b40]: Intel Corporation Device [8086:4942] (rev 40)
f7:00.0 Co-processor [0b40]: Intel Corporation Device [8086:4942] (rev 40)
root@szns-p-se201-2-n2:~# ls  /dev/qat*
/dev/qat_adf_ctl  /dev/qat_dev_processes
root@szns-p-se201-2-n2:~# adf_ctl status
Checking status of all devices.
There is 2 QAT acceleration device(s) in the system:
 qat_dev0 - type: 4xxx,  inst_id: 0,  node_id: 0,  bsf: 0000:f3:00.0,  #accel: 1 #engines: 9 state: up
 qat_dev1 - type: 4xxx,  inst_id: 1,  node_id: 0,  bsf: 0000:f7:00.0,  #accel: 1 #engines: 9 state: up
root@szns-p-se201-2-n2:~# docker run -it --cap-add=IPC_LOCK --security-opt seccomp=unconfined --security-opt apparmor=unconfined $(for i in `ls /dev/vfio/*`; do echo --device $i; done) --device /dev/qat_adf_ctl --device /dev/qat_dev_processes --device /dev/usdm_drv --cpuset-cpus  2-3 --env QAT_POLICY=1 --ulimit memlock=524288000:524288000 qat_crypto_base openssl speed -engine qatengine -elapsed -async_jobs 72  -multi 2 rsa2048
Forked child 0
Forked child 1
QAT_HW device not available & QAT_SW not enabled. Using OpenSSL_SW!
Engine "qatengine" set.
QAT_HW device not available & QAT_SW not enabled. Using OpenSSL_SW!
Engine "qatengine" set.
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+R1:21558:2048:10.00
+DTP:2048:public:rsa:10
+R1:21989:2048:10.00
+DTP:2048:public:rsa:10
+R2:458028:2048:10.00
+R2:458339:2048:10.00
Got: +F2:2:2048:2198.900000:45833.900000 from 0
Got: +F2:2:2048:2155.800000:45802.800000 from 1
version: 3.0.16
built on: Wed Aug 13 06:49:50 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0xfb417ffef3bfb7ef
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000230s 0.000011s   4354.7  91636.7