[Intel C627]Application is stuck in qat_dh_generate_key().

[Alfago12]

Hi Team,

Version information:
driver: QAT1.7.L.4.18.1-00001
engine: QAT_Engine-0.6.14
Openssl: 1.1.1q

Hardware: Intel C627

My application is stuck in qat_dh_generate_key() after upgrading QAT driver and engine.
Note that the memory driver of QAT engine is qat_contig_mem which can be seen from the configure information at the bottom of the issue.
Stack trace:
Thread 1 (Thread 0x7f8451f77f80 (LWP 6078)):
#0 0x00007f84532c0c77 in sched_yield () from /lib64/libc.so.6
#1 0x00007f844f8a5a2b in qat_dh_generate_key (dh=0x7f8451a3bee0) at qat_hw_dh.c:479
And this is triggered by DH_generate_key(dh) in my application.

And after enabling debug for both QAT driver and engine, I got qat_engine_debug.log and attached it.
qat_engine_debug.log

QAT driver seems to be installed after executing "lsmod |grep qa".

dmesg:
[ 71.603452] usdm_drv: Loading USDM Module Version 0.7.1 ...
[ 71.603454] usdm_drv: IOCTLs: c0507100, c0507101, 7102, c0047104
[ 71.604862] Loading QAT CONTIG MEM Module ...
[ 71.645856] c6xx 0000:04:00.0: Enabling default configuration
[ 71.956848] c6xx 0000:04:00.0: qat_dev0 started 10 acceleration engines
[ 72.000848] c6xx 0000:05:00.0: Enabling default configuration
[ 72.311839] c6xx 0000:05:00.0: qat_dev1 started 10 acceleration engines
[ 72.354846] c6xx 0000:06:00.0: Enabling default configuration
[ 72.409206] igb 0000:85:00.0 port9: igb: port9 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 72.409325] IPv6: ADDRCONF(NETDEV_CHANGE): port9: link becomes ready
[ 72.665826] c6xx 0000:06:00.0: qat_dev2 started 10 acceleration engines
[ 72.688975] QAT: Stopping all acceleration devices.
[ 72.688980] c6xx 0000:04:00.0: qat_dev0 stopped 10 acceleration engines
[ 72.689464] c6xx 0000:04:00.0: Resetting device qat_dev0
[ 72.689465] c6xx 0000:04:00.0: Function level reset
[ 72.797966] c6xx 0000:05:00.0: qat_dev1 stopped 10 acceleration engines
[ 72.798529] c6xx 0000:05:00.0: Resetting device qat_dev1
[ 72.798530] c6xx 0000:05:00.0: Function level reset
[ 72.901964] c6xx 0000:06:00.0: qat_dev2 stopped 10 acceleration engines
[ 72.902519] c6xx 0000:06:00.0: Resetting device qat_dev2
[ 72.902520] c6xx 0000:06:00.0: Function level reset
[ 74.009175] c6xx 0000:04:00.0: Starting acceleration device qat_dev0.
[ 74.359864] c6xx 0000:04:00.0: qat_dev0 started 10 acceleration engines
[ 74.385812] QDM: iommu_pass_through is set to 0
[ 74.387457] c6xx 0000:05:00.0: Starting acceleration device qat_dev1.
[ 74.735857] c6xx 0000:05:00.0: qat_dev1 started 10 acceleration engines
[ 74.758956] c6xx 0000:06:00.0: Starting acceleration device qat_dev2.
[ 75.092824] c6xx 0000:06:00.0: qat_dev2 started 10 acceleration engines
...
[ 108.954019] WARNING: CPU: 31 PID: 6078 at mm/mmap.c:1807 mmap_region+0x564/0x650
[ 108.954020] Modules linked in: igb_uio(O) qat_c62x(O) qat_contig_mem(O) usdm_drv(O) intel_qat(O) fwbrnd(O) ha(O) network_bypass(O) caswell_bpgen3(O) caswell_pmbus(O) ixgbe(O) cp_uio(O) log_msgid(O) mcast(O) wproxy(O) rp_tproxy(O) tproxy(O) tp_v6_defrag(O) antidos(O) vserver_filter(O) intf_filter(O) bridge_mac(O) ti_bridge(O) share_ip(O) miglog(O)
[ 108.954035] CPU: 31 PID: 6078 Comm: proxyd Kdump: loaded Tainted: G O 5.4.202 #1
[ 108.954036] Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 5.6.5 07/27/2015
[ 108.954039] RIP: 0010:mmap_region+0x564/0x650
[ 108.954040] Code: 48 8b 04 24 48 85 c0 0f 84 ea fc ff ff 8b 15 ab 40 10 01 48 f7 d8 48 c7 c7 c0 26 9f 81 48 89 c6 e8 91 49 1f 00 e9 cd fc ff ff <0f> 0b e9 9d fe ff ff 48 8b 7c 24 08 4c 89 4c 24 10 4d 63 e6 49 c7
[ 108.954041] RSP: 0018:ffffc9000ae03d50 EFLAGS: 00010206
[ 108.954043] RAX: 0000000000000000 RBX: ffff88903a399e20 RCX: ffffea0000000000
[ 108.954044] RDX: 8000000000000027 RSI: 000ffffffffff000 RDI: 00007f8441aa0000
[ 108.954044] RBP: 00007f8441a7a000 R08: 0000000000000000 R09: 0000000000000008
[ 108.954045] R10: ffff88903a399ec0 R11: ffff8890597f0780 R12: ffff88903a399e30
[ 108.954046] R13: ffff88905926ec00 R14: 00007f8441a80000 R15: ffff8890597f0780
[ 108.954047] FS: 00007f8451f77f80(0000) GS:ffff88905f8c0000(0000) knlGS:0000000000000000
[ 108.954048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 108.954049] CR2: 00007f8450330000 CR3: 00000010385a2005 CR4: 00000000001606e0
[ 108.954050] Call Trace:
[ 108.954057] do_mmap+0x397/0x590
[ 108.954060] vm_mmap_pgoff+0x92/0xf0
[ 108.954063] ksys_mmap_pgoff+0x18c/0x230
[ 108.954066] ? ksys_ioctl+0x58/0x70
[ 108.954069] do_syscall_64+0x68/0x3c0
[ 108.954074] ? __do_page_fault+0x23d/0x480
[ 108.954080] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 108.954083] RIP: 0033:0x7f84532d3402
[ 108.954084] Code: 00 00 00 00 0f 1f 40 00 41 f7 c1 ff 0f 00 00 75 27 55 48 89 fd 53 89 cb 48 85 ff 74 3b 41 89 da 48 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 66 5b 5d c3 0f 1f 00 48 8b 05 59 5a 0d 00 64
[ 108.954085] RSP: 002b:00007ffecdc793f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
[ 108.954086] RAX: ffffffffffffffda RBX: 0000000000002001 RCX: 00007f84532d3402
[ 108.954087] RDX: 0000000000000003 RSI: 0000000000040000 RDI: 0000000000000000
[ 108.954087] RBP: 0000000000000000 R08: 000000000000003c R09: ffff88903a3e0000
[ 108.954088] R10: 0000000000002001 R11: 0000000000000246 R12: 0000000000000157
[ 108.954089] R13: 00007f8451a23100 R14: 0000000000000000 R15: 0000000000000100
[ 108.954091] ---[ end trace cc675b012022be0a ]---
/#

/# bin/adf_ctl status
Checking status of all devices.
There is 3 QAT acceleration device(s) in the system:
qat_dev0 - type: c6xx, inst_id: 0, node_id: 0, bsf: 0000:04:00.0, #accel: 5 #engines: 10 state: up
qat_dev1 - type: c6xx, inst_id: 1, node_id: 0, bsf: 0000:05:00.0, #accel: 5 #engines: 10 state: up
qat_dev2 - type: c6xx, inst_id: 2, node_id: 0, bsf: 0000:06:00.0, #accel: 5 #engines: 10 state: up

#18
./qat_contig_mem_test
seg mapped to 0x7f3f99c6d000, virtualAddress in seg 0xffff8890531f7000, length 64
Hello world!

PASS Verify for QAT Contig Mem Test.

configure for QAT engine:
./configure
--with-qat_hw_dir=xxxxxx
--with-qat_hw_install_dir=xxxx
--with-openssl_dir=xxxxxx
--with-openssl_install_dir=xxxxx
--enable-qat_hw_multi_thread
--enable-qat_hw_contig_mem
--disable-qat_hw_lenstra_protection ;

Try to get more debug information according to Intel® QuickAssist Technology Debugging Guide, June 2021, Revision 1.4(Document Number: 621658).

2.6 How to Enable Debug Symbols for the QAT Driver
For the QAT driver, you can enable debugging with these changes:
quickassist/build_system/build_files/env_files/environment.mk ICP_DEFENSES_ENABLED ?= n
quickassist/build_system/build_files/common.mk $(PROG_ACY)_OPT_LEVEL?=0
Makefile (change it after ./configure, then make and install) CFLAGS = -g
2.7 How to Enable Debug Compile for QAT_engine
For the QAT_engine / OpenSSL*, you can enable debugging with these changes: QAT_Engine:

1. run ./configure with additional parameters such as the following:
   --enable-qat_debug
   --enable-qat_warnings
   --enable-qat_mem_warnings
   --enable-qat_mem_debug
   --with-qat_debug_file=/qat_engine_debug.log
2. Check QAT Engine Makefile and make the following changes:
   Change cflags = -shared -fPIC -Wall -Wformat -Wformat-security -O2 -D_FORTIFY_SOURCE=2 -fstack-protector to cflags = -shared -fPIC -Wall -Wformat -Wformat-security –O0 -g -D_FORTIFY_SOURCE=0 -fstack-protector
   Change "CFLAGS = -g -O2" to "CFLAGS = -g –O0"
   Change "CXXFLAGS = -g -O2" to "CXXFLAGS = -g –O0"

For QAT engine, after configure as follows.
./configure
--with-qat_hw_dir=$(TOPDIR)/kernel/modules/qat_driver
--with-qat_hw_install_dir=$(istldir)
--with-openssl_dir=$(curdir)/$(openssl_pkg)
--with-openssl_install_dir=$(curdir)/$(openssl_pkg)
--enable-qat_hw_multi_thread
--enable-qat_hw_contig_mem
--disable-qat_hw_lenstra_protection
--enable-qat_warnings
--enable-qat_debug
--enable-qat_mem_warnings
--enable-qat_mem_debug
--with-qat_debug_file=/var/log/qat_engine_debug.log ;

#26
According to this issue, I modify all QAT driver config files /etc/c6xx_dev0.conf and change NumProcesses from 1 to 2. But after that, the following command still fails.

openssl engine -t -c -v qatengine

(qatengine) Reference implementation of QAT crypto engine(qat_hw) v0.6.14
[RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, X25519, X448]
QAT_HW initialization Failed
[ unavailable ]
ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD,
GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL,
GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE,
GET_NUM_CRYPTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE,
SET_EPOLL_TIMEOUT, SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD,
ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING,
GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME,
ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP

qat_engine_debug.log
[WARN][4080.712398] PID [12795] Thread [7f51c5d71500][e_qat.c:944:bind_qat()] QAT Warnings enabled.
[DEBUG][4080.712413] PID [12795] Thread [7f51c5d71500][e_qat.c:945:bind_qat()] QAT Debug enabled.
[WARN][4080.712416] PID [12795] Thread [7f51c5d71500][e_qat.c:946:bind_qat()] qatengine - Reference implementation of QAT crypto engine(qat_hw) v0.6.14
[DEBUG][4080.712508] PID [12795] Thread [7f51c5d71500][e_qat.c:994:bind_qat()] Registering QAT HW supported algorithms
[DEBUG][4080.712517] PID [12795] Thread [7f51c5d71500][qat_hw_dsa.c:118:qat_get_DSA_methods()] QAT HW DSA registration succeeded
[DEBUG][4080.712520] PID [12795] Thread [7f51c5d71500][qat_hw_dh.c:130:qat_get_DH_methods()] QAT HW DH registration succeeded
[DEBUG][4080.712524] PID [12795] Thread [7f51c5d71500][qat_evp.c:1009:qat_get_RSA_methods()] QAT HW RSA Registration succeeded
[DEBUG][4080.712530] PID [12795] Thread [7f51c5d71500][qat_evp.c:864:qat_get_EC_methods()] QAT HW ECDSA Registration succeeded
[DEBUG][4080.712532] PID [12795] Thread [7f51c5d71500][qat_evp.c:923:qat_get_EC_methods()] QAT HW ECDH Registration succeeded
[DEBUG][4080.712539] PID [12795] Thread [7f51c5d71500][qat_hw_ciphers.c:278:qat_create_cipher_meth()] QAT HW AES_CBC_128_HMAC_SHA registration succeeded
[DEBUG][4080.712542] PID [12795] Thread [7f51c5d71500][qat_hw_ciphers.c:278:qat_create_cipher_meth()] QAT HW AES_CBC_128_HMAC_SHA registration succeeded
[DEBUG][4080.712544] PID [12795] Thread [7f51c5d71500][qat_hw_ciphers.c:278:qat_create_cipher_meth()] QAT HW AES_CBC_256_HMAC_SHA registration succeeded
[DEBUG][4080.712546] PID [12795] Thread [7f51c5d71500][qat_hw_ciphers.c:278:qat_create_cipher_meth()] QAT HW AES_CBC_256_HMAC_SHA registration succeeded
[DEBUG][4080.712580] PID [12795] Thread [7f51c5d71500][e_qat.c:519:qat_engine_init()] QAT Engine initialization:
[DEBUG][4080.712583] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:586:qat_init()] QAT_HW initialization:
[DEBUG][4080.712586] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:587:qat_init()] - External polling: OFF
[DEBUG][4080.712588] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:588:qat_init()] - Heuristic polling: OFF
[DEBUG][4080.712590] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:589:qat_init()] - SW Fallback: OFF
[DEBUG][4080.712592] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:590:qat_init()] - Inline polling: OFF
[DEBUG][4080.712594] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:591:qat_init()] - Internal poll interval: 10000ns
[DEBUG][4080.712596] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:592:qat_init()] - Epoll timeout: 1000ms
[DEBUG][4080.712599] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:593:qat_init()] - Event driven polling mode: OFF
[DEBUG][4080.712601] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:594:qat_init()] - Instance for thread: OFF
[DEBUG][4080.712603] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:595:qat_init()] - Max retry count: 5
[WARN][4080.712664] PID [12795] Thread [7f51c5d71500][qat_hw_init.c:609:qat_init()] icp_sal_userStart failed
[DEBUG][4080.712756] PID [12795] Thread [7f51c5d71500][qat_hw_prf.c:107:qat_prf_pmeth()] QAT HW PRF Registration succeeded
[DEBUG][4080.712763] PID [12795] Thread [7f51c5d71500][qat_evp.c:422:qat_x25519_pmeth()] QAT HW X25519 registration succeeded
[DEBUG][4080.712765] PID [12795] Thread [7f51c5d71500][qat_evp.c:478:qat_x448_pmeth()] QAT HW ECDH X448 Registration succeeded
[DEBUG][4080.712768] PID [12795] Thread [7f51c5d71500][e_qat.c:407:qat_engine_destroy()] ---- Destroying Engine...

Even after I use USDM as memory driver for QAT engine, my application still encounters the same issue.
Thread 1 (Thread 0x7f882d0d4f80 (LWP 6105)):
#0 0x00007f882e41dc77 in sched_yield () from /lib64/libc.so.6
#1 0x00007f882a9f850b in qat_dh_generate_key () from /lib/qatengine.so

configure QAT engine:
./configure
--with-qat_hw_dir=xxxx
--with-qat_hw_install_dir=xxxx
--with-openssl_dir=xxxx
--with-openssl_install_dir=xxxx
--disable-qat_hw_lenstra_protection ;

Regards,

Allen

[Jaime_Lizarme]

Hello,

Thank you for contacting Intel Embedded Community.

To better assist you with your question. I am transferring this thread to the proper server category.

Best regards,

Jaime L.

Intel Customer Support Engineer