topic Re: Re:QAT Driver Error in Nginx error log in Intel® QuickAssist Technology (Intel® QAT)

QAT Driver Error in Nginx error log

SankarKumar — Fri, 31 May 2024 06:59:50 GMT

In CentOS Linux release 8.2.2004 system, we are using async_mode nginx compiled with QAT Driver, openssl and QAT Engine.

QAT Driver 4.4.0

OpenSSL-1.1.1c

QAT Engine 0.5.42

Async Mode Nginx v0.4.0

We are facing the below issues in one particular server

2024/05/28 00:16:48 [crit] 3551626#0: *177676548 SSL_do_handshake() failed (SSL: error:800750D6:lib(128):qat_ecdh_compute_key:presultx pdata malloc failure error:1422F006:SSL routines:ssl_generate_pkey_group:EVP lib) while SSL handshaking, client: <ip>, server: 0.0.0.0:443

ioctl_alloc_slab:957 mmap on memory allocated through ioctl failed

$ dmesg | grep alloc

[305593.110012] nginx: page allocation failure: order:9, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null)

[564790.240682] nginx: page allocation failure: order:9, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null)

[1016557.551178] usdm_drv: userMemAlloc:395 Unable to allocate memory slab or wrong alignment: 00000000425ceaec

[1016557.669693] usdm_drv: dev_mem_alloc:608 userMemAlloc failed

We are facing the below issue also.

[crit] 3694212#0: *188570779 SSL_do_handshake() failed (SSL: error:800650ED:lib(128):build_decrypt_op_buf:p q dmp dmq convert to fb failure) while SSL handshaking, client: <ip>, server: [::]:443
ioctl_alloc_slab:957 mmap on memory allocated through ioctl failed

Please let us know what are the reasons for these errors and how can we overcome these errors?

Re:QAT Driver Error in Nginx error log

Ronny_G_Intel — Mon, 03 Jun 2024 22:46:44 GMT

Hi SankarKumar,

Hugepage memory is used up and no 2MB slab memory neither, can be checked with `cat /proc/meminfo` and `cat /proc/buddyinfo`

Usually if check with `free -m` the buffer/cache number will be high.

I would recommend allocating enough hugepage for Nginx. If there are 32 Nginx worker processes, during reload, the maximum number of work processes will be 64.

insmod ./usdm_drv.ko max_huge_pages=N max_huge_pages_per_process=32

N should be larger than or equal to 32 * 64

Please let me know your comments.

Regards,

Ronny G

Re: Re:QAT Driver Error in Nginx error log

SankarKumar — Tue, 04 Jun 2024 10:40:36 GMT

Hi Ronny,

Thank you for your response.

In our system, we are seeing enough memory available and also we can see enough hugepage allocation for Nginx.

But, we are still facing the issues. Is there any other possible solution for overcoming the issue?

Regards,

Sankar Kumar G.

Re:QAT Driver Error in Nginx error log

Ronny_G_Intel — Tue, 04 Jun 2024 16:14:58 GMT

Hi SankarKumar,

If you are making use of hugepages? how are they configured?

Can you provide details of /proc/meminfo... or better yet monitor the memory when recreating this failure are report results?

I am looking at the software versions installed, and they are quite old. Can you try with later versions? See below:

OpenSSL 3.2 series is the latest version. You are using OpenSSL version 1.1.1 which is out of support and shouldnt be used as recommended by https://www.openssl.org/source/

QAT engine latest version is v1.6.0 https://github.com/intel/QAT_Engine/releases, you are running 0.5.42.

Also, Asynch Mode for NGINX* has been validated with the following:

OpenSSL-3.0.10

QAT engine v1.3.1

QATzip v1.1.2

You may refer to the following website: https://github.com/intel/asynch_mode_nginx

Are you able to query OpenSSL for details on the QAT Engine? ( i.e. openssl engine -t -c -vvvv qat)

Regards,

Ronny G

Re:QAT Driver Error in Nginx error log

Ronny_G_Intel — Thu, 06 Jun 2024 16:12:54 GMT

Hi SankarKumar,

Did you have a chance to check on my previous recommendations?

Regards,

Ronny G

Re:QAT Driver Error in Nginx error log

Ronny_G_Intel — Mon, 10 Jun 2024 16:55:29 GMT

Hi SankarKumar,

I hope all is well with you!

I am just following up on this request, please let me know if you have any update regarding this case; otherwise, I'm going to set this ticket to Close status. You may reopen it or create a new community post if you need further assistance.

Regards,

Ronny G

Re:QAT Driver Error in Nginx error log

Ronny_G_Intel — Tue, 11 Jun 2024 19:53:24 GMT

Hi SankarKumar,

I hope all is well with you.

I am going to set up this case to close-pending status, feel free to reopen if you still need assistance.

Regards,

Ronny G