QAT Enabled but operating at low speed - Proxmox

[cwest]

Hi,

I've installed Intel's QAT engine via the PPA on an Ubuntu 22.04 FIPS-enabled VM, and speed tests via OpenSSL do not seem to be accelerated properly. Results:

fips:~# openssl speed -elapsed rsa2048
...
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000742s 0.000023s   1348.0  42954.6

fips:~# openssl speed -engine qatengine -elapsed rsa2048
QAT_SW - Processor unsupported: AVX512F = 0, AVX2 = 0, VAES = 0, VPCLMULQDQ = 0
Engine "qatengine" set.
...
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000798s 0.000023s   1253.0  42592.5

These should be hardware tests, so QAT_SW does not worry me. What does is that the speeds are virtually identical.

The card I'm using appears to have been passed through correctly, and show up as registered with drivers and what appear to be the appropriate modules:

fips:~# lspci -nnk | grep -A 3 8086:37c8
00:10.0 Co-processor [0b40]: Intel Corporation C62x Chipset QuickAssist Technology [8086:37c8] (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8970 [8086:0002]
	Kernel driver in use: c6xx
	Kernel modules: qat_c62x
00:11.0 Co-processor [0b40]: Intel Corporation C62x Chipset QuickAssist Technology [8086:37c8] (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8970 [8086:0002]
	Kernel driver in use: c6xx
	Kernel modules: qat_c62x
00:1b.0 Co-processor [0b40]: Intel Corporation C62x Chipset QuickAssist Technology [8086:37c8] (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8970 [8086:0002]
	Kernel driver in use: c6xx
	Kernel modules: qat_c62x

 However, QAT's system service fails to start, with the message

Oct 03 19:05:47 tls-proxy-fips systemd[1]: Starting QAT service...
Oct 03 19:05:47 tls-proxy-fips qat_init[6736]: lspci: No PFs found, so assume qatlib is running on a VM
Oct 03 19:05:47 tls-proxy-fips qatmgr[6808]: No devices found
Oct 03 19:05:47 tls-proxy-fips qatmgr[6806]: No QAT device found
Oct 03 19:05:47 tls-proxy-fips systemd[1]: qat.service: Control process exited, code=exited, status=1/FAILURE

 and checking if the engine is enabled seems to succeed, but note the errors at the bottom (perhaps related to https://github.com/openssl/openssl/commit/d6bf4a2218aeb246ba7d34f02e895c37569c8265

fips:~# openssl engine -c -t -vvv qatengine

QAT_SW - Processor unsupported: AVX512F = 0, AVX2 = 0, VAES = 0, VPCLMULQDQ = 0
(qatengine) Reference implementation of QAT cr.ypto engine(qat_hw & qat_sw) v2.0.0
 [RSA, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, id-aes128-CCM, id-aes192-CCM, id-aes256-CCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2]
     [ available ]
     ENABLE_EXTERNAL_POLLING: Enables the external polling interface to the engine.
          (input flags): NO_INPUT
     POLL: Polls the engine for any completed requests
          (input flags): NO_INPUT
 ...
     SW_ALGO_BITMAP: Set the SW algorithm bitmap and reload the algorithm registration
          (input flags): STRING
40B7BE06D07F0000:error:1280006A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:../cr.ypto/dso/dso_dlfcn.c:188:symname(EVP_PKEY_base_id): /usr/lib/x86_64-linux-gnu/engines-3/qatengine.so: undefined symbol: EVP_PKEY_base_id
40B7BE06D07F0000:error:1280006A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:../cr.ypto/dso/dso_lib.c:176:

 I'm a little bit at a loss for what to do. Is the engine enabled? Not enabled? What should be done to get it into a usable state?

(Note: I have edited the occurrence of a certain apparently-forbidden word to cr.ypto.)

[DiegoV_Intel]

Hi,

If the QAT service fails to start you are definitely not using the QAT devices in those tests.

I recommend before jumping to QAT_Engine, to test first that QAT is working properly. The first question is which QAT driver you are using? From the above outputs, I see you are using QAT hardware c62x, therefore the driver to use is the one available at Intel® QuickAssist Technology (Intel® QAT) Driver for Linux* for Customer Enabling (CE) Release.

Knowing which is the right Intel QAT driver to use may be confusing as there are several versions available depending on the hardware version. This article has a brief summary of the available options: Which Intel® QuickAssist Technology (Intel® QAT) Driver Should Be Used Based on the Platform?.

Once you have installed the right driver, it's time to check if the sample application runs. If it runs, it means the QAT service is up and running. You can learn more about how to run the sample application in the Intel® QuickAssist Technology (Intel® QAT) Getting Started Guide for Linux* for Customer Enabling (CE) Release.

Lastly, if all looks good with the sample application, then you can proceed with QAT_Engine. The setup instructions are available in the GitHub repo Intel® QuickAssist Technology(QAT) OpenSSL* Engine.

Note that the way to setup QAT_Engine also depends on which hardware version you are using. From your description, I think you were following the setup process for QAT hardware 4xxx but yours is c62x, so you would have to install QAT_Engine from source.

Hope all this is clear. Let me know if any doubts.

Regards,
Diego V.

[cwest]

Hi,

I went through the installation instructions for the driver, service, and engine, and successfully built and installed the driver and ran the test application (reported something around 87 Gbps for AES). The service also now runs and appears to correctly configure the device. Using system-supplied OpenSSL, I can activate qatengine and run speed tests against it, but speeds are exactly the same as described above (despite dmesg giving a clear indication that I am in fact activating the card -- if I kill the speed test before it finishes, the kernel driver complains about orphan rings). What else can I do to test that the driver is actually being used?