Assistance Requested: Integrating QAT with StrongSwan on Linux Kernel - Counters Not Incrementing

[lpereira]

Hello, QAT Community,

I am currently working on integrating QAT (Quality and Acceleration Technology) with StrongSwan on the Linux kernel and have encountered a challenge that I hope you can help me overcome. We have successfully loaded the necessary modules for QAT operation, but we are facing an issue where, despite what seems to be a correct configuration, QAT is not being utilized when generating traffic through StrongSwan.

Here are the details of our current setup:

Loaded QAT Modules:

qat_api 565248 0
qat_c3xxx 20480 0
intel_qat 270336 3 qat_api,usdm_drv,qat_c3xxx
We confirmed that the modules are correctly loaded by using the lsmod | grep qat command, which returns the list of modules above.

Issue: Despite the modules being loaded, when we generate traffic expecting QAT to be utilized, we observe that the counters located in /sys/kernel/debug/qat_c3xxx_0000:01:00.0/fw_counters do not increment, indicating that QAT is not effectively being used.

[root@lucas-node2 ~]# cat /sys/kernel/debug/qat_c3xxx_0000\:01\:00.0/fw_counters
+------------------------------------------------+
| FW Statistics for Qat Device |
+------------------------------------------------+
| Firmware Requests [AE 0]: 0 |
| Firmware Responses[AE 0]: 0 |
| RAS Events [AE 0]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 1]: 0 |
| Firmware Responses[AE 1]: 0 |
| RAS Events [AE 1]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 2]: 0 |
| Firmware Responses[AE 2]: 0 |
| RAS Events [AE 2]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 3]: 0 |
| Firmware Responses[AE 3]: 0 |
| RAS Events [AE 3]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 4]: 0 |
| Firmware Responses[AE 4]: 0 |
| RAS Events [AE 4]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 5]: 0 |
| Firmware Responses[AE 5]: 0 |
| RAS Events [AE 5]: 0 |
+------------------------------------------------+
[root@lucas-node2 ~]# adf_ctl status
Checking status of all devices.
There is 1 QAT acceleration device(s) in the system:
qat_dev0 - type: c3xxx, inst_id: 0, node_id: 0, bsf: 0000:01:00.0, #accel: 3 #engines: 6 state: up
[root@lucas-node2 ~]# systemctl status qat
● qat.service - QAT service
Loaded: loaded (/usr/lib/systemd/system/qat.service; enabled; vendor preset: disabled)
Active: active (exited) since Thu 2024-03-28 17:58:08 -03; 3 days ago
Main PID: 5180 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/qat.service

Steps Already Taken:

Verification of the correct loading of QAT modules.
Review of StrongSwan configuration to ensure it is set up to use hardware acceleration.
Checking system and QAT logs for potential errors, to no avail.
I am seeking guidance, insights, or shared experiences that might help me identify and solve this issue. Any advice on how to ensure that QAT is correctly utilized with StrongSwan would be immensely appreciated.

Thank you in advance for any support you can offer!

Sincerely,

[Ronny_G_Intel]

Hi lpereira,

I would probably suggest checking QAT installation and configuration first.

How did you compile it? What compilation flag did you use when installing? Refer to https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html table 4.

Are enabling vf's? Can you run service #qat_service status and provide me the results?

You may also use the following command to check QAT status #adf_ctl status 

See https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html section 5.3.1

Finally, to check that QAT is properly functioning, you may run #cpa_sample_code

See https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html section 4.1.2

If QAT is working properly, I would recommend checking with StrongSwan, there may be different way to bind QAT engine to StrongSwan or it may be possible that QAT would only be able to accelerate specific algorithms, this would also depend on kernel configuration.

Regards,

Ronny G

[lpereira · Accepted solution]

Thank you for your help Mr. Ronny,

We made the QAT works here, with the Intel's support.

---
Hi Joel,

I hope this message finds you well. Thank so much for you help!!

Indeed, I had not applied any configure flags during the build process, which, as you correctly anticipated, resulted in the command cat /proc/crypto | grep qat yielding no output. Following your advice, I rebuilt the driver with the configure flag --enable-qat-lkcf and it worked!!

The system is now operating as intended. Data transfer through the tunnel is functioning smoothly, with the fw_counters showing incremental activity!

I cannot thank you enough for your invaluable assistance and expert guidance throughout this process. Your support was crucial in navigating these technical challenges.

Best regards,

Esta mensagem pode conter informação confidencial ou privilegiada, sendo seu sigilo protegido por lei. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor, avise imediatamente ao remetente, respondendo o e-mail e em seguida apague-a.Agradecemos sua cooperação.

This message may contain confidential or privileged information and its confidentiality is protected by law. If you are not the addressed or authorized person to receive this message, you must not use, copy, disclose or take any action based on it or any information herein. If you have received this message by mistake, please advise the sender immediately by replying the e-mail and then deleting it. Thank you for your cooperation.

De: Schuetze, Joel D 
Enviado: sexta-feira, 5 de abril de 2024 10:01
Para: Lucas Vicente Pereira 
Cc: Igor de Souza Santos 
Assunto: RE: Assistance Requested: Integrating QAT with StrongSwan on Linux Kernel

Você não costuma receber emails de Joel. Saiba por que isso é importante
Hi Lucas,

Thank-you for providing these details. Much appreciated!

From these steps, it looks like we may be missing important configure flag for the QAT driver:

Here is excerpt from the Getting Started Guide where configuration flags are presented.

Without this flag, we would not be registering crypto algorithms with LKCF.

We can verify this with this command:

cat /proc/crypto | grep qat

I suspect if we ran this command prior to reinstalling QAT driver with this flag set we would not see output from the above command.

Please let me know if this addresses the issue observed.

Cheers,

Joel

From: Lucas Vicente Pereira 
Sent: Thursday, April 4, 2024 5:26 AM

Subject: RE: Assistance Requested: Integrating QAT with StrongSwan on Linux Kernel

Dear Joel, good morning

Thank you for your response.

I am trying to offer all informations:

---

cat versionfile
PACKAGE_TYPE=QAT

PACKAGE_OS=L

PACKAGE_VERSION_MAJOR_NUMBER=4

PACKAGE_VERSION_MINOR_NUMBER=24

PACKAGE_VERSION_PATCH_NUMBER=0

PACKAGE_VERSION_BUILD_NUMBER=00005

---

We used this package: https://downloadmirror.intel.com/795697/QAT.L.4.24.0-00005.tar.gz

We compiled from:

driver_install.sh from repo https://github.com/intel/QAT_Engine.git, branch master.

--

Conf is the /etc/c3xxx_dev0.conf:

```

ServicesProfile = DEFAULT

ConfigVersion = 2

#Default values for number of concurrent requests*/
CyNumConcurrentSymRequests = 512
CyNumConcurrentAsymRequests = 64

#Statistics, valid values: 1,0
statsGeneral = 1
statsDh = 1
statsDrbg = 1
statsDsa = 1
statsEcc = 1
statsKeyGen = 1
statsDc = 1
statsLn = 1
statsPrime = 1
statsRsa = 1
statsSym = 1

# This flag is to enable device auto reset on heartbeat error
AutoResetOnError = 0

##############################################
# Kernel Instances Section
##############################################
[KERNEL]
NumberCyInstances = 1
NumberDcInstances = 0

# Crypto - Kernel instance #0
Cy0Name = "IPSec0"
Cy0IsPolled = 0
Cy0CoreAffinity = 0

##############################################
# User Process Instance Section
##############################################
[SHIM]
NumberCyInstances = 6
NumberDcInstances = 2
NumProcesses = 1
LimitDevAccess = 0

# Crypto - User instance #0
Cy0Name = "IPSEC0"
Cy0IsPolled = 1
# List of core affinities
Cy0CoreAffinity = 0

# Crypto - User instance #1
Cy1Name = "IPSEC1"
Cy1IsPolled = 1
# List of core affinities
Cy1CoreAffinity = 1

# Crypto - User instance #2
Cy2Name = "IPSEC2"
Cy2IsPolled = 1
# List of core affinities
Cy2CoreAffinity = 2

# Crypto - User instance #3
Cy3Name = "IPSEC3"
Cy3IsPolled = 1
# List of core affinities
Cy3CoreAffinity = 3

# Crypto - User instance #4
Cy4Name = "IPSEC4"
Cy4IsPolled = 1
# List of core affinities
Cy4CoreAffinity = 4

# Crypto - User instance #5
Cy5Name = "IPSEC5"
Cy5IsPolled = 1
# List of core affinities
Cy5CoreAffinity = 5

# Data Compression - User instance #0
Dc0Name = "Dc0"
Dc0IsPolled = 1
# List of core affinities
Dc0CoreAffinity = 0

# Data Compression - User instance #1
Dc1Name = "Dc1"
Dc1IsPolled = 1
# List of core affinities
Dc1CoreAffinity = 1

##############################################
# User Process Instance Section
##############################################
[SSL]
NumberCyInstances = 6
NumberDcInstances = 2
NumProcesses = 1
LimitDevAccess = 0

# Crypto - User instance #0
Cy0Name = "SSL0"
Cy0IsPolled = 1
# List of core affinities
Cy0CoreAffinity = 0

# Crypto - User instance #1
Cy1Name = "SSL1"
Cy1IsPolled = 1
# List of core affinities
Cy1CoreAffinity = 1

# Crypto - User instance #2
Cy2Name = "SSL2"
Cy2IsPolled = 1
# List of core affinities
Cy2CoreAffinity = 2

# Crypto - User instance #3
Cy3Name = "SSL3"
Cy3IsPolled = 1
# List of core affinities
Cy3CoreAffinity = 3

# Crypto - User instance #4
Cy4Name = "SSL4"
Cy4IsPolled = 1
# List of core affinities
Cy4CoreAffinity = 4

# Crypto - User instance #5
Cy5Name = "SSL5"
Cy5IsPolled = 1
# List of core affinities
Cy5CoreAffinity = 5

# Data Compression - User instance #0
Dc0Name = "Dc0"
Dc0IsPolled = 1
# List of core affinities
Dc0CoreAffinity = 0

# Data Compression - User instance #1
Dc1Name = "Dc1"
Dc1IsPolled = 1
# List of core affinities
Dc1CoreAffinity = 1

```

---

[root@lucas-node2 QAT_Engine]# adf_ctl status
Checking status of all devices.
There is 1 QAT acceleration device(s) in the system:
qat_dev0 - type: c3xxx, inst_id: 0, node_id: 0, bsf: 0000:01:00.0, #accel: 3 #engines: 6 state: up

Show status from cli "adf_ctl"

also, test cli with cpa_sample_code, when I use this command the "fw_counters" increments correctly...

---

Thank you for your help.

Sincerely,

Lucas Pereira

Chief Technology Officer

Forma

Descrição gerada automaticamente com confiança baixa Forma

Descrição gerada automaticamente com confiança baixa

Esta mensagem pode conter informação confidencial ou privilegiada, sendo seu sigilo protegido por lei. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor, avise imediatamente ao remetente, respondendo o e-mail e em seguida apague-a.Agradecemos sua cooperação.

This message may contain confidential or privileged information and its confidentiality is protected by law. If you are not the addressed or authorized person to receive this message, you must not use, copy, disclose or take any action based on it or any information herein. If you have received this message by mistake, please advise the sender immediately by replying the e-mail and then deleting it. Thank you for your cooperation.

Enviado: quarta-feira, 3 de abril de 2024 17:13
Assunto: RE: Assistance Requested: Integrating QAT with StrongSwan on Linux Kernel

Hi Lucas,

Thanks for reaching out! Can you confirm the QAT driver that is being used?

Are we using in-tree QAT kernel modules? Or were they installed from QAT package?

Thanks,

Joel

Atenciosamente,

[Ronny_G_Intel]

Hi lpereira,

I see that this issue has been resolved with Joel's help.

I will be closing this ticket but please feel free to reopen or create a new one if you still need help.

Regards,

Ronny G

By the way, I removed all personal information previously posted due to privacy concerns.