Quick Assist QAT qzip decompress fails for a qzip compressed file

[Karl3]

Hi Intel Community,

We have an HPE DL 380 Gen11 server with two Intel Xeon Platinum 8462y processors.

https://www.intel.com/content/www/us/en/products/sku/232383/intel-xeon-platinum-8462y-processor-60m-cache-2-80-ghz/specifications.html

We have been using the Intel Quick Assist (QAT) qzip (QAT zip) command to compress large backup files. 

We have been compressing 12 X 275 GB database backup files in parallel. This usually completes as 12 compresses in about 24 minutes.

Occasionally (approx 1 time in 20) the compression command creates a compressed file that can not be decompressed by qzip (QAT zip). The compressed file appears to be a valid archive because it can be decompressed by gzip (GNU zip) which results in a decompressed file that is identical to the original file.

The qzip (QAT zip) compress command is as follows:

qzip -k database.bak.5 -O gzipext -A deflate -L 6 > database.bak.5.qzip.comp.log 2> database.bak.5.qzip.comp.err

This creates a database.bak.5.gz file.

The qzip (QAT zip) decompress command is as follows:

qzip -d -k database.bak.5.gz -O gzipext -A deflate > database.bak.5.qzip.decomp.log 2> database.bak.5.qzip.decomp.err

This is the error is generated by this decompress command.

doProcessBuffer:Decompression failed with error: -2
Process file error: -2

This error happens consistently for this the database.bak.5.gz file. It results in a truncated version of the original database.bak.5 file.

Note: That we have successfully compressed and decompressed the exact same database.bak.5 file 19 times successfully.
Note: That the database.bak.5.gz file can be decompressed by gzip (GNU zip) which results in the exact same file as the original.
Note: We also have a RHEL 8.8 server were we see the same issue.

The processor formware is Server Platform Services (SPS) Firmware 6.0.4.75.0
The RHEL kernel is 4.18.0-513.18.1.el8_9.x86_64.

The Quick Assist packages are:

# rpm -qa | grep linux-firmware
linux-firmware-20230824-120.git0e048b06.el8_9.noarch

#rpm -qa | grep qat | sort
qatlib-23.02.0-2.el8_8.x86_64
qatlib-service-23.02.0-2.el8_8.x86_64
qatzip-1.1.2-1.el8_8.x86_64
#

What does this error mean? Is this a known issue? Do you have any advise on resolving this issue?

Any advice would be greatly appreciated.

Regards, Karl

[Ronny_G_Intel]

Hi Karl,

I am currently looking into this issue but I will need some extra time to do some more research about this error when executing the decompress command:

doProcessBuffer:Decompression failed with error: -2

Process file error: -2

I checked our internal records and found no previous reports, and the fact that it happens approximately 1 time in 20 attempts makes it very difficult to replicate and troubleshoot.

I would assume that you using QATzip (https://github.com/intel/QATzip) and that you followed installation instructions and any applicable recommendations, please confirm and provide any other pertinent detail.

I will get back to you as soon as possible.

Regards,

Ronny G

[Karl3]

Hi Ronny, 

Thanks for responding to my post.

I used the qzip executable as provided through the Red Hat repositories.

I just installed and executed the qzip command like any other executable provided by Red Hat.

# yum install linux-firmware

# yum install qatlib qatzip

We have some compressed files that consistently fail with the error - however, the smallest file is 143 GB. I tried reproducing with smaller files but I couldn't reproduce the issue.

Regards, Karl

[Ronny_G_Intel]

Hi Karl3,

Can you plaese check what version of QATzip are you using?
I know it may be difficult to test, but can you test separately with the latest QATzip from GitHub? Here is the URL: https://github.com/intel/QATzip

On the other hand, would you be willing to provide us with the file that cannot be decompressed? With Compress and Verify (CnV) this should not be possible as we are decompressing each block after compressing to ensure we can decompress the data. It would also be interesting to see if see any CnV errors being logged. We can check this with command similar to: cat /sys/kernel/debug/qat_4xxx_0000\:09\:00.0/cnv_errors (update the BDF to align with QAT endpoint on your system).

Thanks,
Ronny G

[Karl3]

Hi Ronny G, 

We have RHEL 8.8 and RHEL 8.9 installed and the QATzip version is v1.1.2 on both.

$ qzip -V
qzip v1.1.2
Copyright (C) 2021 Intel Corporation.

$

I have a case open with Red Hat. They have provided a copy of the RHEL 9.4.0 beta qatlib/qatlib-service/qatzip packages to install on RHEL8 to try out. qzip -V also returns v1.1.2 for this installation. These RHEL 9.4.0 beta packages also produce the same error. 

I have had a look at the QATzip instructions from Git Hub. They look complex and I don't have the expertise to work out what is applicable on a RHEL8 system. I will ask Red Hat support to review and see if they can provide any guidance. 

I checked the /sys/kernel/debug/qat_4xxx_0000* directories on the server and I don't see any cnv_errors files. Under that directory, I just see dev_cfg and config files.

Regarding the file that can not be decompressed - is it possible for you to provide something to gather diagnostics from the file rather than uploading the file? 

Regards, Karl

[Ronny_G_Intel]

Hi Karl,

Thanks for the additional details and I understand that you dont want try QATzip from GitHub.

Allow me a couple of days to do some more research and I will get back to you.

Thanks,

Ronny G

[Karl3]

Hi Ronny G, 

Are you able to direct message me?

Regards, Karl

[Karl3]

Hi Ronny G,

We can upload an example file that fails to decompress. The file will be 135 GB.

Please direct message me with secure SFTP credentials for this case so I can upload the file.

Regards, Karl

[Ronny_G_Intel]

Hi Karl,

We can do email or web ticketing (similar to direct message).

I will send you a quick note to initiative conversation.

Thanks,

Ronny 

[Ronny_G_Intel]

This is great. Thank you.

Let me find the best way to receive your upload. I will get back to you soon.

Thanks,

Ronny G