inflate error in ipp_gzip sample code

[Kamil_Z_]

Hello

I am dealing with issue concerning gzip decompression in multi-thread mode. I use sample program provided in ipp\\interfaces\\data-compression\\ipp_gzip. When I compress attached file in multi-thread mode (chunks > 1) and fastest compression level (-1), and then try to decompress it (also in multi-thread mode) the inflation process fails and program crashes.

Position in source code: function stored_block (file ipp_gzip_inflate_mt.c),
if(state->in_left == 0) <---- this condition is satisfied
return ST_ERROR;

Could you confirm this issue?

Kamil Zukowski

[Sergey_K_Intel]

Hi Kamil,

Could you specify what command lines do you use? Exactly.

> ipp_gzip.exe -m 4 -1 test.dat

>ipp_gzip.exe -d test.dat.gz

The above lines don't show problem. As far as I remember, you don't need to specify MT mode for decompression. It starts automatically if .gz file is compressed MT and executing CPU has more than one cores.

Regards,

Sergey

[Kamil_Z_]

Hello,

I noticed there is no problem for 4 or 8 threads, the error occurs for 2, 6 or 12 threads. Command lines:
> ipp_gzip.exe -m 2 -1 test
> ipp_gzip.exe -d test.gz
output: ipp_gzip.exe: test.gz: inflate error

Kamil Zukowski

[Sergey_K_Intel]

Hi Kamil,

Thank you for notifying us about the problem. We reproduced that. It will be fixed. Meanwhile, try the following:

in the file ipp_gzip_inflate_mt.c:515-517 change in_len comparison from 4 to 3, i.e.

[cpp] if(state->in_len < 3) { if(state->in_left == 0) return ST_ERROR; [/cpp] It must start to work. Here we need to check if there no regressions after this fix.

Regards and thanks again,

Sergey

[Kamil_Z_]

Hello,

Thank you very much. This change solves the problem

Best regards,
Kamil Zukowski

[Gennady_F_Intel]

Hello Kamil,

I checked with the latest version 7.0 Update 7: compressed < -- > decompressed and compare the outputs and I had the same results:

..\Intel\Composer XE 2011 SP1\Samples\en_US\IPP\ipp-samples\data-compression\ipp_compress\bin\intel64_cl10>

>ippcompress.exe -gzip -6 test.bin test.gz

Compressed file test.bin (154163872 bytes) to file test.gz (89518047 bytes)

ippdecompress.exe test1.gz test1.rest

Decoded file test1.gz (89518047 bytes) to file test1.rest (154163872 bytes)

>md5sum.exe test.bin

58b561f75b23b493767846fdf30db648 *test.bin

>md5sum.exe test1.rest

58b561f75b23b493767846fdf30db648 *test1.rest

====

Package ID: w_ipp-samples_p_7.0.7.064

Package Contents: Intel IPP Samples for Windows*

Package ID: w_ipp-samples_p_7.0.7.064Package Contents: Intel IPP Samples for Windows*

Regards, Gennady

[Kamil_Z_]

Hello Gennady,

Thank you for response. Did you also try example in \ipp\interfaces\data-compression\ipp_gzip\ with previously proposed code modification? It is essential to reproduce this problem.

Besides ippcompress seems to be single-threaded and -6 sets an input block size, not number of chunks (threads).

Best regards
Kamil Zukowski

[Gennady_F_Intel]

Hi Kamil,

yes, it was my fault. yes, I received the same error with the latest IPP 7.0.7:

>ipp_gzip.exe -d test.bin.gz

ipp_gzip.exe: test.bin.gz: crc error in chunk 0

in the case if of modification Sergey suggested to do into the previous thread.

we will fix it asap.

--Gennady

[Kamil_Z_]

Hello Gennady,

Thank you very much. I would really appreciate it. Even if diagnosis of this issue is not trivial and you are unable to fix it at once please let me know so I will prepare a workarround in my solution.

Best regards
Kamil Zukowski

[Kamil_Z_]

Hello again,

By the way in file ipp_gzip.inflate_mt.c there is line 366:
[cpp]
vm_params.to_pos = file_info.st_size; /* Don't read last 8 bytes (4 bytes of CRC and 4 bytes of encoded file length */[/cpp]
In comment there is an information that last 8 bytes should not be read as they are CRC and file length (not compressed data) - this is clear. However variable "to_pos" has value "file_info.st_size" - total file length, it means these 8 bytes are included in inflate process after all, aren't they? Could you explain it?

Best regards,
Kamil Zukowski

[Sergey_K_Intel · Accepted solution]

Hi Kamil,

Regarding your last decoding issue, please try the following modification of lines 515-517 of ipp_gzip_inflate_mt.c

[cpp] if((state->in_len + (state->buf_len / 8)) < 4) { /* If not enough input data to get stored block header */ if(state->in_left == 0) return ST_ERROR; [/cpp]

Here, we need to take into account bit buffer length too. It must work well in both of your test cases.

On your question about suspicious comment above it looks like outdated comment, not completely correct. When we decode the chunk, the decoding process stops when last block decoding returns status equal toST_STREAM_END. So, the decision is made not on amount of data remaining, but on decoding status. You may ignore this comment, we will remove it.

Thank you for findings,

Sergey

[Kamil_Z_]

Hello Sergey,

I applied your modification and it helped for all my faulty cases. I am going to keep running tests for a while and if I find another case I will provide it.

Thank you very much for your help.

Best regards,
Kamil Zukowski