topic inflate error in ipp_gzip sample code in Intel® Integrated Performance Primitives

inflate error in ipp_gzip sample code

Kamil_Z_ — Tue, 24 Apr 2012 10:30:04 GMT

Hello

I am dealing with issue concerning gzip decompression in multi-thread mode. I use sample program provided in ipp\\interfaces\\data-compression\\ipp_gzip. When I compress attached file in multi-thread mode (chunks > 1) and fastest compression level (-1), and then try to decompress it (also in multi-thread mode) the inflation process fails and program crashes.

Position in source code: function stored_block (file ipp_gzip_inflate_mt.c),
if(state->in_left == 0) <---- this condition is satisfied
return ST_ERROR;

Could you confirm this issue?

Kamil Zukowski

inflate error in ipp_gzip sample code

Sergey_K_Intel — Tue, 24 Apr 2012 15:38:49 GMT

Hi Kamil,

Could you specify what command lines do you use? Exactly.

> ipp_gzip.exe -m 4 -1 test.dat

>ipp_gzip.exe -d test.dat.gz

The above lines don't show problem. As far as I remember, you don't need to specify MT mode for decompression. It starts automatically if .gz file is compressed MT and executing CPU has more than one cores.

Regards,

Sergey

inflate error in ipp_gzip sample code

Kamil_Z_ — Wed, 25 Apr 2012 05:14:11 GMT

Hello,

I noticed there is no problem for 4 or 8 threads, the error occurs for 2, 6 or 12 threads. Command lines:
> ipp_gzip.exe -m 2 -1 test
> ipp_gzip.exe -d test.gz
output: ipp_gzip.exe: test.gz: inflate error

Kamil Zukowski

inflate error in ipp_gzip sample code

Sergey_K_Intel — Wed, 25 Apr 2012 11:01:54 GMT

Hi Kamil,

Thank you for notifying us about the problem. We reproduced that. It will be fixed. Meanwhile, try the following:

in the file ipp_gzip_inflate_mt.c:515-517 change in_len comparison from 4 to 3, i.e.

[cpp] if(state->in_len < 3) { if(state->in_left == 0) return ST_ERROR; [/cpp] It must start to work. Here we need to check if there no regressions after this fix.

Regards and thanks again,

Sergey

inflate error in ipp_gzip sample code

Kamil_Z_ — Wed, 25 Apr 2012 11:39:13 GMT

Hello,

Thank you very much. This change solves the problem

Best regards,
Kamil Zukowski

inflate error in ipp_gzip sample code

Gennady_F_Intel — Mon, 16 Jul 2012 17:02:43 GMT

Hello Kamil,

I checked with the latest version 7.0 Update 7: compressed < -- > decompressed and compare the outputs and I had the same results:

..\Intel\Composer XE 2011 SP1\Samples\en_US\IPP\ipp-samples\data-compression\ipp_compress\bin\intel64_cl10>

>ippcompress.exe -gzip -6 test.bin test.gz

Compressed file test.bin (154163872 bytes) to file test.gz (89518047 bytes)

ippdecompress.exe test1.gz test1.rest

Decoded file test1.gz (89518047 bytes) to file test1.rest (154163872 bytes)

>md5sum.exe test.bin

58b561f75b23b493767846fdf30db648 *test.bin

>md5sum.exe test1.rest

58b561f75b23b493767846fdf30db648 *test1.rest

====

Package ID: w_ipp-samples_p_7.0.7.064

Package Contents: Intel IPP Samples for Windows*

Package ID: w_ipp-samples_p_7.0.7.064Package Contents: Intel IPP Samples for Windows*

Regards, Gennady

inflate error in ipp_gzip sample code

Kamil_Z_ — Tue, 17 Jul 2012 05:12:20 GMT

Hello Gennady,

Thank you for response. Did you also try example in \ipp\interfaces\data-compression\ipp_gzip\ with previously proposed code modification? It is essential to reproduce this problem.

Besides ippcompress seems to be single-threaded and -6 sets an input block size, not number of chunks (threads).

Best regards
Kamil Zukowski

inflate error in ipp_gzip sample code

Gennady_F_Intel — Tue, 17 Jul 2012 18:13:17 GMT

Hi Kamil,

yes, it was my fault. yes, I received the same error with the latest IPP 7.0.7:

>ipp_gzip.exe -d test.bin.gz

ipp_gzip.exe: test.bin.gz: crc error in chunk 0

in the case if of modification Sergey suggested to do into the previous thread.

we will fix it asap.

--Gennady

inflate error in ipp_gzip sample code

Kamil_Z_ — Tue, 17 Jul 2012 20:55:56 GMT

Hello Gennady,

Thank you very much. I would really appreciate it. Even if diagnosis of this issue is not trivial and you are unable to fix it at once please let me know so I will prepare a workarround in my solution.

Best regards
Kamil Zukowski

inflate error in ipp_gzip sample code

Kamil_Z_ — Wed, 18 Jul 2012 08:41:54 GMT

Hello again,

By the way in file ipp_gzip.inflate_mt.c there is line 366:
[cpp]
vm_params.to_pos = file_info.st_size; /* Don't read last 8 bytes (4 bytes of CRC and 4 bytes of encoded file length */[/cpp]
In comment there is an information that last 8 bytes should not be read as they are CRC and file length (not compressed data) - this is clear. However variable "to_pos" has value "file_info.st_size" - total file length, it means these 8 bytes are included in inflate process after all, aren't they? Could you explain it?

Best regards,
Kamil Zukowski

inflate error in ipp_gzip sample code

Sergey_K_Intel — Tue, 24 Jul 2012 13:18:00 GMT

Hi Kamil,

Regarding your last decoding issue, please try the following modification of lines 515-517 of ipp_gzip_inflate_mt.c

[cpp] if((state->in_len + (state->buf_len / 8)) < 4) { /* If not enough input data to get stored block header */ if(state->in_left == 0) return ST_ERROR; [/cpp]

Here, we need to take into account bit buffer length too. It must work well in both of your test cases.

On your question about suspicious comment above it looks like outdated comment, not completely correct. When we decode the chunk, the decoding process stops when last block decoding returns status equal toST_STREAM_END. So, the decision is made not on amount of data remaining, but on decoding status. You may ignore this comment, we will remove it.

Thank you for findings,

Sergey

inflate error in ipp_gzip sample code

Kamil_Z_ — Thu, 26 Jul 2012 05:22:32 GMT

Hello Sergey,

I applied your modification and it helped for all my faulty cases. I am going to keep running tests for a while and if I find another case I will provide it.

Thank you very much for your help.

Best regards,
Kamil Zukowski