IPP ZLIB sample issue

[andrei_m]

Hello!

Our windows application uses libpng to decode PNG files, libpng in turn uses zlib library for decoding. Currently we are evaluating the IPP library to test if there will be any speed improvements. So I have downloaded your 5.3 IPP_ZLIB sample and replaced our original zlib with the optimized one and it seems to be failing on decoding. libpng retuns "Not enough image data" error. Further brief investigation showed that there is an issue with the inflate() function in zlib. Project was compiled on VS 2005 without openmp flag.

Unfortunately I haven't done any deep investigation, maybe somebody have expirienced the same issue and have a solution?

Thanks,
Andrei.

[Vladimir_Dudnik]

Thanks for reporting that issue. Could you please provide simple test case to reproduce that? We do test ipp zlib with libnpg and did not see the issue, of course it may depned on parameters used.

Regards,
Vladimir

[andrei_m]

Hi Vladimir,

Thank you for your response. Basically, I use an example provided with the libpng library, with one exception:

[cpp]png_transforms = PNG_TRANSFORM_BGR | PNG_TRANSFORM_STRIP_16 | PNG_TRANSFORM_EXPAND;
png_read_png(png_ptr, info_ptr, png_transforms, png_voidp_NULL);[/cpp]

There is another thing, I use custom reader, which basically reads the image from memory and it works with non-optimized zlib, so it's not the issue.

I also attached an image which failed the decoding.

Thanks,
Andrei.

[Vladimir_Dudnik · Accepted solution]

What version of IPP do you use? We have fixed several issues in IPP 5.3 update 3 data compression sample. Note, thenew version of IPP 6.0 was just released

Regards,
Vladimir

[andrei_m]

Hi Vladimir,

I was using IPP 5.3 update 4 (to be exact "w_ipp_ia32_p_5.3.4.087.exe"). Anyway I will try out the new IPP 6.0 version and check if it solves my problem.

Thanks,
Andrei.

[andrei_m]

Upgrading to IPP 6.0 solved my problem.

Thanks.

[Vladimir_Dudnik]

Great, thanks for updating on this!

Regards,
Vladimir

[xiazhou_cnd]

The newly released IPP Zlib (6.0.0.142) is much better than the previous one (6.0.0.126), but still has serious bugs. For example, the Lines from 769 to 771:

s->in_tmpbuff_next = pSrc;
s->in_tmpbuff_rem = ((ippflush != IppLZ77FinishFlush)? s->in_tmpbuff_avail : 0) + srcLen;
s->in_tmpbuff_avail = chslen;

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

[Sergey_K_Intel]

Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

[xiazhou_cnd]

Quoting - sergey_kh

Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

[xiazhou_cnd]

My code "zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen)" is not to copyuser's buffer to temporary buffer. In fact, it is to move the data that are left in the temporary buffer without being processed by ippsEncodeLZ77_8u function. Please note that pSrc here is the temporary buffer pointer that is increased by number of chars really processed by ippsEncodeLZ77_8u function, andsrcLen is the number of data left in the temporary storage. If we don't move them to the beginning position of the temporary buffer (which is the position that in_tmpbuff_st_ptr points to), how those data will be flushed out if ippsEncodeLZ77SelectHuffMode function chooses "stored block" as the best compression method, or be processed by other chosen method?

When IPP Zlib is plugged into my web optimization application, it will produce wrong encoding every 2-3 hours. It is hard to re-produce such bug in a standalone mode.

By the way, when the new ZLIB functions of IPP Data Compression is to be changed and released?

Quoting - xiazhou_cnd

Quoting - sergey_kh

Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

[Vladimir_Dudnik]

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

[xiazhou_cnd]

Thank you very much for the information.

There are other bugs in the IPP Zlib also, besides the one I pointed out in the previous email. For example, in the lines 808 to 812:

if (ippflush == IppLZ77FinishFlush && s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, IppLZ77NoFlush, pLZ77State );
} else {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, ippflush, pLZ77State );
}

should be replaced by:

if (s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_rem, &pDst, &dstLen, ippflush==IppLZ77FinishFlush?IppLZ77NoFlush:ippflush, pLZ77State );
}

In lines 750 to 751:

if( (s->wrap == 1) && (chslen != 0) )
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );

should be replaced by:

if( chslen != 0 ) {

#ifdef GZIP

if(s->wrap == 2)

strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );

else

#endif

if(s->wrap == 1)

strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );

}

And so on.

Quoting - Vladimir Dudnik

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

[xiazhou_cnd]

Sorry there is a typo in my previous email. The code should be

if( chslen != 0 ) {
#ifdef GZIP
if(s->wrap == 2)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
else
#endif
if(s->wrap == 1)
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );
}

Quoting - xiazhou_cnd

Thank you very much for the information.

There are other bugs in the IPP Zlib also, besides the one I pointed out in the previous email. For example, in the lines 808 to 812:

if (ippflush == IppLZ77FinishFlush && s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, IppLZ77NoFlush, pLZ77State );
} else {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, ippflush, pLZ77State );
}

should be replaced by:

if (s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_rem, &pDst, &dstLen, ippflush==IppLZ77FinishFlush?IppLZ77NoFlush:ippflush, pLZ77State );
}

In lines 750 to 751:

if( (s->wrap == 1) && (chslen != 0) )
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );

should be replaced by:

if( chslen != 0 ) {

#ifdef GZIP

if(s->wrap == 2)

strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );

else

#endif

if(s->wrap == 1)

strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );

}

And so on.

Quoting - Vladimir Dudnik

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

[xiazhou_cnd]

I have the similar question about IPP ijg (Jpeg module). Is there any plan to re-write it in the near future?

[Sergey_K_Intel]

Quoting - xiazhou_cnd

When IPP Zlib is plugged into my web optimization application, it will produce wrong encoding every 2-3 hours. It is hard to re-produce such bug in a standalone mode.

In order to pin down the problem, please, try "deflate.c" from IPP 5.3 (attached). If you problem will have gone, the origin is definitely in "estimate" functionalityof IPP sample. We'll be thinking of what can be done. Though, cannot promise the solution, since IPP zlib sampleis totally redesigned (i.e. is returned back to its roots) in upcoming IPP 6.1 release. It's not worth spending resources on the solution.

Regards,
Sergey

[xiazhou_cnd]

Thanks for the reply.

One more question:

After calling ippsEncodeLZ77Reset_8u function, do we still need to call ippsEncodeLZ77Init_8u, in order to re-use the struct IppLZ77State_8u?

Quoting - sergey_kh

Quoting - xiazhou_cnd

In order to pin down the problem, please, try "deflate.c" from IPP 5.3 (attached). If you problem will have gone, the origin is definitely in "estimate" functionality of IPP sample. We'll be thinking of what can be done. Though, cannot promise the solution, since IPP zlib sample is totally redesigned (i.e. is returned back to its roots) in upcoming IPP 6.1 release. It's not worth spending resources on the solution.

Regards,
Sergey

[Vladimir_Dudnik]

What are the reasons do you see to re write IJG (libjpeg) which use IPP functions? Do you have any issues with IPP IJG sample?

Regards,
Vladimir

[xiazhou_cnd]

I've tested IPP IJG by the standalone program provided by your samples. It works fine and I am to try to see whether it's working in my web optimization environment. If it is to be re-written soon, I'll wait.

I am in the process to re-write IPP Zlib by myself, so I have a question here:

After calling ippsEncodeLZ77Reset_8u function, do we still need to call ippsEncodeLZ77Init_8u, in order to re-use the struct IppLZ77State_8u?

Thanks!

Quoting - Vladimir Dudnik

What are the reasons do you see to re write IJG (libjpeg) which use IPP functions? Do you have any issues with IPP IJG sample?

Regards,
Vladimir

[Sergey_K_Intel]

You don't need to call "Init" after "Reset". After resetting LZ77 structure is usable with no additional efforts. As long as you aren't planning to change important things as compression level or checksum computing method. For that you need either another structure or re-initialize the old.

Regarding "I am in the process to re-write IPP Zlib by myself...". IPP ZLIB functions are redesigned so, that almost no changes will require to be done over original ZLIB code. So, if you are going to use IPP ZLIB functions, you'd better implement your application to work with original ZLIB. Then "couple lines out/couple lines in" and your application works with IPP ZLIB.

Regards,
Sergey

[xiazhou_cnd]

Now I'veintegrated IPP ijg intomy own standalone test program and got corrupted images (the bottom part of images are corrupted). I haven't changed my testing code, but added a call of ippStaticInit() to initialize IPP. If I undefine USE_IPP in jconfig.h, the results are correct, while if Idefine USE_IPP, the results are wrong.

So my conclusion is either IPP ijg has bugs or its interfaces don't confirm the original ijg's.

Please let me know your comments.