Intel® Integrated Performance Primitives
Deliberate problems developing high-performance vision, signal, security, and storage applications.

IPP ZLIB sample issue

andrei_m
Beginner
1,872 Views

Hello!

Our windows application uses libpng to decode PNG files, libpng in turn uses zlib library for decoding. Currently we are evaluating the IPP library to test if there will be any speed improvements. So I have downloaded your 5.3 IPP_ZLIB sample and replaced our original zlib with the optimized one and it seems to be failing on decoding. libpng retuns "Not enough image data" error. Further brief investigation showed that there is an issue with the inflate() function in zlib. Project was compiled on VS 2005 without openmp flag.

Unfortunately I haven't done any deep investigation, maybe somebody have expirienced the same issue and have a solution?

Thanks,
Andrei.

0 Kudos
1 Solution
Vladimir_Dudnik
Employee
1,805 Views
What version of IPP do you use? We have fixed several issues in IPP 5.3 update 3 data compression sample. Note, thenew version of IPP 6.0 was just released
Regards,
Vladimir

View solution in original post

0 Kudos
31 Replies
Vladimir_Dudnik
Employee
1,367 Views
Thanks for reporting that issue. Could you please provide simple test case to reproduce that? We do test ipp zlib with libnpg and did not see the issue, of course it may depned on parameters used.
Regards,
Vladimir

0 Kudos
andrei_m
Beginner
1,367 Views
Hi Vladimir,

Thank you for your response. Basically, I use an example provided with the libpng library, with one exception:

[cpp]png_transforms = PNG_TRANSFORM_BGR | PNG_TRANSFORM_STRIP_16 | PNG_TRANSFORM_EXPAND;
png_read_png(png_ptr, info_ptr, png_transforms, png_voidp_NULL);[/cpp]

There is another thing, I use custom reader, which basically reads the image from memory and it works with non-optimized zlib, so it's not the issue.

I also attached an image which failed the decoding.

Thanks,
Andrei.

0 Kudos
Vladimir_Dudnik
Employee
1,806 Views
What version of IPP do you use? We have fixed several issues in IPP 5.3 update 3 data compression sample. Note, thenew version of IPP 6.0 was just released
Regards,
Vladimir

0 Kudos
andrei_m
Beginner
1,367 Views

Hi Vladimir,

I was using IPP 5.3 update 4 (to be exact "w_ipp_ia32_p_5.3.4.087.exe"). Anyway I will try out the new IPP 6.0 version and check if it solves my problem.

Thanks,
Andrei.

0 Kudos
andrei_m
Beginner
1,367 Views

Upgrading to IPP 6.0 solved my problem.

Thanks.

0 Kudos
Vladimir_Dudnik
Employee
1,367 Views
Great, thanks for updating on this!
Regards,
Vladimir

0 Kudos
xiazhou_cnd
Beginner
1,367 Views

The newly released IPP Zlib (6.0.0.142) is much better than the previous one (6.0.0.126), but still has serious bugs. For example, the Lines from 769 to 771:

s->in_tmpbuff_next = pSrc;
s->in_tmpbuff_rem = ((ippflush != IppLZ77FinishFlush)? s->in_tmpbuff_avail : 0) + srcLen;
s->in_tmpbuff_avail = chslen;

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

0 Kudos
Sergey_K_Intel
Employee
1,367 Views
Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
Quoting - sergey_kh
Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
My code "zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen)" is not to copyuser's buffer to temporary buffer. In fact, it is to move the data that are left in the temporary buffer without being processed by ippsEncodeLZ77_8u function. Please note that pSrc here is the temporary buffer pointer that is increased by number of chars really processed by ippsEncodeLZ77_8u function, andsrcLen is the number of data left in the temporary storage. If we don't move them to the beginning position of the temporary buffer (which is the position that in_tmpbuff_st_ptr points to), how those data will be flushed out if ippsEncodeLZ77SelectHuffMode function chooses "stored block" as the best compression method, or be processed by other chosen method?

When IPP Zlib is plugged into my web optimization application, it will produce wrong encoding every 2-3 hours. It is hard to re-produce such bug in a standalone mode.

By the way, when the new ZLIB functions of IPP Data Compression is to be changed and released?

Quoting - xiazhou_cnd
Quoting - sergey_kh
Quoting - xiazhou_cnd

should be replaced by:

zmemcpy_ipp(s->in_tmpbuff_ptr, pSrc, srcLen);
s->in_tmpbuff_next = s->in_tmpbuff_ptr + srcLen;
s->in_tmpbuff_rem = srcLen;
s->in_tmpbuff_avail = s->in_tmpbuff_size - srcLen;

Please let me know if I am wrong.

Hi! If you have sample data where the code fails, please provide us with this.

Regarding this particular piece, I would doubt.

First, we don't need to copy user's buffer to temporary storage, because it has been already done in lines 728-738. Then, temporary buffer pointers are increased by number of chars really processed by ippsEncodeLZ77_8u function (line 748). Only these chars will be flushed out if ippsEncodeLZ77SelectHuffMode function will choose "stored block" as the best compression method.

Anyway, many thanks for your observations, it helps us stay slim.

Note, that in future releases of IPP ZLIB sample this code will be removed, since the ZLIB functions of IPP Data Compression will be changed and therefore ZLIB sample will be changed too. Thus, ZLIB will stay original with only few lines changed. It is planned for IPP 6.1 Beta release.

Regards,
Sergey

0 Kudos
Vladimir_Dudnik
Employee
1,367 Views

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
Thank you very much for the information.

There are other bugs in the IPP Zlib also, besides the one I pointed out in the previous email. For example, in the lines 808 to 812:

if (ippflush == IppLZ77FinishFlush && s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, IppLZ77NoFlush, pLZ77State );
} else {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, ippflush, pLZ77State );
}

should be replaced by:

if (s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_rem, &pDst, &dstLen, ippflush==IppLZ77FinishFlush?IppLZ77NoFlush:ippflush, pLZ77State );
}

In lines 750 to 751:

if( (s->wrap == 1) && (chslen != 0) )
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );

should be replaced by:
if( chslen != 0 ) {
#ifdef GZIP
if(s->wrap == 2)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
else
#endif
if(s->wrap == 1)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
}
And so on.
Quoting - Vladimir Dudnik

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
Sorry there is a typo in my previous email. The code should be

if( chslen != 0 ) {
#ifdef GZIP
if(s->wrap == 2)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
else
#endif
if(s->wrap == 1)
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );
}


Quoting - xiazhou_cnd
Thank you very much for the information.

There are other bugs in the IPP Zlib also, besides the one I pointed out in the previous email. For example, in the lines 808 to 812:

if (ippflush == IppLZ77FinishFlush && s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, IppLZ77NoFlush, pLZ77State );
} else {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_avail, &pDst, &dstLen, ippflush, pLZ77State );
}

should be replaced by:

if (s->in_tmpbuff_rem) {
status = ippsEncodeLZ77StoredBlock_8u( &s->in_tmpbuff_st_ptr, (int*)&s->in_tmpbuff_rem, &pDst, &dstLen, ippflush==IppLZ77FinishFlush?IppLZ77NoFlush:ippflush, pLZ77State );
}

In lines 750 to 751:

if( (s->wrap == 1) && (chslen != 0) )
strm->adler = adler32( strm->adler, s->in_tmpbuff_ptr, chslen );

should be replaced by:
if( chslen != 0 ) {
#ifdef GZIP
if(s->wrap == 2)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
else
#endif
if(s->wrap == 1)
strm->adler = crc32( strm->adler, s->in_tmpbuff_ptr, chslen );
}
And so on.
Quoting - Vladimir Dudnik

Hello,

we do not have a firm data for the next IPP release yet. Basically it might be sometime in H1'09

Regards,
Vladimir

0 Kudos
xiazhou_cnd
Beginner
1,367 Views

I have the similar question about IPP ijg (Jpeg module). Is there any plan to re-write it in the near future?

0 Kudos
Sergey_K_Intel
Employee
1,367 Views
Quoting - xiazhou_cnd
When IPP Zlib is plugged into my web optimization application, it will produce wrong encoding every 2-3 hours. It is hard to re-produce such bug in a standalone mode.


In order to pin down the problem, please, try "deflate.c" from IPP 5.3 (attached). If you problem will have gone, the origin is definitely in "estimate" functionalityof IPP sample. We'll be thinking of what can be done. Though, cannot promise the solution, since IPP zlib sampleis totally redesigned (i.e. is returned back to its roots) in upcoming IPP 6.1 release. It's not worth spending resources on the solution.

Regards,
Sergey

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
Thanks for the reply.

One more question:

After calling ippsEncodeLZ77Reset_8u function, do we still need to call ippsEncodeLZ77Init_8u, in order to re-use the struct IppLZ77State_8u?

Quoting - sergey_kh
Quoting - xiazhou_cnd
In order to pin down the problem, please, try "deflate.c" from IPP 5.3 (attached). If you problem will have gone, the origin is definitely in "estimate" functionality of IPP sample. We'll be thinking of what can be done. Though, cannot promise the solution, since IPP zlib sample is totally redesigned (i.e. is returned back to its roots) in upcoming IPP 6.1 release. It's not worth spending resources on the solution.

Regards,
Sergey

0 Kudos
Vladimir_Dudnik
Employee
1,367 Views
What are the reasons do you see to re write IJG (libjpeg) which use IPP functions? Do you have any issues with IPP IJG sample?
Regards,
Vladimir

0 Kudos
xiazhou_cnd
Beginner
1,367 Views
I've tested IPP IJG by the standalone program provided by your samples. It works fine and I am to try to see whether it's working in my web optimization environment. If it is to be re-written soon, I'll wait.

I am in the process to re-write IPP Zlib by myself, so I have a question here:

After calling ippsEncodeLZ77Reset_8u function, do we still need to call ippsEncodeLZ77Init_8u, in order to re-use the struct IppLZ77State_8u?

Thanks!

Quoting - Vladimir Dudnik
What are the reasons do you see to re write IJG (libjpeg) which use IPP functions? Do you have any issues with IPP IJG sample?
Regards,
Vladimir

0 Kudos
Sergey_K_Intel
Employee
1,367 Views

You don't need to call "Init" after "Reset". After resetting LZ77 structure is usable with no additional efforts. As long as you aren't planning to change important things as compression level or checksum computing method. For that you need either another structure or re-initialize the old.

Regarding "I am in the process to re-write IPP Zlib by myself...". IPP ZLIB functions are redesigned so, that almost no changes will require to be done over original ZLIB code. So, if you are going to use IPP ZLIB functions, you'd better implement your application to work with original ZLIB. Then "couple lines out/couple lines in" and your application works with IPP ZLIB.

Regards,
Sergey

0 Kudos
xiazhou_cnd
Beginner
1,171 Views
Now I'veintegrated IPP ijg intomy own standalone test program and got corrupted images (the bottom part of images are corrupted). I haven't changed my testing code, but added a call of ippStaticInit() to initialize IPP. If I undefine USE_IPP in jconfig.h, the results are correct, while if Idefine USE_IPP, the results are wrong.

So my conclusion is either IPP ijg has bugs or its interfaces don't confirm the original ijg's.

Please let me know your comments.
0 Kudos
Reply