Drop In Data Compression Code

[Joseph_S_Intel]

Some of you may be interested in the data compression technologies offered by IntelIPP. In particular Intel IPP offers highly optimized versions of zlib, gzip and bzip2/libbz2. The Intel IPPzlib and libbz2 implementations have identical APIs to the open source verisons and can be used without modification in your code. The IPP bzip2 and gzip command line syntax is compatible with the open source versions. You can find the IPP implementations in the Intel IPP data compression samples. In addition ipp_zlib and ipp_gzip were redesigned in Intel IPP 6.1 for improved performance and better zlib integration.Ipp_libbz2 and ipp_gzip are threaded and can provide dramatically better performance for compression and decompression than the open source versions. Intel IPP 6.1 update 2 even offers pre-built libraries for ease of adoption. Comments?

[Mikael_Grev]

Hello Walter,

One thing that would be useful is a table with the different codecs, their compression rate for common file types and their relative and absolute speed. I think that would save a lot of double work.

Also, I think you should license the QuickLZ (quicklz.com) algorithm. It is so much faster than anything you've got that it would be worth while. It would be targeted for compressed pipes and such since the format isn't standard. I use it for compressing lossless images in a VNC-like setup and it works like a charm. I am in no way affiliated with the codec btw, I just use it. You might want to benchmark it and compare. It is not threaded though so you can get even more out of it (especially with a decent threading algorithm and not just the simple IPP one). And maybe you can even make it faster per thread with your performance knowlege within Intel.

Cheers,
Mikael

[mwrighton]

Quoting - Walter Shands (Intel)

Some of you may be interested in the data compression technologies offered by IntelIPP. In particular Intel IPP offers highly optimized versions of zlib, gzip and bzip2/libbz2. The Intel IPPzlib and libbz2 implementations have identical APIs to the open source verisons and can be used without modification in your code. The IPP bzip2 and gzip command line syntax is compatible with the open source versions. You can find the IPP implementations in the Intel IPP data compression samples. In addition ipp_zlib and ipp_gzip were redesigned in Intel IPP 6.1 for improved performance and better zlib integration.Ipp_libbz2 and ipp_gzip are threaded and can provide dramatically better performance for compression and decompression than the open source versions. Intel IPP 6.1 update 2 even offers pre-built libraries for ease of adoption. Comments?

I've integrated this into our code. We are now seeing valgrind complaints about UMRs andoverlapping memcpy's. It was clean before, with the 'stock' zlib distribution.

Did anyone run valgrind/purify before releasing this?

[Joseph_S_Intel]

Which version of Intel IPP are you using? A number of fixes related to UMRs and overlapping memcpy's have been fixed starting with Intel IPP 6.1.5.