libipp_z fails a simple compresson test

[prohaskatokutek_com]

this program works with libz, fails with ipp_z, using l_ipp-samples_p-6.0.0.142 on linux, and icc 10.1.022.

#include
#include
#include
#include
#include

static void ztest(int length, int clevel, int nzeros) {
unsigned char *b = (unsigned char *) malloc(length);
assert(b);

int i;
for (i = 0; i < nzeros; i++)
b = 0;
for (; i < length; i++)
b = random();

uLongf clength = compressBound(length);
unsigned char *cb = (unsigned char *) malloc(clength);
assert(cb);

int r;
uLongf dlength = clength;
r = compress2(cb, &dlength, b, length, clevel);
assert(dlength <= clength);
assert(r == Z_OK);

unsigned char *nb = (unsigned char *) malloc(length);
assert(nb);

uLongf ulength = length;
r = uncompress(nb, &ulength, cb, dlength);
assert(r == Z_OK);
assert(ulength == (uLongf) length);
assert(memcmp(b, nb, length) == 0);

free(cb);
free(nb);
free(b);
}

int main(int argc, char *argv[]) {
ztest(120001, 1, 120001);
return 0;
}

[Sergey_K_Intel]

Hi,

Yes, this is known issue. Please, use either previous IPP's zlib sample files (of IPP version 5.3).
You can also take these two - deflate.h and deflate.c. They must be OK.
The coming (6.1) IPP version is free of this bug.

Thank you,
Sergey

[prohaskatokutek_com]

Quoting - Sergey Khlystov@Intel

Hi,

Yes, this is known issue. Please, use either previous IPP's zlib sample files (of IPP version 5.3).
You can also take these two - deflate.h and deflate.c. They must be OK.
The coming (6.1) IPP version is free of this bug.

Thank you,
Sergey

The code fixed the first bug in the library. I now have a second one. How can I get the 6.1 version of the code?
Thanks

[Vladimir_Dudnik]

IPP 6.1 is coming soon, please stay tuned. We expect it will be available sometime in Q2 this year.

Vladimir

[prohaskatokutek_com]

Quoting - Sergey Khlystov@Intel

Hi,

Yes, this is known issue. Please, use either previous IPP's zlib sample files (of IPP version 5.3).
You can also take these two - deflate.h and deflate.c. They must be OK.
The coming (6.1) IPP version is free of this bug.

Thank you,
Sergey

your code fixes the simple compression test. however, the attached program fails with the new code. the compress bound is not being honored.

thanks

#include
#include
#include
#include
#include

static void ztest(int length, int clevel, int nzeros) {
unsigned char *b = (unsigned char *) malloc(length);
assert(b);

int i;
for (i = 0; i < nzeros; i++)
b = 0;
for (; i < length; i++)
b = random();

uLongf clength = compressBound(length);
unsigned char *cb = (unsigned char *) malloc(clength);
assert(cb);

int r;
uLongf dlength = clength;
r = compress2(cb, &dlength, b, length, clevel);
assert(dlength <= clength);
assert(r == Z_OK);

unsigned char *nb = (unsigned char *) malloc(length);
assert(nb);

uLongf ulength = length;
r = uncompress(nb, &ulength, cb, dlength);
assert(r == Z_OK);
assert(ulength == length);
assert(memcmp(b, nb, length) == 0);

free(cb);
free(nb);
free(b);
}

int main(int argc, char *argv[]) {
ztest(1000000, 1, 0);
return 0;
}

[Sergey_K_Intel]

OK. It's clear now. The problem is in uncompressible data and zlib's estimation function (compressBound). For truly uncompressible data zlib uses the specific "stored block" format (i.e. output data is just a copy of input data + some header). And zlib's "compressBound" function computes that "worst case" result length.

IPP zlib function implementation does have estimation functionality via "ippsEncodeLZ77SelectHuffMode_8u" function call, but this functionality is not in zlib source code. So, IPP-ZLIB compress function uses dynamic Huffman coding for uncompressible data. As a result the amount of data produced is a bit more than original zlib's "stored block" method.

In another words, zlib "compressBound" function's computation, in general, cannot be used to predict compressed data size. I don't know the exact numbers, but very empirically compressed data length is <= 1.002*(input data size)+11 bytes.

So, if you change in your code

uLongf clength = compressBound(length);

to
uLongf clength = (uLongf)(length * 1.002 + 11);

(or something similar and more elegant), the "cb" buffer will be long enough to hold the compressed data.

Thanks and regards,
Sergey