topic Re: LZO compression/decompression in memory in Intel® Integrated Performance Primitives

LZO compression/decompression in memory

constantine-vassilev — Wed, 03 Dec 2008 21:57:21 GMT

Is it possible to LZO compress/decompress in memory?

Re: LZO compression/decompression in memory

Sergey_K_Intel — Thu, 04 Dec 2008 09:18:13 GMT

Is it possible to LZO compress/decompress in memory?

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

Re: LZO compression/decompression in memory

constantine-vassilev — Thu, 04 Dec 2008 19:18:08 GMT

Quoting - sergey_kh

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

Re: LZO compression/decompression in memory

Sergey_K_Intel — Fri, 05 Dec 2008 09:41:38 GMT

Quoting - thstart

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount, 
             Ipp8u** outDataPtr, Ipp32u* outCount)
{
    Ipp32u outBytes;
    if (inCount > outAvail) { // Output data may not fit into outbuf
        *outDataPtr = output_buffer;
        *outCount = SOME_HUGE_VALUE - outAvail;
        outAvail = SOME_HUGE_VALUE;
        outPtr = output_buffer;
        return TAKE_COMPRESSED_DATA;
    } else {
        outBytes = outAvail;
        ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
        outAvail -= outBytes;
        return GIVE_MORE;
    }
}

int main()
{
    while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
        Ipp8u* localOutputPtr;
        Ipp32u localOutputCount;
        if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
                                                            ==TAKE_COMPRESSED_DATA)
            doSomethingWithData(localOutputPtr, localOutputCount);
        else
            printf("Compress function is asking more!n");
    }
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means 
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers 
to separate compressed blocks. For example, first may come the length of next compressed block. 
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help? 
Regards,
Sergey

Re: LZO compression/decompression in memory

constantine-vassilev — Fri, 05 Dec 2008 17:35:07 GMT

Quoting - sergey_kh

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)
doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

LZO Compression Functions

This section describes Intel IPP data compression functions, that implement the LZO

(Lempel-Ziv-Oberhumer) compressed stream format. This format and algorithm use 64KB

compression dictionary and do not require additional memory for decompression. (See

original code of the LZO library at http://www.oberhumer.com.)

Constantine

Re: LZO compression/decompression in memory

Sergey_K_Intel — Mon, 08 Dec 2008 08:57:03 GMT

Quoting - thstart

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

Constantine

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

Re: LZO compression/decompression in memory

constantine-vassilev — Thu, 11 Dec 2008 01:31:49 GMT

Quoting - sergey_kh

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

Hot to do that?

Re: LZO compression/decompression in memory

Sergey_K_Intel — Thu, 11 Dec 2008 11:19:22 GMT

Quoting - thstart

Hot to do that?

Oops! I cannot do this for you )). Simply imagine how your compression routine must look like. What must it take as input? Will there be any output from it? Or, it must look like a "black-box", compressing and writing out the compressed data. What is "stream" in your interpretation? Example of routine, which digests input data and from time to time produces some compressed output is above.

Regards,
Sergey