LZO compression/decompression in memory

[constantine-vassilev]

Is it possible to LZO compress/decompress in memory?

[Sergey_K_Intel]

Quoting - thstart

Is it possible to LZO compress/decompress in memory?

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

[constantine-vassilev]

Quoting - sergey_kh

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

[Sergey_K_Intel]

Quoting - thstart

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount, 
             Ipp8u** outDataPtr, Ipp32u* outCount)
{
    Ipp32u outBytes;
    if (inCount > outAvail) { // Output data may not fit into outbuf
        *outDataPtr = output_buffer;
        *outCount = SOME_HUGE_VALUE - outAvail;
        outAvail = SOME_HUGE_VALUE;
        outPtr = output_buffer;
        return TAKE_COMPRESSED_DATA;
    } else {
        outBytes = outAvail;
        ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
        outAvail -= outBytes;
        return GIVE_MORE;
    }
}

int main()
{
    while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
        Ipp8u* localOutputPtr;
        Ipp32u localOutputCount;
        if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
                                                            ==TAKE_COMPRESSED_DATA)
            doSomethingWithData(localOutputPtr, localOutputCount);
        else
            printf("Compress function is asking more!n");
    }
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means 
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers 
to separate compressed blocks. For example, first may come the length of next compressed block. 
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help? 
Regards,
Sergey

[constantine-vassilev]

Quoting - sergey_kh

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)
doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

LZO Compression Functions

This section describes Intel IPP data compression functions, that implement the LZO

(Lempel-Ziv-Oberhumer) compressed stream format. This format and algorithm use 64KB

compression dictionary and do not require additional memory for decompression. (See

original code of the LZO library at http://www.oberhumer.com.)

Constantine

[Sergey_K_Intel]

Quoting - thstart

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

Constantine

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

[constantine-vassilev]

Quoting - sergey_kh

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

Hot to do that?

[Sergey_K_Intel]

Quoting - thstart

Hot to do that?

Oops! I cannot do this for you )). Simply imagine how your compression routine must look like. What must it take as input? Will there be any output from it? Or, it must look like a "black-box", compressing and writing out the compressed data. What is "stream" in your interpretation? Example of routine, which digests input data and from time to time produces some compressed output is above.

Regards,
Sergey