Intel® Integrated Performance Primitives
Deliberate problems developing high-performance vision, signal, security, and storage applications.
6704 Discussions

LZO compression/decompression in memory

constantine-vassilev
648 Views

Is it possible to LZO compress/decompress in memory?

0 Kudos
7 Replies
Sergey_K_Intel
Employee
649 Views
Quoting - thstart

Is it possible to LZO compress/decompress in memory?

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

0 Kudos
constantine-vassilev
649 Views
Quoting - sergey_kh

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

0 Kudos
Sergey_K_Intel
Employee
649 Views
Quoting - thstart

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)

doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey

0 Kudos
constantine-vassilev
649 Views
Quoting - sergey_kh

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)

doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey


Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

LZO Compression Functions

This section describes Intel IPP data compression functions, that implement the LZO

(Lempel-Ziv-Oberhumer) compressed stream format. This format and algorithm use 64KB

compression dictionary and do not require additional memory for decompression. (See

original code of the LZO library at http://www.oberhumer.com.)

Constantine

0 Kudos
Sergey_K_Intel
Employee
649 Views
Quoting - thstart

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

Constantine

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

0 Kudos
constantine-vassilev
649 Views
Quoting - sergey_kh

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey


Hot to do that?

0 Kudos
Sergey_K_Intel
Employee
649 Views
Quoting - thstart


Hot to do that?

Oops! I cannot do this for you )). Simply imagine how your compression routine must look like. What must it take as input? Will there be any output from it? Or, it must look like a "black-box", compressing and writing out the compressed data. What is "stream" in your interpretation? Example of routine, which digests input data and from time to time produces some compressed output is above.

Regards,
Sergey

0 Kudos
Reply