Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
15 Views

LZO compression/decompression in memory

Is it possible to LZO compress/decompress in memory?

0 Kudos
7 Replies
Highlighted
Employee
15 Views

Quoting - thstart

Is it possible to LZO compress/decompress in memory?

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

0 Kudos
Highlighted
15 Views

Quoting - sergey_kh

Hi! Please, detalize - what do you mean with "in memory"? All compression/decompression ops for all IPP DC domain methods are done in memory. Need clarification.

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

0 Kudos
Highlighted
Employee
15 Views

Quoting - thstart

I need a simple example how to compress/decompress a buffer in memory.
LZO example uses fread/fwrite bytewise. Fread/fwrite uses internal pointer
advanced further and ready for the next operation. I do not see how to
make and efficient equivalent of fread/fwrite operation for memory buffer.

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)

doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey

0 Kudos
Highlighted
15 Views

Quoting - sergey_kh

Hi, if I got you right, you want some kind of ZLIB's stream. Thus, to have a "compress" function which accumulates compressed data somewhere, until there is no place in internal buffer and, then, flushes its internal buffer to user program.

If I wanted a function like this, I would do (note: very roughly, not tested, not even compiled ))) :

static Ipp8u* output_buffer[SOME_HUGE_VALUE];
static Ipp8u* outPtr=output_buffer;
static Ipp32u outAvail = SOME_HUGE_VALUE;

int compress(const Ipp8u* inDataPtr, Ipp32u inCount,
Ipp8u** outDataPtr, Ipp32u* outCount)
{
Ipp32u outBytes;
if (inCount > outAvail) { // Output data may not fit into outbuf
*outDataPtr = output_buffer;
*outCount = SOME_HUGE_VALUE - outAvail;
outAvail = SOME_HUGE_VALUE;
outPtr = output_buffer;
return TAKE_COMPRESSED_DATA;
} else {
outBytes = outAvail;
ippsEncodeLZO_8u(inDataPtr, inCount, &outPtr, &outBytes);
outAvail -= outBytes;
return GIVE_MORE;
}
}

int main()
{
while((inCount=thereIsAnyInputData(upperSource, myInputBuffer)) > 0) {
Ipp8u* localOutputPtr;
Ipp32u localOutputCount;
if (compress(myInputBuffer, inCount, &localOutputPtr, &localOutputCount)
==TAKE_COMPRESSED_DATA)

doSomethingWithData(localOutputPtr, localOutputCount);
else
printf("Compress function is asking more!n");
}
}

P.S. unlike ZLIB functions, LZO is oriented toward block compressing. Input block->lzo->output block.

But there is one problem. In LZO functions all input pointers are constant. This means that input pointers are not moved
by the functions. So, if you have read compressed buffer consisting of several compressed block, there are no means
to separate one compressed block from the others. This is why in LZO-oriented packers there are specific markers
to separate compressed blocks. For example, first may come the length of next compressed block.
I.e. | data_length | data.... | data_length | data... | ... and so on.
Having these markers you can send to decoding function exactly required number of bytes.
Does it help?
Regards,
Sergey


Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

LZO Compression Functions

This section describes Intel IPP data compression functions, that implement the LZO

(Lempel-Ziv-Oberhumer) compressed stream format. This format and algorithm use 64KB

compression dictionary and do not require additional memory for decompression. (See

original code of the LZO library at http://www.oberhumer.com.)

Constantine

0 Kudos
Highlighted
Employee
15 Views

Quoting - thstart

Yes, I need an in memory decompression and a working code.

The documentation is a bit misleading. On page 13-90 from ippsman.pdf I read "compressed stream format. "

Constantine

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey

0 Kudos
Highlighted
15 Views

Quoting - sergey_kh

Yes. Sorry. The word "stream" will be removed or substituted with "data". Anyway, to convert block compression into stream-like compression is not very difficult.

Regards,
Sergey


Hot to do that?

0 Kudos
Highlighted
Employee
15 Views

Quoting - thstart


Hot to do that?

Oops! I cannot do this for you )). Simply imagine how your compression routine must look like. What must it take as input? Will there be any output from it? Or, it must look like a "black-box", compressing and writing out the compressed data. What is "stream" in your interpretation? Example of routine, which digests input data and from time to time produces some compressed output is above.

Regards,
Sergey

0 Kudos