Community
cancel
Showing results for 
Search instead for 
Did you mean: 
DF
Beginner
88 Views

proper use of ippsDecode(Encode)LZ77DynamicHuff_8u / ippsDecode(Encode)LZ77_8u

Hello,

I've still been playing around with a basic routine of encodelz77 / encodelz77dynamichuff to compression a buch of different blocks of data and found out several things. One is that you have to to use the setstatus routine before each one to the proper LZ77Process or LZ77HuffProcess or it doesn't work, also when done with each block of encoding you have to use ippsEncodeLZ77Reset after each block or you get uncompressible data in the next or even encoding failures. On the decoe side you have to do the same thing, but you can't use reset because if you assign a new pair buffer the reset changes the size but keeps the same buffer (I don't think it shoudl change it at all?). I've found that the ippsDecodeLZ77DynamicHuff function can crash based on data created with the encode routines (with access deined when somewhere in ippsSet is trying to update some memory location). For the most part it will work if one the encode size you can do it without any looping (just one encode/dynamichuff/reset each time). There isn't much detail I can find on proper use of these functions, can someone detail them with a basic loop that would be reliable. Those are the only thing's i'm interested in, not zlib compatiblity or anything like that.

TIA!!

0 Kudos
4 Replies
DF
Beginner
88 Views

after a lot of trial and error, I think i got it... do your encodelz77/encodelz77dynamichuf until you get streamend (it must internally maintain that state based on the last encode call) then when done with the buffer being compressed (presuming you don't want it to continue to stream) call ippencodereset.

On the decode, do the same thing, waiting for streamend, and you must call reset again if not part of a larger stream (not individual buffers).

Now, there is a bug with the decodereset in that it resets the size of the pairs array back to the default of 0x2000 even if you've change the size (keeps the new buffer you may have assigned) so look out there. They shoudl probably fix that bug.

Ying_H_Intel
Employee
88 Views

Hi DF,

I justguesssome ofproblem you discribled here. maybe simplier than ippsDecode(Encode)LZ77,in Zlib sample code,it use awhile loop to handle theblock data,like,

int DecompressLZSS( Ipp8u *pSrc, int SrcLen, Ipp8u *pDst, int *pDstLen ) {

IppStatus st;
IppLZSSState_8u * pLZSSState = NULL;
Ipp8u * pTSrc = pSrc,
* pTDst = pDst;
int srcLen = SrcLen,
dstLen = *pDstLen,
tDstLen = 0;

/* inits the lzss state structure for decoding */
if( ippStsNoErr != ( st = ippsDecodeLZSSInitAlloc_8u( &pLZSSState ) ) ) {
fprintf( stderr, "Error while initalloc 4 decode lz77.exiting...\n");
return st;
}

do {
dstLen = GZIPBLOCKSIZE; /* decode using a small blocks of GZIPBLOCKSIZE length */
tDstLen += dstLen;

/* decode source vector pSrc to destination pDst and returns len of decoded vector to dstLen*/
st = ippsDecodeLZSS_8u( &pTSrc, &srcLen, &pTDst, &dstLen, pLZSSState );

/* if error is not due to the insufficies output buffer */
if( st != ippStsNoErr && st != ippStsDstSizeLessExpected ) {
fprintf( stderr, "Error %d while decode lz77.exiting...\n", st);
return st;
}

tDstLen -= dstLen;

} while( st != ippStsNoErr );

*pDstLen = tDstLen;
dstLen = tDstLen;

/* frees the space, allocated for pLZSSState state structure */
ippsLZSSFree_8u( pLZSSState );
return 0;
} /* DecompressLZSS() */

ippsDecode(Encode)LZ77DynamicHuff should have more complex exit condition because,

The function ippsDecodeLZ77DynamicHuff is declared in the ippdc.h file. This function performs decoding of the source data ppSrc with the dynamic Huffman codes (see RFC1951 specification).

Exit from the ippsDecodeLZ77FixedHuff function occurs when:

  • the input stream has ended, but the end-of-block marker is not decoded and the function returns the ippStsSrcSizeLessExpected status.
  • the input stream has not ended, but the destination buffer is full and the function returns the ippStsDstSizeLessExpected status.
  • the input stream has ended, the end-of-stream marker is decoded and the function returns the ippStsNoErr status.


And you alsomentioned, other problem like ippsDecodeLZ77Reset_8u etc.Could you please providea smallrunable test code so we can see what is the exact probelm? (if the code is confidential, you can post as private).

and you mentioned"there is a bug with the decodereset in that it resets the size of the pairs array back to the default of 0x2000 even if you've change the size (keeps the new buffer you may have assigned) so look out there" , we need to have a sample case to reproduce the problem too.

Best Regards,
Ying

DF
Beginner
88 Views

Basically if you init for decompress as:

#define IPPpairSize 0xFF00

#if defined(IPPpairSize)
IppLZ77Pair* pairs=(IppLZ77Pair*) ippsMalloc_8u(IPPpairSize*sizeof(IppLZ77Pair));
// ensure enough memory
if (pairs==NULL) {
return ERROR_OUTOFMEM;
}
#endif
// create and setup the ipp state buffer
if (ippsDecodeLZ77InitAlloc_8u(IppLZ77NoChcksm, &IPPLZ77State)!=ippStsNoErr) {
// on failure - clean up and exit
#if defined(IPPpairSize)
ippsFree(pairs);
#endif
IPPLZ77State=NULL;
return ERROR_OUTOFMEM;
}
// setup the new larger pairs buffer to use.
#if defined(IPPpairSize)
ippsDecodeLZ77SetPairs_8u(pairs, 0, IPPpairSize, IPPLZ77State);
#endif

// now you see pairlen is 0xFF00
IppLZ77Pair *ppair;
int paircur, pairlen;
ippsDecodeLZ77GetPairs_8u(&ppair, &paircur, &pairlen, IPPLZ77State);


// Then in your routine somewhere (even here) called:

ippsDecodeLZ77Reset_8u(IPPLZ77State);

// Then next time you did :

ippsDecodeLZ77GetPairs_8u(&ppair, &paircur, &pairlen, IPPLZ77State);

You'll see the pairlen is now 0x2000 and no longer 0xFF00

Ying_H_Intel
Employee
88 Views

Hi DF,

Thanks for the test case. I get more informationfromIPP developer.

Sorry,these LZ77 interfacementioned here are kinda deprecated and is not used for a while.IPP_GZIP in IPP version 6.0 had use the funtions.

IPP_GZIP sample uses the following functions for Intel Performance Primitives Data Compression library:

IPP_GZIP Function IPP Function Function Description
Compression/Test ippsEncodeLZ77InitAlloc_8u Allocates memory and initializes the encoding state
ippsEncodeLZ77SetStatus_8u Sets the deflate status to the desired value in the LZ77 encoding state structure
ippsEncodeLZ77GetPairs_8u Retrieves pair data from the LZ77 encoding state
ippsEncodeLZ77_8u Performs LZ77 encoding
ippsEncodeLZ77FixedHuff_8u Performs fixed Huffman encoding
ippsEncodeLZ77DynamicHuff_8u Performs dynamic Huffman encoding
ippsEncodeLZ77Flush_8u Writes the checksum and total length of the input data to the end of the stream
ippsLZ77Free_8u Frees memory allocated for the LZ77 encoding and decoding structures
ippsCRC32_8u Computes the CRC32 checksum for the source data buffer
Decompression ippsDecodeLZ77InitAlloc_8u Allocates memory and initializes the LZ77 decoding structure
ippsDecodeLZ77SetStatus_8u Sets the inflate status to the desired value in the LZ77 decoding state structure
ippsDecodeLZ77GetBlockType_8u Determines the type of encoded data block
ippsDecodeLZ77FixedHuffFull_8u Performs LZ77 and fixed Huffman decoding
ippsDecodeLZ77DynamicHuffFull_8u Performs LZ77 and dynamic Huffman decoding
ippsDecodeLZ77StoredBlock_8u Performs stored block (RFC 1951) decoding

So if you'd like see the example, you can refer to it. (attachment ) and Please notes, we may consider to remove the old interface in the further version. so may not bug fix on these.

Therearenew LZ77 interface based on ippsDeflateLZ77_8u/ippsInflate_8u functions in current version,you can see them fromreadme.htmunder the folder /interface/IPP_GZIP.

Intel IPP Functions Used In The Interface

IPP_GZIP interface uses the following functions from Intel Performance Primitives Data Compression library:

IPP_GZIP Function IPP Function Function Description
Compression ippsDeflateLZ77_8u Performs LZ77 encoding
ippsDeflateHuff_8u Performs Huffman encoding
ippsCRC32_8u Computes the CRC32 checksum for the source data buffer
Decompression/Test ippsInflate_8u Performs LZ77 decoding
ippsInflateBuildHuffTable Performs Huffman decoding
IPP_GZIP Function IPP Function Function Description
Compression ippsDeflateLZ77_8u Performs LZ77 encoding
ippsDeflateHuff_8u Performs Huffman encoding
ippsCRC32_8u Computes the CRC32 checksum for the source data buffer
Decompression/Test ippsInflate_8u Performs LZ77 decoding
ippsInflateBuildHuffTable Performs Huffman decoding


And they are used in IPP data compression sample too.

Best Regards,
Ying

Reply