proper use of ippsDecode(Encode)LZ77DynamicHuff_8u / ippsDecode(Encode)LZ77_8u

[DF]

Hello,

I've still been playing around with a basic routine of encodelz77 / encodelz77dynamichuff to compression a buch of different blocks of data and found out several things. One is that you have to to use the setstatus routine before each one to the proper LZ77Process or LZ77HuffProcess or it doesn't work, also when done with each block of encoding you have to use ippsEncodeLZ77Reset after each block or you get uncompressible data in the next or even encoding failures. On the decoe side you have to do the same thing, but you can't use reset because if you assign a new pair buffer the reset changes the size but keeps the same buffer (I don't think it shoudl change it at all?). I've found that the ippsDecodeLZ77DynamicHuff function can crash based on data created with the encode routines (with access deined when somewhere in ippsSet is trying to update some memory location). For the most part it will work if one the encode size you can do it without any looping (just one encode/dynamichuff/reset each time). There isn't much detail I can find on proper use of these functions, can someone detail them with a basic loop that would be reliable. Those are the only thing's i'm interested in, not zlib compatiblity or anything like that.

TIA!!

[DF]

after a lot of trial and error, I think i got it... do your encodelz77/encodelz77dynamichuf until you get streamend (it must internally maintain that state based on the last encode call) then when done with the buffer being compressed (presuming you don't want it to continue to stream) call ippencodereset.

On the decode, do the same thing, waiting for streamend, and you must call reset again if not part of a larger stream (not individual buffers).

Now, there is a bug with the decodereset in that it resets the size of the pairs array back to the default of 0x2000 even if you've change the size (keeps the new buffer you may have assigned) so look out there. They shoudl probably fix that bug.

[Ying_H_Intel]

Hi DF,

I justguesssome ofproblem you discribled here. maybe simplier than ippsDecode(Encode)LZ77,in Zlib sample code,it use awhile loop to handle theblock data,like,

int DecompressLZSS( Ipp8u *pSrc, int SrcLen, Ipp8u *pDst, int *pDstLen ) {

IppStatus st;
IppLZSSState_8u * pLZSSState = NULL;
Ipp8u * pTSrc = pSrc,
* pTDst = pDst;
int srcLen = SrcLen,
dstLen = *pDstLen,
tDstLen = 0;

/* inits the lzss state structure for decoding */
if( ippStsNoErr != ( st = ippsDecodeLZSSInitAlloc_8u( &pLZSSState ) ) ) {
fprintf( stderr, "Error while initalloc 4 decode lz77.exiting...\n");
return st;
}

do {
dstLen = GZIPBLOCKSIZE; /* decode using a small blocks of GZIPBLOCKSIZE length */
tDstLen += dstLen;

/* decode source vector pSrc to destination pDst and returns len of decoded vector to dstLen*/
st = ippsDecodeLZSS_8u( &pTSrc, &srcLen, &pTDst, &dstLen, pLZSSState );

/* if error is not due to the insufficies output buffer */
if( st != ippStsNoErr && st != ippStsDstSizeLessExpected ) {
fprintf( stderr, "Error %d while decode lz77.exiting...\n", st);
return st;
}

tDstLen -= dstLen;

} while( st != ippStsNoErr );

*pDstLen = tDstLen;
dstLen = tDstLen;

/* frees the space, allocated for pLZSSState state structure */
ippsLZSSFree_8u( pLZSSState );
return 0;
} /* DecompressLZSS() */

ippsDecode(Encode)LZ77DynamicHuff should have more complex exit condition because,

The function ippsDecodeLZ77DynamicHuff is declared in the ippdc.h file. This function performs decoding of the source data ppSrc with the dynamic Huffman codes (see RFC1951 specification).

Exit from the ippsDecodeLZ77FixedHuff function occurs when:

• the input stream has ended, but the end-of-block marker is not decoded and the function returns the ippStsSrcSizeLessExpected status.
• the input stream has not ended, but the destination buffer is full and the function returns the ippStsDstSizeLessExpected status.
• the input stream has ended, the end-of-stream marker is decoded and the function returns the ippStsNoErr status.

And you alsomentioned, other problem like ippsDecodeLZ77Reset_8u etc.Could you please providea smallrunable test code so we can see what is the exact probelm? (if the code is confidential, you can post as private).

and you mentioned"there is a bug with the decodereset in that it resets the size of the pairs array back to the default of 0x2000 even if you've change the size (keeps the new buffer you may have assigned) so look out there" , we need to have a sample case to reproduce the problem too.

Best Regards,
Ying

[DF]

Basically if you init for decompress as:

#define IPPpairSize 0xFF00

#if defined(IPPpairSize)
IppLZ77Pair* pairs=(IppLZ77Pair*) ippsMalloc_8u(IPPpairSize*sizeof(IppLZ77Pair));
// ensure enough memory
if (pairs==NULL) {
return ERROR_OUTOFMEM;
}
#endif
// create and setup the ipp state buffer
if (ippsDecodeLZ77InitAlloc_8u(IppLZ77NoChcksm, &IPPLZ77State)!=ippStsNoErr) {
// on failure - clean up and exit
#if defined(IPPpairSize)
ippsFree(pairs);
#endif
IPPLZ77State=NULL;
return ERROR_OUTOFMEM;
}
// setup the new larger pairs buffer to use.
#if defined(IPPpairSize)
ippsDecodeLZ77SetPairs_8u(pairs, 0, IPPpairSize, IPPLZ77State);
#endif

// now you see pairlen is 0xFF00
IppLZ77Pair *ppair;
int paircur, pairlen;
ippsDecodeLZ77GetPairs_8u(&ppair, &paircur, &pairlen, IPPLZ77State);


// Then in your routine somewhere (even here) called:

ippsDecodeLZ77Reset_8u(IPPLZ77State);

// Then next time you did :

ippsDecodeLZ77GetPairs_8u(&ppair, &paircur, &pairlen, IPPLZ77State);

You'll see the pairlen is now 0x2000 and no longer 0xFF00

[Ying_H_Intel]

Hi DF,

Thanks for the test case. I get more informationfromIPP developer.

Sorry,these LZ77 interfacementioned here are kinda deprecated and is not used for a while.IPP_GZIP in IPP version 6.0 had use the funtions.

IPP_GZIP sample uses the following functions for Intel Performance Primitives Data Compression library:

IPP_GZIP Function	IPP Function	Function Description
Compression/Test	ippsEncodeLZ77InitAlloc_8u	Allocates memory and initializes the encoding state
	ippsEncodeLZ77SetStatus_8u	Sets the deflate status to the desired value in the LZ77 encoding state structure
	ippsEncodeLZ77GetPairs_8u	Retrieves pair data from the LZ77 encoding state
	ippsEncodeLZ77_8u	Performs LZ77 encoding
	ippsEncodeLZ77FixedHuff_8u	Performs fixed Huffman encoding
	ippsEncodeLZ77DynamicHuff_8u	Performs dynamic Huffman encoding
	ippsEncodeLZ77Flush_8u	Writes the checksum and total length of the input data to the end of the stream
	ippsLZ77Free_8u	Frees memory allocated for the LZ77 encoding and decoding structures
	ippsCRC32_8u	Computes the CRC32 checksum for the source data buffer
Decompression	ippsDecodeLZ77InitAlloc_8u	Allocates memory and initializes the LZ77 decoding structure
	ippsDecodeLZ77SetStatus_8u	Sets the inflate status to the desired value in the LZ77 decoding state structure
	ippsDecodeLZ77GetBlockType_8u	Determines the type of encoded data block
	ippsDecodeLZ77FixedHuffFull_8u	Performs LZ77 and fixed Huffman decoding
	ippsDecodeLZ77DynamicHuffFull_8u	Performs LZ77 and dynamic Huffman decoding
	ippsDecodeLZ77StoredBlock_8u	Performs stored block (RFC 1951) decoding

So if you'd like see the example, you can refer to it. (attachment ) and Please notes, we may consider to remove the old interface in the further version. so may not bug fix on these.

Therearenew LZ77 interface based on ippsDeflateLZ77_8u/ippsInflate_8u functions in current version,you can see them fromreadme.htmunder the folder /interface/IPP_GZIP.

Intel IPP Functions Used In The Interface

IPP_GZIP interface uses the following functions from Intel Performance Primitives Data Compression library:

IPP_GZIP Function	IPP Function	Function Description
Compression	ippsDeflateLZ77_8u	Performs LZ77 encoding
	ippsDeflateHuff_8u	Performs Huffman encoding
	ippsCRC32_8u	Computes the CRC32 checksum for the source data buffer
Decompression/Test	ippsInflate_8u	Performs LZ77 decoding
	ippsInflateBuildHuffTable	Performs Huffman decoding

IPP_GZIP Function	IPP Function	Function Description
Compression	ippsDeflateLZ77_8u	Performs LZ77 encoding
	ippsDeflateHuff_8u	Performs Huffman encoding
	ippsCRC32_8u	Computes the CRC32 checksum for the source data buffer
Decompression/Test	ippsInflate_8u	Performs LZ77 decoding
	ippsInflateBuildHuffTable	Performs Huffman decoding

And they are used in IPP data compression sample too.

Best Regards,
Ying