Issue with LZO functions

[bronxzv]

I usethe LZO compression functionsavailable in IPP included in Intel C++ Pro (Win 32 v11.1.038) for a commercial project. They are fast and very easy to use, though I have some use cases where the expanded data is different than source data before compression (oughhh !). I have several repeatable cases where the checksum is different for source data and compressed + decompressed data,latest example :
checksum of source data = 298828635
checksum of compressed + decompressed data = 298828361
comparing the data it looks like thereisat least 2 bytesthat are different

NB: most of the cases I have tested have exactly the same checksum so it looks like an issue with IPP not with the way my code use the library

I use the "IppLZO1XST" method for "ippsEncodeLZOInit_8u"

the bug is exactly the same if I use "ippsDecodeLZO_8u" or "ippsDecodeLZOSafe_8u" and both functions return "ippStsNoErr"

Provided that you are not already aware of the problem, I can provide source data that trigger the bug (100% repeatable), the smallest dataset is a 1524 KB file, just ask me

[Ying_H_Intel]

hello bronxzv.

I recalled, we havea known issue using LZO method 'IppLZO1XMT' reportedat
<http://software.intel.com/en-us/forums/showthread.php?t=65389>.

Could you please see if it is sameissue? The fix version is IPP 6.1 update 1 which is in Compiler 11.1.046 if linux. <http://software.intel.com/en-us/articles/which-version-of-ipp--mkl--tbb-is-installed-with-intel-compiler-professional-edition/>

Best Regards,
Ying

[bronxzv]

Quoting - Merryhu@intel (Intel)

hello bronxzv.

I recalled, we havea known issue using LZO method 'IppLZO1XMT' reportedat
<http://software.intel.com/en-us/forums/showthread.php?t=65389>.

Could you please see if it is sameissue? The fix version is IPP 6.1 update 1 which is in Compiler 11.1.046 if linux. <http://software.intel.com/en-us/articles/which-version-of-ipp--mkl--tbb-is-installed-with-intel-compiler-professional-edition/>

Best Regards,
Ying

Hi Ying,

Thanks for your quick reply. The bug reported at your link is about the multithreaded (MT) versionof the codecbut I use thesingle thread(ST) version so N=1 andI'm afraid it's another bug. NB: I use the single thread version because the IPP documentation says that the MT version use a non-standard format for compressed data and we don't want to be stuck with tons of legacy data if IPP no more support the proprietary format in a few years down the line.

I think I use a pretty recent version of IPP 6.1.x, I installed it early August 2009 from this package : w_cproc_p_11.1.038_ia32.exe

Now, if there is a patch available for IPP (with LZO related fixes) before the next update of the full compiler I'll be interested to test it. Otherwise I'll suggest to escalate the issue to the IPP team and provide me an email where I'll be allowed to send examples of data that trigger the issue.

[Ying_H_Intel]

Hello,

Right, you should be using the latest IPP 6.1 update 1. The issuelooksnew. Could you please send me your test case?

Regards,
Ying

[Sergey_K_Intel]

Hi bronxzv, Ying,

No, this may be another problem which wasn't fixed in update 1.
There are some input data cases, which even being compressed in single-thread mode, look like multi-thread compressed data. So, LZO decode function wrongly treats them as multi-thread-compressed and applies multi-thread decompression, which gives incorrect output.
This problem was fixed in late July and this fix didn't get to any of releases yet.

There's no way out of this, but to try using IppLZO1XMT compression mode.
AFAIK, the next 6.1 Gold update is planned on November this year. I'll try to make sure that it'll be free of this problem.

Best wishes,
Sergey

Edited after seeing your discussion: bronxzv is perferctly right saying regarding "legacy problems". In latest MT-mode compressed format we have to modify it in order to get farther away from ST (i.e. LZO standard) format.

[bronxzv]

Hi Sergey,

Thanks for your detailed answer. The bad side effect of the MT pathmakes a lot of sense as a likely explanation since LZO is well known AFAIK as a bullet proof codec. So I'll wait for the fixed version, thoughwaiting until November for a showstopperalready fixed in July looks quite long IMO

[Sergey_K_Intel]

Quoting - bronxzv

thoughwaiting until November for a showstopperalready fixed in July looks quite long IMO

I was incorrect saying "didn't get to release". There were no releases since update 1. All releases are planned and scheduled. We cannot do local patching, because it will lead to total mess.But arelease - even update -is too heavy thing to do often.

Regards,
Sergey

[bronxzv]

Quoting - Sergey Khlystov (Intel)

I was incorrect saying "didn't get to release". There were no releases since update 1. All releases are planned and scheduled. We cannot do local patching, because it will lead to total mess.But arelease - even update -is too heavy thing to do often.

Regards,
Sergey

I have just tested with"w_cproc_p_11.1.051_ia32.exe" and the LZO bug(s) is(are) still there
All the test cases that were not working with "w_cproc_p_11.1.046_ia32.exe" are still triggering the bug
So I'll appreciate a revised status on this issue, i.e. is the next IPP "6.1Gold update" still planed for this November ?

[Sergey_K_Intel]

I have just tested with"w_cproc_p_11.1.051_ia32.exe" and the LZO bug(s) is(are) still there

Let me some time to look into.

Regards,
Sergey

[Sergey_K_Intel]

Quoting - bronxzv

the LZO bug(s) is(are) still there

I see sporadic problem with PX (generic) code. This is another bug and can be avoided using either dynamic linking, or using "ippStaticInit()/ippStaticInitCpu(ippCpuP4)/ippStaticInitCpu(ippCpuC2D)/..." calls in static-linked application. These calls trigger using of other paths/functions within LZO. Though, use these non-PX variants so far.
Thank you for your finding,
Sergey

[bronxzv]

Quoting - Sergey Khlystov (Intel)

I see sporadic problem with PX (generic) code. This is another bug and can be avoided using either dynamic linking, or using "ippStaticInit()/ippStaticInitCpu(ippCpuP4)/ippStaticInitCpu(ippCpuC2D)/..." calls in static-linked application. These calls trigger using of other paths/functions within LZO. Though, use these non-PX variants so far.
Thank you for your finding,
Sergey

haaaahhh yes, thanks!, I have added an "ippStaticInit()" in my initialization code and now it works well (i.e. allthe test cases with the issue are now OK)

FYI themissing"ippStaticInit"was leading to problems only when decompressing not for compressing

[bronxzv]

Quoting - Sergey Khlystov (Intel)

"...ippStaticInitCpu(ippCpuP4)/ippStaticInitCpu(ippCpuC2D)/..." calls in static-linked application. These calls trigger using of other paths/functions within LZO.

out of curiosity I have tested (on a Core i7 940) with some specific IppCpuType values, some findings below:

ippInitCpu(ippCpuUnknown); // KO
ippInitCpu(ippCpuPP); // KO
ippInitCpu(ippCpuPMX); // KO
ippInitCpu(ippCpuPPR); // KO
ippInitCpu(ippCpuPII); // KO
ippInitCpu(ippCpuPIII); // KO
ippInitCpu(ippCpuP4); // OK
ippInitCpu(ippCpuP4HT); // OK
ippInitCpu(ippCpuP4HT2); // OK
ippInitCpu(ippCpuCentrino); // OK
ippInitCpu(ippCpuNehalem); // OK
ippInitCpu(ippCpuSSE); // KO
ippInitCpu(ippCpuSSE2); // OK

so it looks like there is at least2 code paths, one using SSE2 that works fine and another one with the LZO decompression bug(s)

my conclusion: calling "ippStaticInit" fix the issue probably only on SSE2 enabled targets with the bug still lurking for older CPUs