Access violation in release code

[paul3]

We are using the IPP JPEG library as a replacement for the IJG JPEG library. The application is experiencing failures on random image files when processing 180,000+ images.

This failure does not occur on the same image file twice. It happens only with the release build of the IPP JPEG library and apparently not with the debug build. At least we have been getting the failure on every run with the release library and not yet with the debug build.

The failure is in ippji7.dll and is always trying to load 16 bytes from an address ending in ff3. Access to the next 4k page fails - it is not allocated.

Any suggestions?

[paul3]

More information:

- This fails only on a dual-Xeon machine.

- Turning UseIPP off after calling CreateDecompress eliminates the problem, so it is clearly something in the IPP code.

We are continuing to try to track this down. It is extremely unclear what might be happening, but it is beginning to look more and more like it is something in the IPP JPEG code.

[Vladimir_Dudnik]

Hi,

Thank you for letting us know about some problems, but could you please provide a bit more information?Specifically, what version of IPP do you use? What operating system? What hardware platform? Did you tried static IPP libraries or you only use DLL?

In your first message you said that fail caused by ippji7 library, which is Itanium specific library. Now you are talking that this fails only on dual-Xeon system.

Is your application use any kind of threading above IPP?

What compiler did you use to produce executable?

I'll be glad to help you as much as I can, but need also some help from you:)

Regards,
Vladimir

[paul3]

I was misinformed about the name of the DLL involved. Unfortunately, the parameters of this problem make it difficult to track down - I was hoping for something obvious that was on the order of "yes, that problem..." that might have saved some time.

What we have is the 4.2 library linked with MS Visual Studio .NET 2003. Our JPEG handler is in a DLL separate from the main application. Everything is linked to use DLL runtime modules. The image processing sections of the application are using a large number of threads - 20 in the current testing on this machine.

The failure is occurring in ippjt7.dll on al LDDQU instruction at offset 0x4F625. It loading data pointed to by the ebp register which contains 02174FF3. The failure specifically calls out the address 0x02175000, which unallocated.

This point is called from another location in the ippjt7.dll - 0x2E09e, which in turn has been called from the IPP JPEG source. This is in jdsample.c in the function h2v2_fancy_upsample_intellib which calls ippiSampleUpRowH2V2_Triangle_JPEG_8u_C1.

To eliminate just about everything we have control over, we made one minor change toour JPEG processing DLL - after calling jpeg_create_decompress we are now setting jpeg->cinfo.UseIPP = 0. This then eliminates the problem.

The problem manifests itself when running 20 image processing threads on the dual Xeon machine. There is a fair amount of database interaction going on here as well, with most of the testing having been done withan MS SQL 2005 database on a different machine via ODBC. We do not see any problems with less than 40,000 images and the problems are not reproducible. The image files are not the problem because isolating the images and running the application on them does not reproduce the problem and each run with the full set of images produces failures with different image files.

This only happens with the release build of both the application and the our JPEG DLL. If either one of them is a debug build, there are no problems.

The only clue we have at this point is that the LDDQU instruction is pointing to xxxxxFF3 or xxxxxFF4 for every failure.

I am not certain the problem is in IPP by any means. The fact that it disappears with UseIPP = 0 is somewhat of an indication, but the problem also goes away with a debug build of the application using IPP. I therefore do not believe we have isolated the problem at all. It could be anything, including stack corruption in the application. From looking at the code that calls the ippiSampleUpRowH2V2_Triangle_JPEG_8u_C1 function it seems unlikely that the buffer being read is in reality too short. Far more likely that the memory being referenced has been incorrectly released.

We have a public demo in two weeks and it has to work for that. Following that I believe we can try to narrow in on this problem some more. Any advice that you can provide would be helpful.

[jim_dempsey]

>>

The only clue we have at this point is that the LDDQU instruction is pointing to xxxxxFF3 or xxxxxFF4 for every failure.

<<

LDDQU is loading 16 bytes into an xmm register, however depending on alignment issues it may read up to 32 bytes to get the 16 bytes. The xxxxxFF3 and xxxxxFF4 when reading just 16 bytes will advance into the next 4KB address. This may cause a page fault (not your fault).By looking at you call stack can you tell what your sourcemodule passed in to the library function. If you can find out then try to force an alignment of that (those) variables or arrays. There may be an alingment requirement on your arguments.

Jim Dempsey

[paul3]

Thanks for your reply.

This is being called by the IPP JPEG library from the jdsample.c module. I am not entirely sure what the inputs and outputs are for the function involved. I have been able to establish that neither the input or output point to buffers that were allocated by the application or our JPEG wrapper. Both seem to have been allocated by the JPEG library.

There might be an alignment issue, because it would make sense to be loading 16-byte groups from a 16-byte boundary. Hopefully, I can spend some time with this and try to understand what the input and outputs are for both the undocumented IPP JPEG library function and the IPP library function that is being called. I haven't yet tried to find the documentation for ippiSampleUpRowH2V2_Triangle_JPEG_8u_C1 to see what might be happening with it.

If there is an alignment problem, I'm still a little mystified as to why this only fails as rarely as it does - 16-byte boundary alignments are pretty rare unless you go out of your way to get them.

[Vladimir_Dudnik]

Hi,

first of all thanks a lot for such a deep analisys of that problem. Unfortunately it gives no answerabout the reasons for this fault. Of course we tested all IPP functions on all supported architectures and did not face this problem. Could you try to link the same application with IPP static libraries? Another option, if you do not have static libraries, you can try to disable OMP threading by calling ippSetNumThreads(1) somewhere at the beginning of your program?

It should not be alignment problem because in all IPP functions we are trying to process unaligned cases separately.

You also can check the same images with JPEGView sample, which uses the same function and see if the problem is reproducable.

Regards,
Vladimir

[yakov_galka]

We have the same problem. The following code reproduces the problem:

const DWORD page = 8192;
void* mem = VirtualAlloc(0, page*2, MEM_COMMIT, PAGE_READWRITE);
void* dest = VirtualAlloc(0, page, MEM_COMMIT, PAGE_READWRITE);
void* guard = (char*)mem + page;
DWORD oldAccess;
BOOL ok = VirtualProtect(guard, page, PAGE_NOACCESS, &oldAccess);

IppStatus stat = ippiSampleUpRowH2V2_Triangle_JPEG_8u_C1(
(const Ipp8u*)guard - 256,
(const Ipp8u*)guard - 256,
256,
(Ipp8u*)dest+8
);

ippiSampleUpRowH2V2_Triangle_JPEG_8u_C1 fails due to access violation, trying to read 16 bytes from (const Ipp8u*)guard-13, exactly as described by Paul.

Tested against IPP version 5.1, Visual Studio 2005, any build configuration (release/debug, win32/win64). One system is Windows 7 the other is Windows XP, both running on Intel Core2 Quad CPU Q9550 @ 2.83GHz.

Reproducible: always.

[Vladimir_D_Intel1]

Thanks for informing about that issue in IPP v5.1. Please note that the latest version of IPP is 7.0.5. If I remember it correctly this issue was fixed some time ago (could not recall exact version of library where it was fixed). Is it possible for you to check your test case with the latest version of IPP?

Regards,
Vladimir

[Ying_H_Intel]

Right, i recalled similiar issue in JPEG funtions. DPD200134360 ippiDecodeHuffman8x8_Direct_JPEG_1u163_C1 reads beyond input data, in IPP 6.1 bug fix list
http://software.intel.com/en-us/articles/intel-ipp-library-61-fixes-list/

If at least 1 byte of src belongs to 16-byte aligned paragraph - all bytes from this pargraph can be read. Corresponding fix applied in IPP 6.1.

I just try IPP 7.0.5 with yakov.galka's test case on windows 7 64bit. the problem should be fixed in the version.

Thanks
Ying

[yakov_galka]

Thank you for conforming that it was a known bug. We are currently migrating to the latest IPP, and it's indeed not reproducible anymore.

[Naveen_G_Intel]

Hi,

Good to know that latest version works fine..

Thanks,

Naveen Gv

[Eugene1]

Hi,

Im testing IPP Jpeg lib and it looks like it fails on some images, when JpegLibs input buffer size differs from default 4096 value. I can reproduce the problem with djpeg application from your samples, just changing the follwing value:

[cpp]#define INPUT_BUF_SIZE  100096 [/cpp]

I get access violation in jdhuff.c:1001 when memmove tries to use overflowed value as data size.

[cpp]/* Decode a single block's worth of coefficients */
      if(state.ipp_need_update == 1)
      {
        // fails here
        memmove(state.ipp_buffer,
                state.ipp_buffer + state.ipp_bytes_decoded,
                state.ipp_bytes_in_buffer - state.ipp_bytes_decoded);

        state.ipp_bytes_in_buffer = state.ipp_bytes_in_buffer - 
                                    state.ipp_bytes_decoded - 
                                    ipp_bytes_left - 
                                    state.sync_shift;
//...[/cpp]

Most of my test jpegs works fine, so Im attaching failed jpeg.

My configuration:

Windows 7 x64

IPP version: 7.0.205, win32 binaries

w_ipp-samples_p_7.0.6.060

Thanks,

Eugene