topic Test for possible IppiConvValid bug in Intel® Integrated Performance Primitives

Test for possible IppiConvValid bug

Ryan2 — Wed, 15 Aug 2012 14:26:58 GMT

Hi,

I'm getting a -9 mem error with ippiConvValid_32f_C1R even though my system has plenty of memory.

I allocated a 25GB with malloc just to make sure my program's memory allocation wasn't being limited by Windows. No problems there.

Attached to this email is a compiled Windows x64 console app. Everyone please download it and reply with the MAX size you can enter before you get a -9 err along with your system specs. If you want to build it yourself, please go to my other post to download the VS project. http://redfort-software.intel.com/en-us/forums/showthread.php?t=106382&o=a&s=lr

Hopefully there is a solution to this because it's really hampering me!

Best,

Ryan

Test for possible IppiConvValid bug

SergeyKostrov — Thu, 16 Aug 2012 03:12:09 GMT

Quoting Ryan

...I allocated a 25GB with malloc just to make sure my program's memory allocation wasn't being limited by Windows. No problems there...

Ryan,

Did you try to allocate more memory with a 'malloc' CRT function, for example 64GB and higher? Please take a look at
a test-application I created some time ago and attached to that post:

http://redfort-software.intel.com/en-us/forums/showpost.php?p=189221

I'll take a look at your test-application.

Best regards,
Sergey

Test for possible IppiConvValid bug

SergeyKostrov — Thu, 16 Aug 2012 03:19:19 GMT

Quoting Sergey Kostrov

I'll take a look at your test-application...

What about source codes of your test-application? At least simplified version that reproduces the problem
with ippiConvValid_xxx IPP function.

Test for possible IppiConvValid bug

Ryan2 — Thu, 16 Aug 2012 03:55:50 GMT

Hi Sergey,

Attached is the source code for my test app. It's very easy to follow.

Thanks for looking into this for me!

Best,

Ryan

Test for possible IppiConvValid bug

Ryan2 — Thu, 16 Aug 2012 04:03:44 GMT

Hi Sergey,

I ran your mem test app with the CM3 line uncommented:

__int64 _MEM_SIZE = ( 1073741824 * 64.0 ); // 64GB Recommended VM = N/A / 96GB

64-bit Windows platform

Allocating memory with 'malloc' CRT-function - 1D

Succesfully Allocated 64.00GB

64-bit Windows platformAllocating memory with 'malloc' CRT-function - 1DSuccesfully Allocated 64.00GB

Test for possible IppiConvValid bug

Ryan2 — Thu, 16 Aug 2012 14:26:45 GMT

Update,

I also tried my mem test app with IppiConvValid_16s_C1R, IppiConvValid_16u_C1R, and IppiConvValid_8u_C1R and the resutls are the same.

On my machine:

(Dual socket Intel Xeon E5645, 48GB DDR3) - 1.06 before I get -9 mem error.

On three other computers I've tried:

(Dual socket Xeon E5420, 16GB DDR2) -1.89.

(Dual socket Xeon E5420, 16GB DDR2) - 1.89.

(Dual socket i7, 12GB DDR2) - 1.89.

Ryan

Test for possible IppiConvValid bug

SergeyKostrov — Fri, 17 Aug 2012 02:02:06 GMT

Quoting Ryan

Attached is the source code for my test app. It's very easy to follow...

Hi Ryan,

Could you upload as a zip archive, please? I couldn't extract your sources from the rar archive:

   1   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\mem_test.cpp" header broken
   2   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\mem_test.cpp
   3   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\mem_test.vcproj" header broken
   4   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\mem_test.vcproj
   5   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\stdafx.cpp" header broken
   6   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\stdafx.cpp
   7   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\stdafx.h" header broken
   8   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\stdafx.h
   9   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\targetver.h" header broken
10   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\targetver.h
11   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\BuildLog.htm" header broken
12   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\BuildLog.htm
13   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\mem_test.exe.intermediate.manifest" header broken
14   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\mem_test.exe.intermediate.manifest
15   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\mem_test.obj" header broken
16   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\mem_test.obj
17   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\mem_test.pch" header broken
18   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\mem_test.pch
19   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\mt.dep" header broken
20   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\mt.dep
21   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\stdafx.obj" header broken
22   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\stdafx.obj
23   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\vc90.idb" header broken
24   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\vc90.idb
25   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release\vc90.pdb" header broken
26   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test\x64\Release\vc90.pdb
27   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test.ncb" header broken
28   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test.ncb
29   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test.sln" header broken
30   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test.sln
31   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test.suo" header broken
32   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\mem_test.suo
33   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\x64\Release\libiomp5md.dll
34   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\x64\Release\mem_test.exe" header broken
35   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\x64\Release\mem_test.exe
36   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\x64\Release\mem_test.pdb" header broken
37   IppiConvValid_mem_test.rar: Unknown method in IppiConvValid_mem_test\x64\Release\mem_test.pdb
38   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64\Release" header broken
39   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test\x64" header broken
40   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\x64\Release" header broken
41   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\mem_test" header broken
42   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test\x64" header broken
43   IppiConvValid_mem_test.rar: File "IppiConvValid_mem_test" header broken

Best regards,
Sergey

PS: That is really strange but it looks like older versions of WinRAR are not compatible with the newer versions.

Test for possible IppiConvValid bug

SergeyKostrov — Fri, 17 Aug 2012 02:32:24 GMT

Hi Ryan, This ia a short follow up regarding the error -9...

Quoting Ryan

...I'm getting a -9 mem error with ippiConvValid_32f_C1R even though my system has plenty of memory.

I allocated a 25GB with malloc just to make sure my program's memory allocation wasn't being limited by Windows. No problems there.

Attached to this email is a compiled Windows x64 console app. Everyone please download it and reply with the MAX size you can enter before you get a -9 err...

This is how it is declared:

     typedef enum
     {
         ...
         ippStsMemAllocErr           = -9,
         ...
     } IppStatus;

and I think the problem could be possibly related to limitation(s) of some IPP functions, including ippiConvValid_xxx. Your 64-bit system
allows to allocate up to 64GB of memory (!) and you've done two tests. These "magic" limitations were
disccused a couple of times on the IPP forum in the past. Unfortunately, there is no article that describes these limitations
in some functions of IPP library.

Best regards,
Sergey

Test for possible IppiConvValid bug

Ryan2 — Fri, 17 Aug 2012 02:33:32 GMT

Hi Sergey,

That's weird. Sorry about that.

Please see attached for zip.

Best,

Ryan

Test for possible IppiConvValid bug

Ryan2 — Fri, 17 Aug 2012 02:37:48 GMT

Hi Sergey,

Yes I looked up that enum. That's exactly my thinking. I want to know why IppConvValid is having that problem. For it to run out of memory on my system it would have to be doing a HUGE FFT. What I find even stranger is that an i7 with 12gigs of DD2 memory can handle more than my Xeon with 48gigs of DD3.

Ryan

Test for possible IppiConvValid bug

Ryan2 — Fri, 17 Aug 2012 02:44:42 GMT

OK!

I tried using IppConvFull instead of Valid. I was able to enter 3.2 into my mem test program before I got the -9 error code. So WHY, does ConvValid crap out at 1.06?

[cpp]//Allocate repImg
	Image repImg(srcImg.getSize().width + width_pad * 2, srcImg.getSize().height + height_pad * 2);

	//Repeat borders of imgSrc
	ippiCopyReplicateBorder_32f_C1R(srcImg.getPtr(), srcImg.getStepBytes(), srcImg.getSize(), repImg.getPtr(), repImg.getStepBytes(), repImg.getSize(), height_pad, height_pad);

	//Allocate convImg
	Image convImg(repImg.getSize().width + kerImg.getSize().width - 1, repImg.getSize().height + kerImg.getSize().height - 1);

	IppStatus status = ippiConvFull_32f_C1R(repImg.getPtr(), repImg.getStepBytes(), repImg.getSize(), kerImg.getPtr(), kerImg.getStepBytes(), kerImg.getSize(), convImg.getPtr(), convImg.getStepBytes());[/cpp]

Test for possible IppiConvValid bug

SergeyKostrov — Sat, 18 Aug 2012 01:58:46 GMT

Quoting Ryan

Please see attached for zip...

Received and no errors detected. Thank you.

    Testing ...
    testing: IppiConvValid_mem_test/   OK
    testing: IppiConvValid_mem_test/mem_test/   OK
    testing: IppiConvValid_mem_test/mem_test/mem_test.cpp   OK
    testing: IppiConvValid_mem_test/mem_test/mem_test.vcproj   OK
    testing: IppiConvValid_mem_test/mem_test/mem_test.vcproj.AFX-RYAN.afx-admin.user   OK
    testing: IppiConvValid_mem_test/mem_test/stdafx.cpp   OK
    testing: IppiConvValid_mem_test/mem_test/stdafx.h   OK
    testing: IppiConvValid_mem_test/mem_test/targetver.h   OK
    testing: IppiConvValid_mem_test/mem_test/x64/   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/BuildLog.htm   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/mem_test.exe.intermediate.manifest   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/mem_test.obj   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/mem_test.pch   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/mt.dep   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/stdafx.obj   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/vc90.idb   OK
    testing: IppiConvValid_mem_test/mem_test/x64/Release/vc90.pdb   OK
    testing: IppiConvValid_mem_test/mem_test.ncb   OK
    testing: IppiConvValid_mem_test/mem_test.sln   OK
    testing: IppiConvValid_mem_test/mem_test.suo   OK
    testing: IppiConvValid_mem_test/x64/   OK
    testing: IppiConvValid_mem_test/x64/Release/   OK
    testing: IppiConvValid_mem_test/x64/Release/libiomp5md.dll   OK
    testing: IppiConvValid_mem_test/x64/Release/mem_test.exe   OK
    testing: IppiConvValid_mem_test/x64/Release/mem_test.pdb   OK
    No errors detected in compressed data of C:/Documents and ../Desktop/IppiConvValid_mem_test.zip.

Best regards,
Sergey

Test for possible IppiConvValid bug

Ryan2 — Wed, 22 Aug 2012 16:43:32 GMT

What are the results of your testing?

Test for possible IppiConvValid bug

SergeyKostrov — Mon, 27 Aug 2012 23:18:32 GMT

Quoting Ryan

What are the results of your testing?

Hi Ryan,

I'm sorry for some delay with my tests. I confirm that some internal problem with 'ippiConvValid_32f_C1R' exists. Please take a look at my results:

>> Test-case with Kernel size = 3 <<
     ...
     Enter '0' to quit
     Kernel size relative to input size 1080p: 3
     Input width: 7680, Input height: 4320
     Kernel width: 5761, Kernel height: 3241
     IppStatus code for ippiConvValid_8u_C1R: 0
     ...

>> Here is a summary for different Kernel sizes <<

Kernel size relative to input size 1080p: 1
IppStatus code for ippiConvValid_8u_C1R: 0

Kernel size relative to input size 1080p: 2
IppStatus code for ippiConvValid_8u_C1R: 0

Kernel size relative to input size 1080p: 3
IppStatus code for ippiConvValid_8u_C1R: 0

Kernel size relative to input size 1080p: 4
IppStatus code for ippiConvValid_8u_C1R: -9 ( Error: ippStsMemAllocErr )

Kernel size relative to input size 1080p: 5
IppStatus code for ippiConvValid_8u_C1R: N/A because the test application crashed

Best regards,
Sergey

Test for possible IppiConvValid bug

SergeyKostrov — Mon, 27 Aug 2012 23:18:52 GMT

Quoting Ryan

What are the results of your testing?

Please take a look at a screenshot:

'ippiConvValid_32f_C1R' uses lots of memory, especially when kernel size gets bigger, and it crashes
when it tries to allocate more than 1.65GB of memory.

Best regards,
Sergey

Test for possible IppiConvValid bug

SergeyKostrov — Mon, 27 Aug 2012 23:19:12 GMT

Here is your test-case with my small modifications:

[cpp]#include "stdafx.h"

#include 
#include 

#include "..IPPIncludeippi.h"

void doConvolution( float size );

struct Image
{
private:
	Ipp32f *ptr;
	IppiSize size;
	int stepBytes;

public:
	Image( int width, int height )
	{
		ptr = NULL;
		size.width = width;
		size.height = height;
		stepBytes = -1;

		ptr = ::ippiMalloc_32f_C1( width, height, &stepBytes );
	}

	Ipp32f * getPtr()
	{
		return ptr;
	}

	Ipp32f * getPtr( int x, int y )
	{
		Ipp32f *pixelPtr = ( Ipp32f * )( ( char * )ptr + x + y * size.width );
		return pixelPtr;
	}

	IppiSize getSize()
	{
		return size;
	}

	int getStepBytes()
	{
		return stepBytes;
	}

	~Image()
	{
		if( ptr != NULL )
		{
			::ippiFree( ptr );
			ptr = NULL;
		}
	}
};

int _tmain( int argc, _TCHAR *argv[] )
{
	float size = 0.0f;

	std::cout << "Enter '0' to quit" << std::endl;

	while( true )
	{
		std::cout << "Kernel size relative to input size 1080p: ";
		std::cin >> size;

		if( size == 0.0f )
			break;

		doConvolution( size );
	}

	return 0;
}

void doConvolution( float size )
{
	// Allocate srcImg 1920x1080
	Image srcImg( 1920, 1080 );
	_tprintf( _T("Allocate srcImg 1920x1080 - donen") );

	// Initialize srcImg
	{
		Ipp32f *pixelPtr = NULL;
		for( int y = 0; y < srcImg.getSize().height; y++ )
		{
			pixelPtr = srcImg.getPtr( 0, y );
			for( int x = 0; x < srcImg.getSize().width; x++ )
			{
				*pixelPtr = 100;
				pixelPtr++;
			}
		}
	}
	_tprintf( _T("Initialize srcImg - donen") );

	// Allocate kerImg. Size multiplied by srcImg rounded to next odd integer
	Image kerImg( ( int )ceil( srcImg.getSize().width * size ) | 1, ( int )ceil( srcImg.getSize().height * size ) | 1 );
	_tprintf( _T("Allocate kerImg - donen") );

	// Initialize kerImg
	{
		Ipp32f *pixelPtr = NULL;
		for( int y = 0; y < kerImg.getSize().height; y++ )
		{
			pixelPtr = kerImg.getPtr( 0, y );
			for( int x = 0; x < kerImg.getSize().width; x++ )
			{
				*pixelPtr = 100;
				pixelPtr++;
			}
		}
	}
	_tprintf( _T("Initialize kerImg - donen") );

	// Padding is half of kerSize rounded down to nearest even number
	int width_pad = kerImg.getSize().width >> 1;
	int height_pad = kerImg.getSize().height >> 1;

	// Allocate repImg
	Image repImg( srcImg.getSize().width + width_pad * 2, srcImg.getSize().height + height_pad * 2 );
	_tprintf( _T("Allocate repImg - donen") );

	// Repeat borders of imgSrc
	::ippiCopyReplicateBorder_32f_C1R( srcImg.getPtr(), srcImg.getStepBytes(), srcImg.getSize(),
									   repImg.getPtr(), repImg.getStepBytes(), repImg.getSize(),
									   height_pad, height_pad );

	// Allocate convImg
	Image convImg( srcImg.getSize().width, srcImg.getSize().height );
	_tprintf( _T("Allocate convImg - donen") );

	IppStatus status = ::ippiConvValid_32f_C1R( repImg.getPtr(), repImg.getStepBytes(), repImg.getSize(),
												kerImg.getPtr(), kerImg.getStepBytes(), kerImg.getSize(),
												convImg.getPtr(), convImg.getStepBytes() );

	_tprintf( _T("Convolve - donen") );

	std::cout << "Input width: " << repImg.getSize().width << ", Input height: " << repImg.getSize().height << std::endl;
	std::cout << "Kernel width: " << kerImg.getSize().width << ", Kernel height: " << kerImg.getSize().height << std::endl;
	std::cout << "IppStatus code for ippiConvValid_8u_C1R: " << status << std::endl << std::endl;
}
[/cpp]

Test for possible IppiConvValid bug

Thomas_Jensen1 — Tue, 28 Aug 2012 07:28:32 GMT

It seems that getPtr( int x, int y ) is wrong, since it calculates y * size.width instead of y * stepBytes.

Test for possible IppiConvValid bug

SergeyKostrov — Tue, 28 Aug 2012 23:00:15 GMT

Quoting Thomas Jensen

It seems that getPtr( int x, int y ) is wrong, since it calculates y * size.width instead of y * stepBytes.

You have a working test-case ( Thank you, Ryan ) and you could easily verify it. Please let us know results...

Test for possible IppiConvValid bug

SergeyKostrov — Tue, 28 Aug 2012 23:05:24 GMT

Quoting Ryan

What are the results of your testing?

When the test application crashes these two 'ippi' functions are shown in the Call Stack of a Visual Studio 2005:

     ...
     ippiZigzagFwd8x8_16s_C1()               // crash was here
     ...
     ippiCopy_32f_C1R()                               // it worked
     ...

and Output Window shows

Unhandled exception ... 0xC0000005: Access violation writing location ...