Is it safe to call mkl_thread_free_buffers() as first MKL function on a thread?

[djunglas]

Hi there,

I use MKL 10.3.0 on mulitple threads on Windows. Right before a thread finishes I invoke mkl_thread_free_buffers() in order to release all buffers MKL has allocated for this thread.
Sometimes I get the following error from application verifier
vrfcore.dll!VerifierStopMessageEx(_AVRF_LAYER_DESCRIPTOR * LayerDescriptor=0x00396060, unsigned long StopCode=769, unsigned long Param1=4294967295, unsigned long Param2=43962, unsigned long Param3=0, unsigned long Param4=0, _AVRF_STOP_EXTRA * StopExtra=0x00000000, ...) Line 551 C++
vfbasics.dll!CheckTlsIndex(unsigned long Index=4294967295) Line 222 + 0x14 bytes C
vfbasics.dll!AVrfpTlsGetValue(unsigned long dwTlsIndex=4294967295) Line 417 C
mydll.dll!_mkl_serv_mkl_thread_free_buffers() + 0x32 bytes
mydll.dll!_callthreadstartex() Line 348 + 0x6 bytes C
mydll.dll!_threadstartex(void * ptd=0x0809ede8) Line 326 + 0x5 bytes
It seems like this happens when mkl_thread_free_buffers() is the first MKL function I invoke, i.e. when I did not use any MKL function before.
I checked the documentation but did not find any explicit answer, so I wonder whether it is safe to call mkl_thread_free_buffers() as first MKL function? Or will that attempt to access some thread management information that is not set up yet?

Thanks a lot,

Daniel

[barragan_villanueva_]

Hi Daniel,

Thanks for your questions. It looks like it's an unknown problem with mkl_thread_free_buffers().
Could you please create some small testcase to reproduce the problem on our side?

[Julia_S_Intel1]

Thank you, Daniel for your sample program. But I still can't reproduce the issue. Could you please provide more details about your problem: which compiler do you use, how do you link with MKL?

Thanks, Julia

[djunglas]

Here is how I compile/link (I use .bat script under cygwin):

[bash]@echo off

SET MKL=/mkl/ia32/lib/

echo "Compile"
cl -D_CRT_SECURE_NO_DEPRECATE -Gz -MT -W3 -Od -c mklproblem.c

echo "Link"
cl -D_CRT_SECURE_NO_DEPRECATE -Femklproblem.exe mklproblem.obj %MKL%mkl_intel_s.lib %MKL%mkl_sequential.lib %MKL%mkl_core.lib

echo "Run"
mklproblem[/bash]

and here is the output

[bash]"Compile"
Microsoft  32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

mklproblem.c
mklproblem.c(22) : warning C4007: 'main' : must be '__cdecl'
"Link"
Microsoft  32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

Microsoft  Incremental Linker Version 9.00.21022.08
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:mklproblem.exe
mklproblem.obj
/ia32/lib/mkl_intel_s.lib
/ia32/lib/mkl_sequential.lib
/ia32/lib/mkl_core.lib
"Run"

[/bash]

When I run without application verifier then everything is fine. When I run with application verifier I get this backtrace:

[bash]>    ntdll.dll!7c81a251()     
     [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]    
     kernel32.dll!77e424fd()     
     mklproblem.exe!004013a0()     
     vfbasics.dll!AVrfpStandardThreadFunction(void * Context=0x01cd3fe0)  Line 656 + 0x6 bytes    C
     kernel32.dll!77e6482f()     
[/bash]

When I add -DNO_PROBLEM to the compile command then everything is fine.

[Gennady_F_Intel]

Daniel, why are linkingmkl_intel_s.lib instead ofmkl_intel_c.lib? please try with the recommendedmkl_intel_c library first.

[djunglas]

Linking with mkl_intel_c.lib instead of mkl_intel_s.lib gives me

[bash]mklproblem.obj: error LNK2019: unresolved external symbol _MKL_Thread_Free_Buffers@0 referenced in function _threadfunc@4[/bash]

My link command is now

[bash]cl -D_CRT_SECURE_NO_DEPRECATE -Femklproblem.exe mklproblem.obj %MKL%mkl_intel_c.lib %MKL%mkl_sequential.lib %MKL%mkl_core.lib[/bash]

Is that what you wanted me to do or did I mess up?

Thanks,

Daniel

[djunglas]

OK, I am sorry. It seems like I have messed up. Using mkl_intel_c.lib seems to work.
I have slightly modified my example. It now looks like this:

[cpp]#include 
#include 
#include 
#include 
#include 

#ifdef WITH_DGEMM
/* Invoke dgemm with a very simple problem. */
static void
call_dgemm(void)
{
#define ROWS 2
	int m = ROWS, n = ROWS, k = ROWS;
	char transa = 'N';
	char transb = 'N';
	double alpha = 1.0;
	double a[ROWS * ROWS];
	int lda = ROWS;
	double b[ROWS * ROWS];
	int ldb = ROWS;
	double beta = 1.0;
	double c[ROWS * ROWS];
	int ldc = ROWS;
	int i, j;
	for (i = 0; i < ROWS; ++i) {
		for (j = 0; j < ROWS; ++j) {
			a[i * ROWS + j] = (i == j) ? 1.0 : 0.0;
			b[i * ROWS + j] = (i == j) ? 1.0 : 0.0;
			c[i * ROWS + j] = (i == j) ? 1.0 : 0.0;
		}
	}
	printf("Calling dgemm...n");
	dgemm(&transa, &transb, &m, &n, &k, α, a, &lda, b, &ldb, β, c, &ldc);
	printf("dgemm finishedn");
}
#endif /* WITH_DGEMM */

/* Thread worker function. */
static unsigned __stdcall
threadfunc(void *arg)
{
	unsigned int const *action = arg;
	if ( *action ) {
#ifdef WITH_DGEMM
		call_dgemm();
#endif /* WITH_DGEMM */
	} else {
		Sleep(5000);
	}
#ifndef NO_PROBLEM
	mkl_thread_free_buffers();
#endif
	return 0;
}

#define NUM_THREADS 10

int main(void)
{
	HANDLE threads[NUM_THREADS];
	unsigned int action[NUM_THREADS];
	int i;

#ifdef WITH_DGEMM
	call_dgemm();
#endif /* WITH_DGEMM */

	printf("Forking threads ...n");
	for (i = 0; i < NUM_THREADS; ++i) {
		action = i == 0;
		threads = (HANDLE)_beginthreadex(NULL, 0, threadfunc, &action, 0, NULL);
	}
	printf("%d threads forked.n", NUM_THREADS);

	printf("Waiting for threads ...n");
	for (i = 0; i < NUM_THREADS; ++i) {
		WaitForSingleObject(threads, INFINITE);
		CloseHandle(threads);
	}
	printf("%d threads complete.n", NUM_THREADS);
	return 0;
}
[/cpp]

The script I use for building is (the script is still run under cygwin)

[plain]@echo off

del mklproblem.obj
del mklproblem.exe

SET MKL=

echo "Compile"
cl -D_CRT_SECURE_NO_DEPRECATE -arch:SSE2 -MT -Zi -W3 -EHsc -Od -Zm500 -c mklproblem.c

echo "Link"
cl -D_CRT_SECURE_NO_DEPRECATE -Femklproblem.exe mklproblem.obj %MKL%mkl_intel_c.lib %MKL%mkl_sequential.lib %MKL%mkl_core.lib

echo "Run"
mklproblem.exe
[/plain]

I first thought that enabling the code controlled by WITH_DGEMM would make a difference but it does not. I get the same message from application verifier no matter whether I define WITH_DGEMM or not.
When I run the build script I get the following backtrace from application verifier

[bash]>	ntdll.dll!7c81a251() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	vrfcore.dll!VerifierStopMessageEx(_AVRF_LAYER_DESCRIPTOR * LayerDescriptor=0x00396060, unsigned long StopCode=769, unsigned long Param1=4294967295, unsigned long Param2=43962, unsigned long Param3=0, unsigned long Param4=0, _AVRF_STOP_EXTRA * StopExtra=0x00000000, ...)  Line 551	C++
 	vfbasics.dll!CheckTlsIndex(unsigned long Index=4294967295)  Line 222 + 0x14 bytes	C
 	vfbasics.dll!AVrfpTlsGetValue(unsigned long dwTlsIndex=4294967295)  Line 417	C
 	mklproblem.exe!00401142() 	
 	vfbasics.dll!AVrfpTlsGetValue(unsigned long dwTlsIndex=4)  Line 437 + 0x5 bytes	C
 	mklproblem.exe!00401be2() 	
 	mklproblem.exe!004013c0() 	
 	vfbasics.dll!AVrfpStandardThreadFunction(void * Context=0x01cd3fe0)  Line 656 + 0x6 bytes	C
 	kernel32.dll!77e6482f() 	
[/bash]

Like before, if I compile with -DNO_PROBLEM (and therefore do not invoke mkl_thread_free_buffers()) then I do not get any complaints from application verifier. Moreover, when I change the call to mkl_thread_free_buffers() to

[cpp]int buffers = 0;
if ( mkl_mem_stat(&buffers) > 0 || buffers > 0 )
   mkl_thread_free_buffers();[/cpp]

Then again everything is fine (again no matter whether I compile with -DWITH_DGEMM or not).
So I am pretty sure that the problem is somewhere inside mkl_thread_free_buffers().

Daniel

PS: Sorry for the messy way I in which I describe how to reproduce the problem. I am not too familiar with building stuff on Windows.

[Julia_S_Intel1]

Thank you Daniel for your detailed description. I reproduced the problem, working on it now.

[Gennady_F_Intel]

Daniel -the problem has been escalated. Here is a bug tracking number for your reference#DPD200214530

We will let you know when the fix would available.

[djunglas]

Thanks a lot.
What about the workaround I described: First test with mkl_mem_stat() whether there is anything to free and only then call mkl_thread_free_buffers()? Is this a valid workaround or was I just lucky?

[Julia_S_Intel1]

Thats the valid workaround. The problem rises only if you call mkl_thread_free_buffers() when theres no memory used by MKL to free.

[Gennady_F_Intel]

Hello Daniel,

Please check the fix of the problem you reported with the latest Update 5 ( Intel MKL v.10.3.5).

--Gennady