MKL/OpenMp and MS Visual Studio (C++) strange behavior

Robert_Adams · ‎03-11-2010

Hi,

We are trying to use MKL with Microsoft's Visual Studio C++ and are having the following problem.

In some cases when we make lapack calls to MKL we are not getting the right answer or the call hangs. We have noticed that when the MKL calls are in an OpenMP parallel region or OpenMP is completely turned at the compiler level, this never happens.

I am assuming this is a lapack threading issue since I can create a small parallel region and see the problem appear / disappear as I move the lapack call out of / in to the OpenMP parallel region. When the call is in the parallel region we are guessing that the MKL lapack threading turns off. We also notice that if the lapack call is in a sequential region, then the problem occurs only for matrices larger than 100-200 elements.

One more piece of the puzzle is that if in VS2005 we set the compiler to ignore the default library VCOMP then the problem never arises. This makes us think that there is a conflict between the Microsoft vcomp and MKL libiomp5md. We would like to always ignore VCOMP, but when we set VS2008 to ignore VCOMP we get the link error

error LNK2001: unresolved external symbol _You_must_link_with_Microsoft_OpenMP_library

This symbol is defined in libiomp5md. We do not understand why libiomp5md and VS2008 force us to link with vcomp when VS2005 does not?

Does anyone have some tips to resolve this problem that will work with both VS2005 and VS2008. We have observed this problem on Windows Vista 64 (dual core) using both MKL 10.0.5.025 and 10.1.0.018. We have encountered this problem with the lapack routines DSPTRF and DGETRF.

Thanks,

J

Robert_Adams · ‎03-11-2010

"Does anyone have some tips to resolve this problem that will work with both VS2005 and VS2008. We have observed this problem on Windows Vista 64 (dual core) using both MKL 10.0.5.025 and 10.1.0.018. We have encountered this problem with the lapack routines DSPTRF and DGETRF."

As an updated, this problem does not occur with DGETRF but only DSPTRF. I was mistaken about that in my original post.

Thanks,
J

TimP · ‎03-11-2010

Yes, linking with more than one copy of OpenMP library, including the case of linking vcomp when using MKL thread option, may be expected to cause problems. If you have any use of OpenMP in functions compiled with VC, and wish also to use objects built for Intel OpenMP, you must link libiomp5 explicitly in place of vcomp. libiomp5 takes care of all the Microsoft OpenMP references, including the "must link with Microsoft OpenMP" which is added to any .obj built with VC under /openmp. MKL thread library will have references which can't be handled correctly in the presence of vcomp. If you want to use vcomp, you must link mkl_sequential.

If you call MKL functions from a threaded region, MKL threading should be disabled unless you set OMP_NESTED. Without OMP_NESTED, the effect is much the same as it would be with mkl_sequential.

MKL_thread has equivalent of omp parallel if() clauses to limit it to 1 thread for data sets which aren't large enough to benefit from multiple threads. There again, you get much the same result as mkl_sequential, but with more overhead. So, you apparently get lucky and skip OpenMP function calls which aren't compatible with vcomp.

Robert_Adams · ‎03-11-2010

Quoting tim18

Yes, linking with more than one copy of OpenMP library, including the case of linking vcomp when using MKL thread option, may be expected to cause problems. If you have any use of OpenMP in functions compiled with VC, and wish also to use objects built for Intel OpenMP, you must link libiomp5 explicitly in place of vcomp. libiomp5 takes care of all the Microsoft OpenMP references, including the "must link with Microsoft OpenMP" which is added to any .obj built with VC under /openmp. MKL thread library will have references which can't be handled correctly in the presence of vcomp. If you want to use vcomp, you must link mkl_sequential.

Hi Tim,

Thanks. We do not want to use vcomp and we are linking libiomp5 explicitly (in VS2008). However, the linker keeps giving us the error about requiring Microsof's OpenMp.

Here is our link line which excludes vcomp and adds libiomp5md.lib

/OUT:"x64\Release/app.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Program Files\Intel\MKL\10.0.5.025\em64t\lib" /MANIFEST /MANIFESTFILE:"x64\Release\app.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /NODEFAULTLIB:"vcomp" /DEBUG /PDB:"d:\test\x64\Release\app.pdb" /SUBSYSTEM:CONSOLE /OPT:REF /OPT:ICF /DYNAMICBASE:NO /FIXED:No /MACHINE:X64 /ERRORREPORT:PROMPT mkl_solver_ilp64.lib mkl_intel_ilp64.lib mkl_intel_thread.lib mkl_core.lib libiomp5md.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib "..\linear_algebra\x64\release\linear_algebra.lib"

which produces the error

error LNK2001: unresolved external symbol _You_must_link_with_Microsoft_OpenMP_library

VS2005 does not give this error when we exclude vcomp.

Any ideas?

Thanks,

John

Robert_Adams · ‎03-11-2010

Hi,

I attached a short Visual Studio 2008 /Visual C++ 2008 that exhibits the problem I am having. I ran this under Win32/Release. If the linker options are set to ignore vcomp, then the compiler errors with

_You_must_link_with_Microsoft_OpenMP_library

If the linker options are set to not ignore vcomp, then compilation succeeds. Interestingly, if the compiler is set to ignore the vcomp library and the line

#pragma omp parallel for reduction( + : abc )

is removed, then the compile succeeds.

Thanks,

J

TimP · ‎03-11-2010

I see that it misbehaves when I use cl /openmp to drive the link (because vcomp is linked alongside libiomp5), but it's OK when I link with /openmp shut off, even though OpenMP is active in the .obj. In a project, I suppose you must either change the option when making the link, or exclude vcomp by

-nodefaultlib:vcomp.lib
-nodefaultlib:vcompd.lib

as ICL does. Beyond that, after changing paths to conform with my more recent MKL installation, I don't see the problems you report. You don't actually use MKL in the example, so those libraries aren't needed.

Robert_Adams · ‎03-12-2010

Tim,

Thanks for your help. We finally solved the problem. It was an issue with the order of link libraries. Our program required the linking of a main driver, several of our own libraries, and the mkl libraries. Our own libraries had mkl/openmp dependencies. If our own libraries were placed before the mkl libraries on the link line, then only libiomp5md was linked in, but if our own libraries were placed after the mkl libraries on the link line then both libiomp5md and vcomp were being linked in. In Visual Studio, if the Link->Link Library Dependencies option was set to yes, then VS was placing the mkl libraries before our libraries on the link line. The solution was to list our libraries explicitly in the Linker->Comman Line->Additional Options box. This made VS place our libraries first on the link line.

J