Mac High sierra zdotu test not working

[jagla__bernd]

Hi,

I am struggling with the recompilation of R using the Intel MKL. I have identified the problem to the following:

gfortran -c testf.f 
gcc -c test.c
gcc -L${MKL_LIB_PATH} -o test test.o testf.o -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential  

When running the program, I get a sigfault 11 error message.

I tried debugging with gdb but couldn't really pinpoint the problem. It seems all the good libraries are being used (otool -L) points to the MKL libraries...

I am running out of options...

thanks,

Bernd

[mecej4]

The problems are caused by the fact that your chosen name for the executable, i.e., "test", is the same as the name of a standard Linux/Unix utility, see http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html .

Use a different name for your executable file, or set PATH so that your "test" occurs before the system "test".

[jagla__bernd]

Thanks for the reply but I am calling the program like this:

./test

and the unix test doesn't give a sig-fault...

Any other suggestions?

thanks again

Bernd

[Ying_H_Intel]

Hello 

Could you please see if this command line 

>source  /opt/intel/compilers_and_libraries_2018/mac/mkl/bin/mklvars.sh intel64  (important as environment variable settting) 
>gcc -L${MKL_LIB_PATH} -o test test.o testf.o -lmkl_rt

work?

and here is some articles for your reference : 

https://software.intel.com/en-us/articles/quick-linking-intel-mkl-blas-lapack-to-r​ ; (quick)

​and others: 

Using Intel® MKL with R | Intel® Software

https://software.intel.com/en-us/articles/using-intel-mkl-with-r

and 

https://software.intel.com/en-us/articles/build-r-301-with-intel-c-compiler-and-intel-mkl-on-linux

Best Regards,
​Ying 

[mecej4]

I looked at the two files that you attached. Unless you did something behind the scenes (with compiler options, environment variables, etc.), the integer arguments being passed to the MKL routine are default integers, i.e, 4-byte integers. If this guess is correct, the MKL library you used is not a proper match. You should probably try -lmkl_intel_lp64 instead of -lmkl_intel_ilp64. The "ilp64" library is for 64-bit Integer, Long, Pointer.

[jagla__bernd]

Hi Ying,

Thanks for your help. Here is what I did:

MD18-1021:R-patched2 bernd$ source  /opt/intel/compilers_and_libraries_2018/mac/mkl/bin/mklvars.sh intel64  verbose
DYLD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
NLSPATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/locale/%l_%t/%N
CPATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/include:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/include
PKG_CONFIG_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/bin/pkgconfig:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/bin/pkgconfig
MD18-1021:R-patched2 bernd$ echo $MKL_LIB_PATH 
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/
MD18-1021:R-patched2 bernd$ gcc -L${MKL_LIB_PATH} -o test test.o testf.o -lmkl_rt
MD18-1021:R-patched2 bernd$ otool -L test
test:
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
MD18-1021:R-patched2 bernd$ ./test
Segmentation fault: 11
MD18-1021:R-patched2 bernd$ 

I have started from a clean (new High Sierra) installed system with the following additional installations:

MD18-1021:R-patched2 bernd$ brew list
boost		cmake		eigen		freetype	gdb		glib		gsl		jpeg		libgit2		libmpc		libssh2		libunistring	openssl		pixman		texinfo		xz
cairo		curl		fontconfig	gcc		gettext		gmp		isl		libffi		libidn2		libpng		libtiff		mpfr		pcre		pkg-config	wget

and java jdk

and the other variables (from set):

BASH=/bin/bash
CPATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/include:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/include
CPRO_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac
DISPLAY=/private/tmp/com.apple.launchd.pWYIDgzn7y/org.macosforge.xquartz:0
DYLD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
HOSTTYPE=x86_64
LC_CTYPE=UTF-8
LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/
LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
MACHTYPE=x86_64-apple-darwin17
MKLROOT=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl
MKL_LIB_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/
NLSPATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/locale/%l_%t/%N
OMP_NUM_THREADS=2
OPTERR=1
OPTIND=1
OSTYPE=darwin17
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin
PKG_CONFIG_PATH=/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/bin/pkgconfig:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/bin/pkgconfig
SHELL=/bin/bash

Please let me know if you need additional information.

[jagla__bernd]

another sanity check: I removed all the env variables that gcc used such that compilation and then execution failed and added the info specifically later... same result....

# with nothing compilation fails
MD18-1021:R-patched2 bernd$ gcc  -o test test.o testf.o -lmkl_rt 
ld: library not found for -lmkl_rt
clang: error: linker command failed with exit code 1 (use -v to see invocation)
MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -o test test.o testf.o -lmkl_rt 
# execution fails because LDLIBRARY_PATH is not set
MD18-1021:R-patched2 bernd$ ./test
dyld: Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /Users/bernd/Downloads/R-patched2/./test
  Reason: image not found
Abort trap: 6
MD18-1021:R-patched2 bernd$ echo $MKLROOT
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl
# specifially write library location in executable
MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -Wl,-rpath,${MKLROOT}/lib -o test test.o testf.o -lmkl_rt 
# execution problem...
MD18-1021:R-patched2 bernd$ ./test
Segmentation fault: 11

[jagla__bernd]

and with different library options:

MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -Wl,-rpath,${MKLROOT}/lib -o test test.o testf.o -lmkl_rt 
MD18-1021:R-patched2 bernd$ ./test
Segmentation fault: 11
MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -Wl,-rpath,${MKLROOT}/lib -o test test.o testf.o -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential 
MD18-1021:R-patched2 bernd$ ./test
Segmentation fault: 11
MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -Wl,-rpath,${MKLROOT}/lib -o test test.o testf.o -lmkl_intel_lp64 -lmkl_core -lmkl_sequential 
MD18-1021:R-patched2 bernd$ ./test
Segmentation fault: 11

[mecej4]

The test program runs fine on Linux, and I cannot test on a Mac. You could try compiling with traceback enabled and localize the place where the segfault occurs.

[jagla__bernd]

Hi,

thanks for the effort and time. Here is the gdb output. Please let me know what you want me to do as explicit as possible to avoid misunderstandings ;)

MD18-1021:R-patched2 bernd$ gcc -c test.c -g
MD18-1021:R-patched2 bernd$ gfortran -c testf.f -g
MD18-1021:R-patched2 bernd$ gcc -L$MKL_LIB_PATH -Wl,-rpath,${MKLROOT}/lib -o test test.o testf.o -lmkl_rt -g
MD18-1021:R-patched2 bernd$ gdb ./test
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin17.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...done.
(gdb) r
Starting program: /Users/bernd/Downloads/R-patched2/test 
[New Thread 0x2703 of process 80741]
warning: unhandled dyld version (15)

Thread 2 received signal SIGSEGV, Segmentation fault.
0x00000001083e14dc in ?? ()
(gdb) bt
#0  0x00000001083e14dc in ?? ()
#1  0x0000000000000000 in ?? ()
(gdb) 

[jagla__bernd]

Can it be that the order of architectures is important? (since there is this unhandled dyld version problem)... 

I get for the two libraries that are used:

MD18-1021:R-patched2 bernd$ file /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [i386:Mach-O dynamically linked shared library i386]
/usr/lib/libSystem.B.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
/usr/lib/libSystem.B.dylib (for architecture i386):	Mach-O dynamically linked shared library i386


MD18-1021:R-patched2 bernd$ file /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib 
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib: Mach-O universal binary with 2 architectures: [i386:Mach-O dynamically linked shared library i386] [x86_64]
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64

[jagla__bernd]

For reference the system variables are set like this:

HOSTTYPE=x86_64
MACHTYPE=x86_64-apple-darwin17

[jagla__bernd]

the warning is not important and some gdb/high sierra problem. Hello world works:

Starting program: /Users/bernd/Downloads/R-patched2/test2 
[New Thread 0x2603 of process 1635]
warning: unhandled dyld version (15)
Hello, World![Inferior 1 (process 1635) exited normally]
(gdb) 

[jagla__bernd]

Ok, I understood gdb a bit better... zdotu is causing the problem...

Thread 2 hit Breakpoint 1, main () at test.c:12
12	  F77_SYMBOL(test1)(&iflag);
(gdb) s
test1 (iflag=16850) at testf.f:10
10	      zx(1) = (3.1d0,1.7d0)
(gdb) s
11	      zx(2) = (1.6d0,-0.6d0)
(gdb) s
12	      zres = zdotu(2, zx, 1, zx, 1)
(gdb) s

Thread 2 received signal SIGSEGV, Segmentation fault.
0x00000001082534dc in ?? ()

[mecej4]

Using a Fortran main instead of the C main, and setting the compiler options -g and -fbacktrace, would provide useful information regarding the location of the fault. 

[jagla__bernd]

Currently, I am trying to understand gdb better to see if I can locate the zdotu function, which I cannot at the moment. My Fortran experience is already a few decades old and I it is even less evident to me how to transform the function into a real Fortran program.

Do you know how to list the functions from linked libraries in gdb? I get the following, which leads me to try to verify that the libraries are correctly bound:

MD18-1021:R-patched2 bernd$ nm -add-dyldinfo -no-sort -print-file-name  test
test: 0000000100000000 T __mh_execute_header
test: 0000000100000ca0 T _main
test: 0000000100000cd3 T _test1_
test: 0000000100000cc0 T _xerbla_
test:                  U _cabs
test:                  U _exit
test:                  U _zdotu_
test:                  U dyld_stub_binder

But maybe it is just because the libraries are not compiled with -g...

[jagla__bernd]

I have now statically linked the mkl libraries and get the same error at the same line (zdotu). Since there is no debugging information available I don't think that compiling a new Fortran function would work. 

On the internet, I saw comments about the stack not being sufficient, but when I looked at the offending assembler it was a mov command that caused the problem...

Is there a way to get hold of a mkl library that allows further debugging?

Maybe this has something to do with this problem: when compiling R I can link to the Acceleration library from Apple (MKL is causing the problem above), but I don't think BLAS is working in the performance tests: I can only see one CPU active in the task manager and I don't see any effect greater than a factor of 1.3 when using the Accelerate packages or not. (using disable-blas) If you have any idea on how to verify this side of my problem of getting more performance I would appreciate it a lot as well.

[mecej4]

If Fortran is not your forte, you could consider using C-BLAS and LAPACKE, both of which are supported by MKL, provided the number of BLAS and Lapack routines that you use is small. You would then need no Fortran code at all. See, for example, https://software.intel.com/en-us/mkl-developer-reference-c-cblas-dotu#E81FF20A-C401-4A15-A64F-CC0DAA2BD65D .

I have not used Macintoshes since they switched to OSX and Intel CPUs (~ 20 years ago).

[jagla__bernd]

Just to clarify, I ran into this problem on a very fresh Mac high sierra when trying to recompile R with the MKL support as it was highly recommended to me by several sources. Specifically, the configure process didn't work and I was able to narrow down the problem to the compilation of this example, which I could reproduce. 

I don't see which additional information one could gain by only using Fortran, as I have shown that the zdotu function is causing the problem. If you could further explain this, it would clearly motivate me to go further in this direction.

Thanks a lot for your support,

Kind regards,

Bernd

[jagla__bernd]

I just saw this in the R documentation:

On some systems it has been necessary that an external BLAS/LAPACK was built with the same FORTRAN compiler used to build R.

[mecej4]

jagla, bernd wrote:
I don't see which additional information one could gain by only using Fortran, as I have shown that the zdotu function is causing the problem. If you could further explain this, it would clearly motivate me to go further in this direction.

My suggestion was the exact opposite, i.e., to use no Fortran code at all. MKL gives you complete access to BLAS and Lapack from C, with function prototypes provided to check function arguments. As long as you (well, the parts of the source code of R that use BLAS/Lapack) adhere to these prototypes, you do not need to use a Fortran compiler at all, unless there are pieces of R itself that are written in Fortran (I don't know this).

As a "proof-of-concept" exercise, you could attempt to build R using the system-provided BLAS and Lapack. If that works, you could replace those libraries with MKL. You may be able to just relink, without having to recompile anything.