MAGMA Offload

[Matt_S_]

I'm trying to see how the MIC with MAGMA dgemm compares to CUBLAS on my machine. I installed MAGMA with icc and successfully ran testing_dgemm.cpp. I then changed to my problem size (severely oblong) and again, successfully ran testing_dgemm.cpp. Then, I put in

            #pragma offload target(mic) in(d_A:length(sizeA)) in(d_B:length(sizeB)) out(d_C:length(sizeC))
            magmablas_dgemm( opts.transA, opts.transB, M, N, K, alpha, d_A, ldda, d_B, lddb, beta,  d_C, lddc );

in testing_dgemm.cpp and also edited magmablas_dgemm in magmablas_d.h to

void __attribute__((target(mic))) magmablas_dgemm(
    magma_trans_t transA, magma_trans_t transB,
    magma_int_t m, magma_int_t n, magma_int_t k,
    double alpha,
    magmaDouble_const_ptr dA, magma_int_t ldda,
    magmaDouble_const_ptr dB, magma_int_t lddb,
    double beta,
    magmaDouble_ptr       dC, magma_int_t lddc );

I can successfully compile

icc -O3 -DADD_ -Wall -openmp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY -DHAVE_CUBLAS -DMIN_CUDA_ARCH=100 -I/usr/local/cuda-6.0/include -I/opt/intel/composerxe/mkl/include -I/home/mjswartz/Downloads/magma-1.5.0-beta2/include -I/home/mjswartz/Downloads/magma-1.5.0-beta2/control -c testing_dgemm.cpp -o testing_dgemm.o

but when I try and compile

icc -openmp  testing_dgemm.o -o testing_dgemm libtest.a /home/mjswartz/Downloads/magma-1.5.0-beta2/testing/lin/liblapacktest.a -L/home/mjswartz/Downloads/magma-1.5.0-beta2/lib -lmagma -L/opt/intel/composerxe/mkl/lib/intel64 -L/usr/local/cuda-6.0/lib64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lcublas -lcudart -lstdc++ -lm

I get the following errors:

ld: warning: libimf.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)

I've seen that others have had this problem, which they fixed with sourcing compilervars.sh, but I already do that. Any ideas what's going on?

Thanks in advance!

[TimP]

Might this be a "compiler assisted offload" case, which you would need to look up with MKL Link Advisor?

[Matt_S_]

Tim,

I just entered all of my system info into the Intel® Math Kernel Library Link Line Advisor at https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor and changed my compile to

[mjswartz@pn1249269 testing]$ icc -openmp  testing_dgemm.o -o testing_dgemm libtest.a /home/mjswartz/Downloads/magma-1.5.0-beta2/testing/lin/liblapacktest.a -DMKL_ILP64 -openmp -I$MKLROOT/include -offload-attribute-target=mic -offload-option,mic,compiler," -L$MKLROOT/lib/mic -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread" -L/home/mjswartz/Downloads/magma-1.5.0-beta2/lib -lmagma -L/opt/intel/composerxe/mkl/lib/intel64 -L/usr/local/cuda-6.0/lib64 -L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -lcublas -lcudart

but still get

ld: warning: libimf.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)

[Kevin_D_Intel]

Does LD_LIBRARY_PATH include /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64 ?

[TimP]

It looks like you have tried recommended options.  You may need to consult an expert on MAGMA; there should be one or two at Intel (say, by filing a ticket, if you don't get a response here soon).

If you submit a Premier ticket, I suppose an Intel(r) Xeon Phi(tm) development tools category should be OK.   It's not clear to me whether this is an MKL or an icc issue.

[Matt_S_]

Kevin,

I replaced ":" with "\n" after echo $LD_LIBRARY_PATH to better see all the includes. My LD_LIBRARY_PATH includes

   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4
   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4
   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4
   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4
   /opt/intel/impi/5.0.0.028/intel64/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4
   /usr/local/cuda-6.0/lib64
   /usr/local/cula/lib64
   /usr/local/culasparse/lib64
   /usr/local/cuda-6.0/lib64
   /usr/local/cula/lib64
   /usr/local/culasparse/lib64

Not sure why there's so much redundancy. My ~/.bashrc contains

# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

export CUDA_HOME=/usr/local/cuda-6.0

export CULASPARSE_ROOT=/usr/local/culasparse
export CULASPARSE_INC_PATH=$CULASPARSE_ROOT/include
export CULASPARSE_LIB_PATH_32=$CULASPARSE_ROOT/lib
export CULASPARSE_LIB_PATH_64=$CULASPARSE_ROOT/lib64

export CULA_ROOT=/usr/local/cula
export CULA_INC_PATH=$CULA_ROOT/include
export CULA_LIB_PATH_32=$CULA_ROOT/lib
export CULA_LIB_PATH_64=$CULA_ROOT/lib64

export PATH=$PATH:$CUDA_HOME:$CUDA_HOME/bin:$CUDA_HOME/include:$CULA_INC_PATH

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64:$CULA_LIB_PATH_64:$CULASPARSE_LIB_PATH_64

source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
source /opt/intel/composer_xe_2013_sp1.3.174/bin/compilervars.sh intel64

[Kevin_D_Intel]

I'm not sure the variable setting is the culprit but I don't know what is. The warnings pertain to the CPU (i.e. host) side link not the MIC side which is what I have seen be problematic in the past.

There is a bit of redundancy in the variable setting though. In your .bashrc, compilervars sources the mkl/ipp/tbb{vars} scripts so having both mklvars and compilervars is probably creating some of that.

I don't know what all led to the redundancy but it appears about like the equivalent of source-ing compilervars around five times. I wonder if you could manually re-set the variable's setting to remove some of the redundancy just to see whether that has any impact. I don't think it will. As for compilervars, when source'd it should contribute about this much:

   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64
   /opt/intel/mic/coi/host-linux-release/lib
   /opt/intel/mic/myo/lib
   /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
   /opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.4

[Matt_S_]

Kevin,

Adding in unset LD_LIBRARY_PATH before all the exports in my .bashrc allowed the second compilation (the one that wasn't working). However, the code runs, but hangs.

[Kevin_D_Intel]

Ok, good to hear the compilation succeeded. I'm still unsure what led to the link warnings.

As for the hang, if that seems related to offloading then you could set the env-var OFFLOAD_REPORT=3 and maybe that will offer some info.

[Mark_G_2]

Matt,

It looks like you are using the CUDA version MAGMA, instead of MIC-MAGMA. There is no magmablas_dgemm in the MIC-MAGMA distribution, only in the CUDA version. There is a magma_dgemm in both MAGMA (for CUDA) and MIC-MAGMA. In MAGMA (for CUDA), magma_dgemm is a wrapper around cublas dgemm. In MIC-MAGMA, magma_dgemm is a wrapper around MKL's dgemm on the Xeon Phi.

I'm a little confused what you are trying to do. Do you have both a Xeon Phi and an Nvidia CUDA GPU installed, and you are trying to compare them? MAGMA doesn't have anything that can simultaneously work on both of those. You need to use MAGMA (for CUDA) on an Nvidia GPU, and MIC-MAGMA on the Xeon Phi.

You can get support specifically for MAGMA and MIC-MAGMA on the MAGMA forum. Please specify what platform (CUDA or MIC) you are using when posting questions there.

http://icl.cs.utk.edu/magma/forum/

  -Mark Gates, MAGMA team at Innovative Computing Lab, U. of Tennessee

[Matt_S_]

Mark,

"I'm a little confused what you are trying to do. Do you have both a Xeon Phi and an Nvidia CUDA GPU installed, and you are trying to compare them?" Exactly!

I was unaware of MIC-MAGMA, I'll take a look and see if I can get that to work.

[Matthias_H_Intel]

If it helps - a former colleague of mine has been working very close with MAGMA team (contributing, working with MAGMA development in Australia ...); should be able to get direct contacts to MAGMA team via him

Just let me know

[Matt_S_]

My MKL evaluation has expired and MAGMAMIC will not let me install without a valid license, so it appears that I am dead in the water. Thank you for the help!