Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.
7074 Discussions

MKL trouble with HPCC

wilshire461
Beginner
‎10-11-2011 07:49 PM
2,218 Views
I'm trying to compile hpcc using the mkl as the LA library but am running up against a roadblock.

This is what I am getting:

[plain][15:41] user@NODE-00/lustre/user/hpcc-1.4.1 make arch=test
mkdir hpl/lib/rcac
mkdir: cannot create directory `hpl/lib/rcac': File exists
make: [all] Error 1 (ignored)
( cd hpl/lib/arch/build ; make arch=rcac -f Makefile.hpcc )
make[1]: Entering directory `/lustre/user/hpcc-1.4.1/hpl/lib/arch/build'
mpicc  -DHPL_CALL_CBLAS -I../../../include -I../../../include/rcac /opt/intel/mkl/include -I/usr/mpi/gcc/mvapich2-1.6/include -fomit-frame-pointer -O3 -funroll-loops -W -Wall -o ../../../../hpcc ../../../lib/rcac/libhpl.a  /opt/intel/mkl/lib/intel64/libmkl_core.a  /usr/mpi/gcc/mvapich2-1.6/lib/libmpich.so -l /usr/mpi/gcc/mvapich2-1.6/include/librdmacm.so
/opt/intel/mkl/include: file not recognized: Is a directory
collect2: ld returned 1 exit status
make[1]: *** [../../../../hpcc] Error 1
make[1]: Leaving directory `/lustre/user/hpcc-1.4.1/hpl/lib/arch/build'
make: *** [all] Error 2[/plain]


Some bits from the make file

[plain]#
ARCH         = $(arch)
#
# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - MPI directories - library ------------------------------------------
# ----------------------------------------------------------------------
# MPinc tells the  C  compiler where to find the Message Passing library
# header files,  MPlib  is defined  to be the name of  the library to be
# used. The variable MPdir is only used for defining MPinc and MPlib.
#
MPdir        = /usr/mpi/gcc/mvapich2-1.6
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.so
#
# ---------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
# LAinc tells the  C  compiler where to find the Linear Algebra  library
# header files,  LAlib  is defined  to be the name of  the library to be
# used. The variable LAdir is only used for defining LAinc and LAlib.
#
LAdir        = /opt/intel/mkl/lib/intel64
LAinc        = /opt/intel/mkl/include
LAlib        = -Wl,--start-group $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_intel_thread.a $(LAdir)/libmkl_core.a -Wl, --en
d-group -lpthread -lm
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)                                                                 
#                                                                                                          
# - Compile time options -----------------------------------------------                                   
#                                                                                                          
# -DHPL_COPY_L           force the copy of the panel L before bcast;                                       
# -DHPL_CALL_CBLAS       call the cblas interface;                                                         
# -DHPL_CALL_VSIPL       call the vsip  library;                                                           
# -DHPL_DETAILED_TIMING  enable detailed timers;                                                           
#                                                                                                          
# By default HPL will:                                                                                     
#    *) not copy L before broadcast,                                                                       
#    *) call the Fortran 77 BLAS interface                                                                 
#    *) not display detailed timing information.                                                           
#                                                                                                          
HPL_OPTS     = -DHPL_CALL_CBLAS                                                                            
#                                                                                                          
# ----------------------------------------------------------------------                                   
#                                                                                                          
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)                                                      
#                                                                                                          
# ----------------------------------------------------------------------                                   
# - Compilers / linkers - Optimization flags ---------------------------                                   
# ----------------------------------------------------------------------                                   
#                                                                                                          
CC           = mpicc                                                                                       
CCNOOPT      = $(HPL_DEFS)                                                                                 
#                                                                                                          
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall                                
#                                                                                                          
LINKER       = mpicc                                                                                       
LINKFLAGS    = $(CCFLAGS)                                                                                  
#                                                                                                          
ARCHIVER     = ar                                                                                          
ARFLAGS      = r                                                                                           
RANLIB       = echo                                                                                        
#                                                                                                          
# ----------------------------------------------------------------------
[/plain]

Any advice woul be much appreciated.
0 Kudos

Link Copied

12 Replies
mecej4
Honored Contributor III
‎10-11-2011 07:55 PM
2,218 Views
It appears to me that this line in the makefile

HPL_INCLUDES=-I$(INCdir)-I$(INCdir)/$(ARCH)$(LAinc)$(MPinc)

should, instead, have been

HPL_INCLUDES=-I$(INCdir)-I$(INCdir)/$(ARCH) -I$(LAinc) -I$(MPinc)

Without those -I prefixes, Make will issue a compile command with directory names in a context where source files are expected by the compiler. That is what is seen in the compiler error message.
0 Kudos
Copy link
wilshire461
Beginner
‎10-11-2011 08:26 PM
2,218 Views
I made the changes you suggested and did a clean make but am still getting the same error
0 Kudos
Copy link
Vladimir_Petrov__Int
New Contributor III
‎10-11-2011 09:15 PM
2,218 Views
Please make sure the changes you made are consistent. Namely you should have only added "-I" to the LAinc dir, since MPinc already had this prefix (MPinc=-I$(MPdir)/include).

Also this link may help you avoid some problems
http://software.intel.com/en-us/articles/performance-tools-for-software-developers-use-of-intel-mkl-in-hpcc-benchmark/

Best regards,
Vladimir
0 Kudos
Copy link
wilshire461
Beginner
‎10-12-2011 07:25 AM
2,218 Views
The change was made but still running into the same error. I also went ahead and removed the LAinc reference entirely just leaving it blank as we do this for our other clusters as it isn't needed but was still getting the errors referencing /opt/intel/mkl/include. So where exactly does the makescript pull that from if it is no where in the makefile?
0 Kudos
Copy link
wilshire461
Beginner
‎10-12-2011 08:23 AM
2,218 Views
So in going back through the guide that was linked Build MPI MKL FFTW library that the make file for that does not include an option to make the library against mvapich? Is there a way to make against mvapich or should I just give up on trying to use mvapich?
0 Kudos
Copy link
wilshire461
Beginner
‎10-12-2011 12:11 PM
2,218 Views
Okay so I realized that when you try and make hpcc it goes into the hpl directory as well and uses that make file to put together hpl. So I made a copy of the make file and put it into the HPL directory. I am however now getting the following error:
[plain]mpicc  -DHPL_CALL_CBLAS -I../../../include -I../../../include/test   -fomit-frame-pointer -O3 -funroll-loops -W -Wall  -o ../../../../hpcc ../../../lib/test/libhpl.a  -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/mkl/lib/intel64/libmkl_intel_thread.a /opt/intel/mkl/lib/intel64/libmkl_core.a -Wl, --end-group -lpthread -lm /usr/mpi/gcc/mvapich2-1.6/lib/libmpich.so
/usr/bin/ld: : No such file: No such file or directory
collect2: ld returned 1 exit status
make[1]: *** [../../../../hpcc] Error 1
make[1]: Leaving directory `/lustre/user/hpcc-1.4.1/hpl/lib/arch/build'
make: *** [all] Error 2[/plain]
0 Kudos
Copy link
Vladimir_Petrov__Int
New Contributor III
‎10-12-2011 12:21 PM
2,218 Views
Quoting wilshire461
So in going back through the guide that was linked Build MPI MKL FFTW library that the make file for that does not include an option to make the library against mvapich? Is there a way to make against mvapich or should I just give up on trying to use mvapich?

Steps to build wrappers for/link with MVAPICH2 should be the same as for MPICH2.

Best regards,
Vladimir
0 Kudos
Copy link
Vladimir_Petrov__Int
New Contributor III
‎10-12-2011 12:26 PM
2,218 Views
Quoting wilshire461
Okay so I realized that when you try and make hpcc it goes into the hpl directory as well and uses that make file to put together hpl. So I made a copy of the make file and put it into the HPL directory. I am however now getting the following error:
- collapse sourceview plaincopy to clipboardprint?
  1. mpicc-DHPL_CALL_CBLAS-I../../../include-I../../../include/test-fomit-frame-pointer-O3-funroll-loops-W-Wall-o../../../../hpcc../../../lib/test/libhpl.a-Wl,--start-group/opt/intel/mkl/lib/intel64/libmkl_intel_lp64.a/opt/intel/mkl/lib/intel64/libmkl_intel_thread.a/opt/intel/mkl/lib/intel64/libmkl_core.a-Wl,--end-group-lpthread-lm/usr/mpi/gcc/mvapich2-1.6/lib/libmpich.so
  2. /usr/bin/ld::Nosuchfile:Nosuchfileordirectory
  3. collect2:ldreturned1exitstatus
  4. make[1]:***[../../../../hpcc]Error1
  5. make[1]:Leavingdirectory`/lustre/user/hpcc-1.4.1/hpl/lib/arch/build'
  6. make:***[all]Error2
[plain]mpicc  -DHPL_CALL_CBLAS -I../../../include -I../../../include/test   -fomit-frame-pointer -O3 -funroll-loops -W -Wall  -o ../../../../hpcc ../../../lib/test/libhpl.a  -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/mkl/lib/intel64/libmkl_intel_thread.a /opt/intel/mkl/lib/intel64/libmkl_core.a -Wl, --end-group -lpthread -lm /usr/mpi/gcc/mvapich2-1.6/lib/libmpich.so
/usr/bin/ld: : No such file: No such file or directory
collect2: ld returned 1 exit status
make[1]: *** [../../../../hpcc] Error 1
make[1]: Leaving directory `/lustre/user/hpcc-1.4.1/hpl/lib/arch/build'
make: *** [all] Error 2[/plain]

Please eliminate the extra space in "-Wl,--end-group".

Best regards,

Vladimir

0 Kudos
Copy link
wilshire461
Beginner
‎10-12-2011 01:17 PM
2,218 Views
Yeah I caught that, still no dice though as now I am getting a bunch of errors with the compile not being able to pull math functions from the library and such. Also when I try to put together the fftw2x_cdft it complains about mpicc not being found however it is installed and working fine as per the code below:

[plain][09:08] user@NODE-00/opt/intel/mkl/interfaces/fftw2x_cdft mpicc
Error: Command line argument is needed!
Usage: mpicc [options] file....

Available mpicc options:
    -echo         - Show exactly what this program is doing.
                    This option should normally not be used.
    -show         - Show the commands that would be used without runnning them.
    -compile-info - Show how to compile a program.
    -link-info    - Show how to link a program.
    -v            - Version info of mpicc and its native compiler gcc.
    -help         - Give this help message.
    -mpe=XXX      - Build with MPE option, XXX.
                    See "mpicc -mpe=help" for more information.
    -profile=XXX  - Use profiling configuration file,
                    /usr/mpi/gcc/mvapich2-1.6/etc/XXX.conf
                    As a special case, libXXX.so or libXXX.la may be used
                    if the library is in /usr/mpi/gcc/mvapich2-1.6/lib.
    -config=XXX   - Use configuration file, mpicc-XXX.conf.
    -cc=xxx       - Reset the native compiler to xxx.
[09:09] user@NODE-00/opt/intel/mkl/interfaces/fftw2x_cdft sudo make libintel64 mpi=mpich2 PRECISION=MKL_SINGLE interface
=ilp64
make lib _IA=intel64 Ibin=bin64
make[1]: Entering directory `/opt/intel/composer_xe_2011_sp1.6.233/mkl/interfaces/fftw2x_cdft'
rm -rf obj
rm -f ../../lib/intel64/libfftw2x_cdft_SINGLE_ilp64.a
mkdir -p obj
mpicc -cc=icc -DMKL_ILP64  -c -w -DMKL_SINGLE -I../../include 
	  wrappers/1d_create_plan.c -o obj/1d_create_plan.o
make[1]: mpicc: Command not found
make[1]: *** [obj/1d_create_plan.o] Error 127
make[1]: Leaving directory `/opt/intel/composer_xe_2011_sp1.6.233/mkl/interfaces/fftw2x_cdft'
make: *** [libintel64] Error 2[/plain]
Here are some of the errors I am now getting:

[bash]mpicc  -DHPL_CALL_CBLAS -I../../../include -I../../../include/rcac   -fomit-frame-pointer -O3 -funroll-loops -W -Wall  -o ../../../../hpcc ../../../lib/rcac/libhpl.a  /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.a  /usr/mpi/gcc/mvapich2-1.6/lib/libmpich.so
../../../lib/rcac/libhpl.a(io.o): In function `HPCC_ProcessGrid':
io.c:(.text+0x91): undefined reference to `sqrt'
../../../lib/rcac/libhpl.a(io.o): In function `HPCC_Defaults':
io.c:(.text+0x3504): undefined reference to `sqrt'
io.c:(.text+0x380e): undefined reference to `sqrt'
../../../lib/rcac/libhpl.a(tstfft.o): In function `HPCC_TestFFT':
tstfft.c:(.text+0x465): undefined reference to `log'
tstfft.c:(.text+0x4b7): undefined reference to `log'
tstfft.c:(.text+0x4ca): undefined reference to `log'
../../../lib/rcac/libhpl.a(mpifft.o): In function `HPCC_MPIFFT':
mpifft.c:(.text+0x8d3): undefined reference to `log'
mpifft.c:(.text+0x916): undefined reference to `log'
../../../lib/rcac/libhpl.a(mpifft.o):mpifft.c:(.text+0x929): more undefined references to `log' follow
../../../lib/rcac/libhpl.a(HPL_pdtest.o): In function `HPL_pdtest':
HPL_pdtest.c:(.text+0x743): undefined reference to `cblas_dgemv'
HPL_pdtest.c:(.text+0xb4b): undefined reference to `cblas_idamax'
../../../lib/rcac/libhpl.a(pdmatcmp.o): In function `pdmatcmp':
pdmatcmp.c:(.text+0xc4): undefined reference to `cblas_idamax'
pdmatcmp.c:(.text+0x117): undefined reference to `cblas_daxpy'
pdmatcmp.c:(.text+0x130): undefined reference to `cblas_idamax'
../../../lib/rcac/libhpl.a(pdtrans.o): In function `pdtrans':
pdtrans.c:(.text+0x3b7): undefined reference to `cblas_dscal'
pdtrans.c:(.text+0x302d): undefined reference to `cblas_daxpy'
pdtrans.c:(.text+0x312e): undefined reference to `cblas_daxpy'
pdtrans.c:(.text+0x31ed): undefined reference to `cblas_dcopy'
pdtrans.c:(.text+0x32e0): undefined reference to `cblas_dcopy'
../../../lib/rcac/libhpl.a(bench_lat_bw_1.5.2.o): In function `cross_ping_pong_controlled':
bench_lat_bw_1.5.2.c:(.text+0x129e): undefined reference to `sqrt'
[/bash]
0 Kudos
Copy link
Vladimir_Petrov__Int
New Contributor III
‎10-12-2011 07:54 PM
2,218 Views

mpicc is most probably not found because sudo does not preserve the PATH variable. You can check this by running something like "sudo printenv PATH".

In order for this to be fixed you need to

either pass the path to the MPI directory via the mpidir parameter of the wrapper's makefile,

or take special actions to preserve the PATH by sudo.

The second series of errors is because you neither linked in libm nor MKL (you have only the threading layer added). The whole group of MKL libraries must be added to the link line.

Best regards,

Vladimir

0 Kudos
Copy link
wilshire461
Beginner
‎10-14-2011 07:16 AM
2,218 Views
I am now getting new and more interesting errors. I managed to get both of the fft bits compiled as suggested by the guide but funnily enough I get the following errors
[plain]In file included from ../../../../FFT/hpccfft.h:43,
                 from ../../../../FFT/bcnrand.c:46:
../../../../FFT/wrapfftw.h:4:18: error: fftw.h: No such file or directory
In file included from ../../../../FFT/hpccfft.h:43,
                 from ../../../../FFT/bcnrand.c:46:
../../../../FFT/wrapfftw.h:21: error: expected specifier-qualifier-list before fftw_complex
../../../../FFT/wrapfftw.h:28: error: expected declaration specifiers or ... before fftw_direction
../../../../FFT/wrapfftw.h:30: error: expected declaration specifiers or ... before fftw_complex
../../../../FFT/wrapfftw.h:30: error: expected declaration specifiers or ... before fftw_complex
In file included from ../../../../FFT/bcnrand.c:46:
../../../../FFT/hpccfft.h:47: error: expected declaration specifiers or ... before fftw_complex
../../../../FFT/hpccfft.h:47: error: expected declaration specifiers or ... before fftw_complex
../../../../FFT/hpccfft.h:48: error: expected ) before * token
../../../../FFT/hpccfft.h:49: error: expected ) before * token
make[1]: *** [../../../../FFT/bcnrand.o] Error 1
[/plain]
Which seems an awful lot like something didn't get compiled correctly. Any thoughts?

Note: I got ATLAS compiled so for the meantime using that as my library and had no problems getting hpcc compiled using that, so for the time being this is an exercise in curiosity while I go forward and do science with the math library that does work.
0 Kudos
Copy link
mecej4
Honored Contributor III
‎10-14-2011 03:37 PM
2,218 Views
The error message is explicit:

../../../../FFT/wrapfftw.h:4:18:error:fftw.h:Nosuchfileordirectory

The required include file is in the /include/fftw directory, to use which you have to add -I/include/fftw to the compiler options.
0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.