Build Problem - OFFLOAD MIC

[Pierpaolo_M_]

Hi,

i am trying to compile a version of my code in which i use OFFLOAD on MIC. I am able to obtain all *.o files but when i try to link them i receive a lot of errors like this:

x86_64-k1om-linux-ld: skipping incompatible /opt/intel/composer_xe_2015.2.164/compiler/lib/intel64/libifport.so.5 when searching for libifport.so.5

My previous code was only MPI-OPENMP without offload. I changed only three files and compiled them using -qoflload=mandatory. I obtain *.o files without any error but then i am not able to link them. 

In attached you can find my whole compilation process and relative errors.

Am I missing some flags in compilation? Is there someone that can give me some insight to solve this problem?

[Kevin_D_Intel]

The errors suggest the offload-side link received and used host-side library paths and those host libraries are not compatible with the Xeon Phi, hence the error. I’m not yet sure why the compiler driver has done just that. I’ll try reproducing this error and see what advice is needed for you to link.

FYI and unrelated to the errors, in general compilation for offload does not require throwing -qoffload. The compiler detects the presence of offload extensions within your source and then triggers the necessary offload compilation/link phases. The flag as you specified instructs the compiler/run-time to force offloading for offload constructs present in your code, which is the default also.

[Kevin_D_Intel]

Actually, the "skipping incompatible" messages are not errors but rather warnings. Those result from explicitly asserting the -l and -L options with explicit host-side paths that are passed to the target-side link; however, that's not what tripped up your build.

What tripped up your build is the error about not finding a libmatmul during the offload link i.e. this error: x86_64-k1om-linux-ld: cannot find -lmatmul

I don't know if that relates to the Intel C++ compiler option, -qopt-matmul. That's unclear since the attached build log only shows mpiifort so I'm not sure why you might be asserting that library for this case unless there's some C/C++ compilations elsewhere that's using that.

You could try removing the -lmatmul and see whether your code links. If you have C/C++ code elsewhere using the related option then removing this likely will produce errors.

Somewhat related, in general when using the compiler driver to drive the link it is not necessary to specify explicit -L/-l options for default libraries that the compiler driver provides itself. You have done that in your case and it is resulting in the numerous "skipping incompatible" messages and the link error.

It might be possible in your case to clean up those messages and errors by removing many of the explicit -L/-l options for the default libs provided by the compiler. I guess what you are able to remove will depend on your app's needs.

[TimP]

Just in case, I'll mention that the implementation of ifort -qopt-matmul seems to have changed significantly in the 16.0 compiler (without corresponding documentation), so you would need to assure that the correct 16.0 libraries are on path if using the newer compilers.

2015.2 is fairly old, there have been several updates in both 2015 and 2016 series.

[Kevin_D_Intel]

I was unaware this was also available with ifort, my apologies. As noted in the both the C++/Fortran UGs, this option is not supported for Xeon Phi™. As with the default set of libraries (e.g. libifcore, libifport, etc.), one should allow the driver to manage inclusion of the appropriate lib and use the -qopt-matmul option instead. The attached build log seems to correlate to a debug build so removing the -lmatmul for that build seems proper.
 
@Tim – Thanks for your clarifying note about -qopt-matmul. Can you please let me know what you found changed in 16.0 w/o corresponding documentation changes and I will follow up on that?

[TimP]

I have by no means a comprehensive view of what changed in recent releases under -qopt-matmul.  However, the old situation was that no auto-parallel effect in matrix multiplication was achieved unless opt-matmul kicked in with calls to external library functions.  If one wanted fully optimized single thread matrix multiplication (possibly with OpenMP threading at a higher level in the loop nests), one had to use -O3 with opt-matmul disabled explicitly.  Now, in 16.0, opt-matmul can produce in-lined code including threading so that cases which aren't big enough to overcome function call overhead are optimized well without having to experiment with opt-matmul on and off.  There may not be a call to the library function even though opt-report says that opt-matmul has inserted a function call.  If there is an actual external call, it may require the new run-time library.  I'm not certain how it works out in offload mode, as I haven't had cases of concern to me where offloaded matmul was competitive.

I mentioned this because a fairly common linking problem occurs when someone installs a new compiler but still links against the old library, possibly because the library path is called out in their build script.  The original post specified that a fairly old library was passed to the linker, but didn't say which compiler version was used to build the objects.  It can get more complicated on MIC due to having to either mount shared object libraries to MIC, and check whether mounts are updated with compiler updates, or copy the shared libraries over, again checking that the matching versions are used.  If offload mode is working correctly, those concerns should relate more to MIC native execution mode.

[Pierpaolo_M_]

Thanks a lot for your answers.

I try to explain better my case, after your very useful suggestions. 

I try starting form my Compiler's version:

mpiifort for the Intel(R) MPI Library 5.0 Update 3 for Linux*
Copyright(C) 2003-2015, Intel Corporation.  All rights reserved.
ifort version 15.0.2

For now, i have this version but i hope that i can buy in the next few months the last version.

In my code i use an external library (Petsc) to solve a linear system with a sparse matrix. Usually to create my own makefile, i start from their suggestions and examples, and from what Kevin wrote there are a lot of unnecessary -L/-l options that they include in their makefile (i think because they don't know in general that these libraries are already available and included in my PATHs, if i use Intel Compiler). So following Kevin suggestions i was able to reduce my error to this more readable one( i have also removed '-qoffload=mandatory' option as suggested):

[pierpaolo@imip1 sourcenew]$ mpiifort -O0 -g -debug inline-debug-info -mcmodel medium -shared-intel -qopenmp -fpp -warn all  MyPetsc.o mpilib.o Openmplib.o MyRand2.o Physics.o Simulation.o Timer.o Particles_mic.o Systems.o Optionals.o Domain_1_mic.o Neutral.o Poisson.o Maxwell_1.o Plasma.o start_1.o output_1.o pic_2_mic.o collision.o main.o  -o 5kW3D -L/home/pierpaolo/Software/petsc/LibPETSc/lib -lpetsc -lHYPRE -lsuperlu_dist_4.3 -lml -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lparmetis -lmetis -lssl -lcrypto -lptesmumps -lptscotch -lptscotcherr -lscotch -lscotcherr -lifport -lifcoremt -lintlc -lrt -lz -ldl -lmpifort -lmpi  -lmpigi -limf -lsvml -lirng -lmatmul -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s
x86_64-k1om-linux-ld: cannot find -lmatmul
x86_64-k1om-linux-ld: cannot find -lmatmul

If i remove '-lmatmul' i receive this error, so i think that Petsc Library, as i configured it, requires in some C/C++ routine this option:

mpiifort -O0 -g -debug inline-debug-info -mcmodel medium -shared-intel -qopenmp -fpp -warn all  MyPetsc.o mpilib.o Openmplib.o MyRand2.o Physics.o Simulation.o Timer.o Particles_mic.o Systems.o Optionals.o Domain_1_mic.o Neutral.o Poisson.o Maxwell_1.o Plasma.o start_1.o output_1.o pic_2_mic.o collision.o main.o  -o 5kW3D -L/home/pierpaolo/Software/petsc/LibPETSc/lib -lpetsc -lHYPRE -lsuperlu_dist_4.3 -lml -lmpicxx -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lparmetis -lmetis -lssl -lcrypto -lptesmumps -lptscotch -lptscotcherr -lscotch -lscotcherr -lifport -lifcoremt -lintlc -lrt -lz -ldl -lmpifort -lmpi  -lmpigi -limf -lsvml -lirng -lipgo -ldecimal -liomp5 -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s
/home/pierpaolo/Software/petsc/LibPETSc/lib/libpetsc.a(dtfe.o): In function `EvaluateFieldJets':
/home/pierpaolo/Software/petsc/include/petsc/private/petscfeimpl.h:225: undefined reference to `matmul_mkl_f64_'
/home/pierpaolo/Software/petsc/include/petsc/private/petscfeimpl.h:225: undefined reference to `matmul_mkl_f64_'
/home/pierpaolo/Software/petsc/LibPETSc/lib/libpetsc.a(plexfem.o): In function `EvaluateFieldJets':
/home/pierpaolo/Software/petsc/include/petsc/private/petscfeimpl.h:225: undefined reference to `matmul_mkl_f64_'

When i configured Petsc, i used these options:

--CFLAGS="-qopenmp -parallel" --CXXFLAGS="-qopenmp -parallel" --FFLAGS="-O3 -xHost -mcmodel medium -shared-intel -qopenmp -parallel -fpp" --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64/ --with-cc=mpiicc --with-cxx=mpiicpc --with-debugger=/opt/intel/composerxe/debugger/gdb/intel64/bin/gdb-ia --with-fc=mpiifort --with-mpiexec="mpiexec.hydra -f ~/mpd.hosts -r ssh"

If I remove  '-qopenmp -parallel' in CFLAGS and CXXFLAGS, do you think is it sufficient to avoid the presence of '-lmatmul' option in their makefile? On the other hand, is it possibile to write a C/C++ option in this configure to avoid '-lmatmul' to appear?Sorry for these, maybe trivial, questions but i don't use C/C++ so i am not an expert on this options.

Thanks in advance

[Kevin_D_Intel]

Thank you for the additional background; good to see you made significant progress.

I think you are still specifying unnecessary default libs but those can be tackled later perhaps. To address the mamul issue, I believe if you remove -lmatmul and then add the compiler option, -qopt-matmul (maybe place it after -qopenmp) for your program, then that might just resolve the matmul reference.   This option is enabled by default if options -O3 and -parallel are used, but you are asserting -O0 currently for your program.

To avoid the need for -lmatmul in relation to the Petsc library, you would remove one or both of -O3 and -parallel, or if you wish to keep those, then add the -qno-opt-matmul (i.e. the negative form) to the Petsc library build options to disable the use of the matmul call.

[Pierpaolo_M_]

Thanks for your reply and for all your help!

I removed -parallel from Petsc configure and everything went fine in my compilation.

[Kevin_D_Intel]

You're welcome. I'm glad to hear that!