Installing NWchem using INTEL compilers + MKL libraries

Rahul_S_4 · ‎10-20-2015

Hello Intel Team,

I am trying to install NWCHEM on the a local university cluster but I am having some problems in compiling NWchem. I am attaching my input script that I’m using to compile Nwchem.

export NWCHEM_TOP=/work/gb_lab/rahulsn/nwchem-6.6

export NWCHEM_TARGET=LINUX64

export ARMCI_NETWORK=OPENIB

export IB_HOME=/usr

export IB_INCLUDE=/usr/include/infiniband

export IB_LIB=/usr/lib64

export IB_LIB_NAME="-libumad -libverbs -lpthread"

export USE_MPI=Y

export USE_MPIF=Y

export USE_MPIF4=Y

export MPI_LOC=/shared/intel/impi_5.0.3

export MPI_INCLUDE="-I/shared/intel/impi_5.0.3/intel64/include"

export MPI_LIB="/shared/intel/impi_5.0.3/intel64/lib/release -L/shared/intel/impi_5.0.3/intel64/lib"

export LIBMPI="-lmpifort -lmpi -lmpigi -ldl -lrt -lpthread"

export NWCHEM_MODULES="all"

export LARGE_FILES=TRUE

export USE_NOFSCHECK=TRUE

export HAS_BLAS=yes

export USE_SCALAPACK=y

export MKLLIB=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64

export MKLINC=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include

export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"

export LAPACK_LIBS="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"

export SCALAPACK="-L$MKLLIB -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"

export SCALAPACK_SIZE=8

export BLAS_SIZE=8

export USE_64TO32=y

export FC=ifort

export CC=icc

make nwchem_config

make CC=icc FC=ifort FOPTIMIZE=-O3 -j4

This is the a part of the error it is showing at the end. I think it is not able to locate SCALPACK libraries on Cyence.

scalapack.F:(.text+0xd19c): undefined reference to `numroc_'

scalapack.F:(.text+0xd1c9): undefined reference to `numroc_'

scalapack.F:(.text+0xd363): undefined reference to `pdsyevd_'

scalapack.F:(.text+0xd719): undefined reference to `numroc_'

scalapack.F:(.text+0xd743): undefined reference to `numroc_'

scalapack.F:(.text+0xd76e): undefined reference to `numroc_'

scalapack.F:(.text+0xd79d): undefined reference to `numroc_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `ga_pdsyevr_':

scalapack.F:(.text+0xdfa8): undefined reference to `blacs_gridinit_'

scalapack.F:(.text+0xdff9): undefined reference to `blacs_gridinfo_'

scalapack.F:(.text+0xe164): undefined reference to `descinit_'

scalapack.F:(.text+0xe1b5): undefined reference to `descinit_'

scalapack.F:(.text+0xe247): undefined reference to `numroc_'

scalapack.F:(.text+0xe26a): undefined reference to `numroc_'

scalapack.F:(.text+0xe3c7): undefined reference to `pdsyevr_'

scalapack.F:(.text+0xe5e8): undefined reference to `pdsyevr_'

scalapack.F:(.text+0xeb21): undefined reference to `numroc_'

scalapack.F:(.text+0xeb4c): undefined reference to `numroc_'

scalapack.F:(.text+0xeb76): undefined reference to `numroc_'

scalapack.F:(.text+0xeba0): undefined reference to `numroc_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `ga_pzheevd_':

scalapack.F:(.text+0xf26a): undefined reference to `blacs_gridinit_'

scalapack.F:(.text+0xf2bb): undefined reference to `blacs_gridinfo_'

scalapack.F:(.text+0xf41e): undefined reference to `descinit_'

scalapack.F:(.text+0xf470): undefined reference to `descinit_'

scalapack.F:(.text+0xf593): undefined reference to `pzheevd_'

scalapack.F:(.text+0xf7db): undefined reference to `pzheevd_'

scalapack.F:(.text+0xfa00): undefined reference to `numroc_'

scalapack.F:(.text+0xfa2c): undefined reference to `numroc_'

scalapack.F:(.text+0xfa58): undefined reference to `numroc_'

scalapack.F:(.text+0xfa84): undefined reference to `numroc_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `ga_pzheevr_':

scalapack.F:(.text+0x10e61): undefined reference to `blacs_gridinit_'

scalapack.F:(.text+0x10eb2): undefined reference to `blacs_gridinfo_'

scalapack.F:(.text+0x1102d): undefined reference to `descinit_'

scalapack.F:(.text+0x1107d): undefined reference to `descinit_'

scalapack.F:(.text+0x11242): undefined reference to `pzheevr_'

scalapack.F:(.text+0x1152d): undefined reference to `pzheevr_'

scalapack.F:(.text+0x11835): undefined reference to `numroc_'

scalapack.F:(.text+0x11864): undefined reference to `numroc_'

scalapack.F:(.text+0x11893): undefined reference to `numroc_'

scalapack.F:(.text+0x118c2): undefined reference to `numroc_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `pxerbla_':

scalapack.F:(.text+0x11a9d): undefined reference to `blacs_gridinfo_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slexit4_':

scalapack.F:(.text+0x11b6b): undefined reference to `blacs_gridexit_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slexit3_':

scalapack.F:(.text+0x11b9b): undefined reference to `blacs_gridexit_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slexit2_':

scalapack.F:(.text+0x11be8): undefined reference to `blacs_gridexit_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slexit_':

scalapack.F:(.text+0x11c0b): undefined reference to `blacs_gridexit_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slinit4_':

scalapack.F:(.text+0x11ddb): undefined reference to `blacs_gridinfo_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slinit3_':

scalapack.F:(.text+0x11ee1): undefined reference to `blacs_gridinfo_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slinit2_':

scalapack.F:(.text+0x120e3): undefined reference to `blacs_gridinit_'

scalapack.F:(.text+0x12134): undefined reference to `blacs_gridinfo_'

/work/gb_lab/rahulsn/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `slinit_':

scalapack.F:(.text+0x1221a): undefined reference to `blacs_gridinit_'

scalapack.F:(.text+0x12247): undefined reference to `blacs_gridinfo_’

It would be really helpful if somebody can help me in preparing my input script. I think I am using the wrong locations of scalpack libraries but the path mentioned in the starting part of the script has all the libraries in the folder. Output of the $MKLLIB folder

libmkl_avx.so libmkl_blacs_openmpi_lp64.a libmkl_gf_ilp64.so libmkl_lapack95_ilp64.a libmkl_vml_avx.so

libmkl_avx2.so libmkl_blacs_sgimpt_ilp64.a libmkl_gf_lp64.a libmkl_lapack95_lp64.a libmkl_vml_avx2.so

libmkl_avx512.so libmkl_blacs_sgimpt_lp64.a libmkl_gf_lp64.so libmkl_mc.so libmkl_vml_avx512.so

libmkl_avx512_mic.so libmkl_blas95_ilp64.a libmkl_gnu_thread.a libmkl_mc3.so libmkl_vml_avx512_mic.so

libmkl_blacs_ilp64.a libmkl_blas95_lp64.a libmkl_gnu_thread.so libmkl_rt.so libmkl_vml_cmpt.so

libmkl_blacs_intelmpi_ilp64.a libmkl_cdft_core.a libmkl_intel_ilp64.a libmkl_scalapack_ilp64.a libmkl_vml_def.so

libmkl_blacs_intelmpi_ilp64.so libmkl_cdft_core.so libmkl_intel_ilp64.so libmkl_scalapack_ilp64.so libmkl_vml_mc.so

libmkl_blacs_intelmpi_lp64.a libmkl_core.a libmkl_intel_lp64.a libmkl_scalapack_lp64.a libmkl_vml_mc2.so

libmkl_blacs_intelmpi_lp64.so libmkl_core.so libmkl_intel_lp64.so libmkl_scalapack_lp64.so libmkl_vml_mc3.so

libmkl_blacs_lp64.a libmkl_def.so libmkl_intel_thread.a libmkl_sequential.a locale

libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.a libmkl_intel_thread.so libmkl_sequential.so

Thanks for the help.

Rahul Singh

Evarist_F_Intel · ‎10-20-2015

Dear Rahul,

May the problem be in the SCALAPACK environment variable: not sure, but shouldn't it be SCALAPACK_LIBS?

The rest looks good. Could you please paste the whole link-line for the file where error happens?

Rahul_S_4 · ‎10-22-2015

Thanks for your reply. I changed the SCALAPACK environment variable to SCALAPACK_LIBS. Now I'm not getting this error but a different one.

===========================================================================

Begin optimization report for: SD_E2_1

Report from: Interprocedural optimizations [ipo]

INLINE REPORT: (SD_E2_1) [23] cr_ccsd_t_E.F(1209,18)

Report from: Loop nest, Vector & Auto-parallelization optimizations [loop, vec, par]

LOOP BEGIN at cr_ccsd_t_E.F(1218,7)

remark #15542: loop was not vectorized: inner loop was already vectorized

LOOP BEGIN at cr_ccsd_t_E.F(1219,7)

remark #25420: Collapsed with loop at line 1220

remark #15542: loop was not vectorized: inner loop was already vectorized

LOOP BEGIN at cr_ccsd_t_E.F(1221,7)

remark #15542: loop was not vectorized: inner loop was already vectorized