Help Compiling HDF5 parallel and h5py parallel

[bukowsky__colton]

I am trying to compile HDF5 with Intel mpi. It is needed by some other software that I am compiling for my Intel Distribution conda python environment. I am trying to use conda to contain everything -- I think the important details are below. I probably have some redundant variable declarations because I have tried SO many combinations:

export NPROCS=4
export SRCBUILDDIR=$HOME/some/tmp/build/dir
export INSTALL_PATH=$HOME/anaconda3/envs/HDF5env
export INTEL_ROOT="${HOME}/intel/compilers_and_libraries_2018.3.222/linux"
export INTEL_BIN="${INTEL_ROOT}/bin/intel64"
export INTEL_LICENSE_FILE=${INTEL_ROOT}/licenses:${HOME}/intel/licenses
export INTEL_MPI_ROOT=${INTEL_ROOT}/mpi
export MKLROOT="${INTEL_ROOT}/mkl"
export MKLPATH="${MKLROOT}/lib/intel64"
export PATH="${MKLROOT}/bin:${INTEL_MPI_ROOT}/intel64/bin:${INTEL_BIN}:${PATH}"
# I am sure that I am doubling up on some env vars by calling the scripts below
export COMPILERVARS_ARCHITECTURE="intel64"
export COMPILERVARS_PLATFORM="linux"
${INTEL_ROOT}/bin/compilervars.sh
${INTEL_ROOT}/mkl/bin/mklvars.sh
${INTEL_ROOT}/mpi/bin64/mpivars.sh
# Not sure if this is the proper way to do this, but it works with my other code that needs MKL
MKLLIBS="-Wl,--no-as-needed,-L${MKLROOT}/lib/intel64 -lmkl_rt -ldl"
export CPATH="$CPATH:${INTEL_ROOT}/mkl/include:${INTEL_ROOT}/mpi/intel64/include:${INTEL_ROOT}/compiler/include:${MEEP_INSTALL_PATH}/include:${MEEP_INSTALL_PATH}/include/gsl:${MEEP_INSTALL_PATH}/lib/python3.6/site-packages/mpi4py/include:${MEEP_INSTALL_PATH}/lib/python3.6/site-packages/numpy/core/include/numpy"
export LIBRARY_PATH="${INSTALL_PATH}/lib:${INTEL_MPI_ROOT}/intel64/lib:${INTEL_MPI_ROOT}/mic/lib:${INTEL_ROOT}/compiler/lib/intel64_lin:"
export LD_LIBRARY_PATH="${LIBRARY_PATH}:$INSTALL_PATH/lib"
export LD_RUN_PATH="${LIBRARY_PATH}:$INSTALL_PATH/lib"

# Linkers and Compilers
export AR=${INTEL_BIN}/xiar
export LD=${INTEL_BIN}/xild
export CC=mpiicc
export CXX=mpiicpc
export MPICC=mpiicc
export MPICXX=mpiicpc
export F77=mpiifort
export F9X=mpiifort
export FC=mpiifort
export I_MPI_CC=icc
# Flags
export CFLAGS="-O3 -no-prec-div -ip -ipo -march=core-avx2 -xHost -xCORE-AVX2 -m64 -shared-intel ${MKLLIBS}"
export CXXFLAGS="${CFLAGS}"
export FFLAGS="-fPIC ${CFLAGS}"
export LDFLAGS="$MKLLIBS"

export HDF5_MPI="ON"
export I_MPI_SHM_LMT=shm
set +e
if [ "$COMPILERS" = "intel" ];
then
# The -fp-models are helpful for the other code I am compiling, thought they would be useful here...
CFLAGS="-ip -ipo $CFLAGS -fp-model source -fp-model strict";
fi
cd $SRCBUILDDIR
wget http://www.zlib.net/zlib-1.2.11.tar.gz
rm -r zlib-1.2.11 && tar -zxvf zlib-1.2.11.tar.gz || tar -zxvf zlib-1.2.11.tar.gz
rm zlib-1.2.11.tar.gz
cd zlib-1.2.11
CFLAGS="$CFLAGS" ./configure --prefix=${SRCBUILDDIR}/zlib-1.2.11
make
make check
make install

# Download hdf5 from here
HDFVERSION="hdf5-1.10.2"
cd $SRCBUILDDIR
wget "https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/$HDFVERSION/src/$HDFVERSION.tar.bz2"
rm -r "$HDFVERSION" && tar -xjf "$HDFVERSION.tar.bz2" || tar -xjf "$HDFVERSION.tar.bz2"
export RUNSERIAL="mpirun -np 1"
export RUNPARALLEL="mpirun -n 4"
export I_MPI_SHM_LMT=shm

### BUILD HDF5 Parallel
cd $HDFVERSION
CFLAGS="$CFLAGS" ./configure --prefix="$INSTALL_PATH" --enable-parallel --enable-shared \
 --with-zlib="$INSTALL_PATH" --enable-static --enable-fortran --enable-hl
make -j $NPROCS
# Most of the tests pass, but some don't and hang up the script
# make -k -j $NPROCS check
make -j $NPROCS install

# The only way I have found to force pip to use Intel Compilers is to set the LDSHARED var inline
LDSHARED="mpiicc -fp-model source -fp-model strict $CFLAGS -L{$MEEP_INSTALL_PATH} -lhdf5 \
-I${INSTALL_PATH}/include" CPPFLAGS="-E $CPPFLAGS" HDF5_DIR=${MEEP_INSTALL_PATH} \
pip -v install --force-reinstall --no-deps --ignore-installed --no-cache-dir --no-binary=h5py h5py;


# ........ Compile some more code that uses parallel HDF5 and parallel h5py below ..........
# for example, gsl and the MEEP suite

Everything compiles, MOST of the hdf5 tests pass, I will edit or update when I go back to look up where it fails.

h5py is where the error shows up and I don't know why. Whenever any code tries to import h5py, or when I test the module load, I get:

(HDF5env) user@localhost:~$ python -m h5py
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/HDF5env/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/user/anaconda3/envs/HDF5env/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/user/anaconda3/envs/HDF5env/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/user/anaconda3/envs/HDF5env/lib/python3.6/site-packages/h5py/__init__.py", line 26, in <module>
    from . import _errors
ImportError: cannot import name '_errors'

I suspect that it might have something to do with the failing HDF5 tests? I have scoured the internet. setting the HDF5_DIR var in the pip command is the only proposed solution I have found that works for others with this error, but does not work for me.

Thanks in advance.