Asynchronous progress slows down my program

[manasi-t24]

Hi,

I am running PETSc code on our institute's supercomputer Pravega. I have compiled my PETSc code after loading the following modules and sourcing mpivars.sh:

1. module load spack
2. . /home-ext/apps/spack/share/spack/setup-env.sh
3. spack load gcc
4. module load compiler/intel/2020.4.304
5. source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

After compilation, I am running the code with the following script where export I_MPI_ASYNC_PROGRESS=0:

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=48
#SBATCH --exclusive
#SBATCH --time=00:03:00
#SBATCH --job-name=ex2
#SBATCH --error=ex2.e%J
#SBATCH --output=ex2.o%J
##SBATCH --partition=standard

module load spack
. /home-ext/apps/spack/share/spack/setup-env.sh
spack load gcc

module load compiler/intel/2020.4.304
source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

# // Below are Intel MPI specific settings //
#export I_MPI_FABRICS=ofi:ofi
#export I_MPI_DEBUG=10
export I_MPI_ASYNC_PROGRESS=0
#export I_MPI_ASYNC_PROGRESS_THREADS=1

cd $SLURM_SUBMIT_DIR
nprocs=96

ulimit -aH
ulimit -c unlimited
ulimit -s unlimited

mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi

*************end of script**************

Time taken by the above job to run is 0.280063 sec.

I am also running the code with the following script where export I_MPI_ASYNC_PROGRESS=1

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=48
#SBATCH --exclusive
#SBATCH --time=00:03:00
#SBATCH --job-name=ex2
#SBATCH --error=ex2.e%J
#SBATCH --output=ex2.o%J
##SBATCH --partition=standard

module load spack
. /home-ext/apps/spack/share/spack/setup-env.sh
spack load gcc

module load compiler/intel/2020.4.304
source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

# // Below are Intel MPI specific settings //
#export I_MPI_FABRICS=ofi:ofi
#export I_MPI_DEBUG=10
export I_MPI_ASYNC_PROGRESS=1
#export I_MPI_ASYNC_PROGRESS_THREADS=1

cd $SLURM_SUBMIT_DIR
nprocs=96

ulimit -aH
ulimit -c unlimited
ulimit -s unlimited

mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi

*************end of script**************

Time taken by the above job to run is 22.659463 sec. 

I observe that my code takes much more time to run when I use I_MPI_ASYNC_PROGRESS=1 than when I use I_MPI_ASYNC_PROGRESS=0. Why is this happening? I read this blog (https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/?elq_cid=9262301_ts1646820353975&erpm_id=11789269_ts1646820353975#gs.seb071) but could not quite understand what needs to be done in order to bring performance improvement. Please help.

Regards,

Manasi

[HemanthCH_Intel]

Hi,

Thanks for reaching out to us.

>>" I have compiled my PETSc code after loading the following modules and sourcing mpivars.sh:"
Could you please provide the steps that you have followed to compile the PETsc code?

>>"mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi"

Could you please provide us the path to the "ex2" binary from the PETsc directory?

Could you please provide the OS details?

Thanks & Regards,

Hemanth.

[manasi-t24]

Hi,

I downloaded the PETSc repository from gitlab using:

git clone -b release https://gitlab.com/petsc/petsc.git petsc-cpu-intel

After loading the following modules:

1. module load spack
2. . /home-ext/apps/spack/share/spack/setup-env.sh
3. spack load gcc
4. module load compiler/intel/2020.4.304
5. source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

I configured the PETSc code with:

./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-make --download-fblaslapack=1

And then I compiled it with:

make PETSC_DIR=/scratch/cdsmanas/petsc-cpu-intel PETSC_ARCH=arch-linux-c-debug all

The directory for ex2 is:

<petsc directory>/src/ksp/ksp/tutorials/ex2

OS and its version:

[cdsmanas@login10 tutorials]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Regards,

Manasi

[HemanthCH_Intel]

Hi,

As we have worked with you internally and your issue has been resolved, we are closing this thread. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks & Regards,

Hemanth.