Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2179 Discussions

Asynchronous progress slows down my program

manasi-t24
Beginner
1,201 Views

Hi,

I am running PETSc code on our institute's supercomputer Pravega. I have compiled my PETSc code after loading the following modules and sourcing mpivars.sh:

  1. module load spack
  2. . /home-ext/apps/spack/share/spack/setup-env.sh
  3. spack load gcc
  4. module load compiler/intel/2020.4.304
  5. source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

After compilation, I am running the code with the following script where export I_MPI_ASYNC_PROGRESS=0:

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=48
#SBATCH --exclusive
#SBATCH --time=00:03:00
#SBATCH --job-name=ex2
#SBATCH --error=ex2.e%J
#SBATCH --output=ex2.o%J
##SBATCH --partition=standard

module load spack
. /home-ext/apps/spack/share/spack/setup-env.sh
spack load gcc

module load compiler/intel/2020.4.304
source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

# // Below are Intel MPI specific settings //
#export I_MPI_FABRICS=ofi:ofi
#export I_MPI_DEBUG=10
export I_MPI_ASYNC_PROGRESS=0
#export I_MPI_ASYNC_PROGRESS_THREADS=1


cd $SLURM_SUBMIT_DIR
nprocs=96

ulimit -aH
ulimit -c unlimited
ulimit -s unlimited

mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi

*************end of script**************

Time taken by the above job to run is 0.280063 sec.

I am also running the code with the following script where export I_MPI_ASYNC_PROGRESS=1

#!/bin/bash
#SBATCH -N 2
#SBATCH --ntasks-per-node=48
#SBATCH --exclusive
#SBATCH --time=00:03:00
#SBATCH --job-name=ex2
#SBATCH --error=ex2.e%J
#SBATCH --output=ex2.o%J
##SBATCH --partition=standard

module load spack
. /home-ext/apps/spack/share/spack/setup-env.sh
spack load gcc

module load compiler/intel/2020.4.304
source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

# // Below are Intel MPI specific settings //
#export I_MPI_FABRICS=ofi:ofi
#export I_MPI_DEBUG=10
export I_MPI_ASYNC_PROGRESS=1
#export I_MPI_ASYNC_PROGRESS_THREADS=1


cd $SLURM_SUBMIT_DIR
nprocs=96

ulimit -aH
ulimit -c unlimited
ulimit -s unlimited

mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi

*************end of script**************

 

Time taken by the above job to run is 22.659463 sec. 

I observe that my code takes much more time to run when I use I_MPI_ASYNC_PROGRESS=1 than when I use I_MPI_ASYNC_PROGRESS=0. Why is this happening? I read this blog (https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/?elq_cid=9262301_ts1646820353975&erpm_id=11789269_ts1646820353975#gs.seb071) but could not quite understand what needs to be done in order to bring performance improvement. Please help.

 

Regards,

Manasi

 

 

 

 

 

 

0 Kudos
3 Replies
HemanthCH_Intel
Moderator
1,127 Views

Hi,

 

Thanks for reaching out to us.

 

>>" I have compiled my PETSc code after loading the following modules and sourcing mpivars.sh:"
Could you please provide the steps that you have followed to compile the PETsc code?

 

>>"mpiexec.hydra -n $nprocs ./ex2 -m 80 -n 80 -ksp_monitor_short -ksp_type pipecg2 -pc_type jacobi"

Could you please provide us the path to the "ex2" binary from the PETsc directory?

 

Could you please provide the OS details?

 

Thanks & Regards,

Hemanth.

 

0 Kudos
manasi-t24
Beginner
1,104 Views

Hi,

I downloaded the PETSc repository from gitlab using:

git clone -b release https://gitlab.com/petsc/petsc.git petsc-cpu-intel

 

After loading the following modules:

  1. module load spack
  2. . /home-ext/apps/spack/share/spack/setup-env.sh
  3. spack load gcc
  4. module load compiler/intel/2020.4.304
  5. source /opt/ohpc/pub/compiler/intel/2020_4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpivars.sh release_mt

 

I configured the PETSc code with:

./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-make --download-fblaslapack=1

 

And then I compiled it with:

make PETSC_DIR=/scratch/cdsmanas/petsc-cpu-intel PETSC_ARCH=arch-linux-c-debug all

 

The directory for ex2 is:

<petsc directory>/src/ksp/ksp/tutorials/ex2

 

OS and its version:

[cdsmanas@login10 tutorials]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

 

Regards,

Manasi

 

 

 

0 Kudos
HemanthCH_Intel
Moderator
1,074 Views

Hi,


As we have worked with you internally and your issue has been resolved, we are closing this thread. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.


Thanks & Regards,

Hemanth.


0 Kudos
Reply