Error in I_MPI_PIN_DOMAIN when combining Intel MPI and OpenMP

[as14]

Hi,

I am running a Coarray Fortran code with Intel-MPI, and additionally am trying to combine this with OpenMP. Intel-MPI should assign a rank to each node I am using, and within that node, all cores should be available to OpenMP as threads.

In my case, each node consists of two sockets with 64 threads each. I only run with 10 for testing, as you will see below.

The MPI functionality and communication between nodes works perfectly, however only one OpenMP thread is visible when running omp_get_num_procs() .

I am using SLURM, and in my job script, I set:

export I_MPI_OFI_PROVIDER=Verbs

export I_MPI_PIN_DOMAIN=node

export OMP_NUM_THREADS=10

Setting the OFI provider to Verbs specifies the communications provider between notes, and the I_MPI_PIN_DOMAIN sets how threads should be pinned within an MPI domain (see here: https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/process-pinning/interoperability-with-openmp-api.html).

I submit the job with:

#SBATCH --nodes=2

#SBATCH --ntasks-per-node=1

#SBATCH --cpus-per-task=10

I have tested with:

1. intel/19.1.3.304 (2020.4.304) with intel/mpi/2021.5.0

2. intel-oneapi-compilers/2022.0.2-gcc-11.2.0-yzi4tsu with intel-oneapi-mpi/2021.4.0-gcc-11.2.0-2e7zm7z

Any help would be greatly appreciated! Thanks  

[AishwaryaCV_Intel]

Hi,

Thank you for posting in Intel Communities.

Could you please try using 'omp' instead 'node' in I_MPI_PIN_DOMAIN 

Could you please provide us with a complete sample reproducer code and steps to reproduce the issue so that we can try it from our end?

Thanks and Regards,

Aishwarya

[as14]

Thanks for getting back to me. I had already tried I_MPI_PIN_DOMAIN as omp, and I have tried again now, but still not working. Please find attached a zip file with the code. You simply need to execute "./run.sh" to create the executable and submit a job to the SLURM job management system.

Let me know what you figure out! Thanks!

[AishwaryaCV_Intel]

Hi,

We have run your code with Intel processor: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz

We were successfully able to run the code and was able to get 10 threads by assigning omp to I_MPI_PIN_DOMAIN

Can find the following attached screenshots for using node and omp as node.png and omp.png respectively.

NOTE: Could you please check I_MPI_PIN_DOMAIN=omp with Intel Processors

Can find the supported Intel Processors here https://www.intel.in/content/www/in/en/support/products/873/processors.html#122139

Thanks And Regards,

Aishwarya

[as14]

Hi, 

Thanks for testing - I have been attempting to run this on AMD processors, which therefore must be the cause of the problem. Thanks anyway!

[AishwaryaCV_Intel]

Hi,

Thank you for your inquiry. We can only offer direct support for Intel hardware platforms that the Intel® oneAPI product supports. Intel provides instructions on how to compile oneAPI code for both CPU and a wide range of GPU accelerators. https://intel.github.io/llvm-docs/GetStartedGuide.html

This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.

Thanks & Regards,

Aishwarya