how to run the hybrid MPI/OpenMP

[dingjun_chencmgl_ca]

Hi, Dear Sir or Madam,

I am using the Intel MPI and OpenMP and Intel Composer XE 2015 to build a hybrid MPI/OpenMP application. For example, if I want to run the executable file of my application on 3 SMP computers with 3 MPI processes and each MPI process consists of 16 OpenMP threads.  Our PC cluster has 3 SMP nodes connected by the Infiniband and each node has 16 cores.  

Could someone tell me how to run with the Intel MPI command: mpiexec.exe in order to implement above running way? where can I find such a study guide for running with Intel MPI?  I look forward to hearing from you and your reply is highly appreciated.

Thanks in advance.

[Gergana_S_Intel]

Hi,

Thanks for getting in touch.

If you're planning on running a hybrid MPI/OpenMP code, make sure you're first linking with the thread-safe Intel MPI Library.  That can be accomplished by using the latest Intel MPI 5.0.2 where thread safety is default.  If you're using anything older, make sure you're linking with the -mt_mpi option.

If you already have an executable that fulfills those requirements, run it as you would any other MPI application and set additional OpenMP settings on the command line.  For example, in the case you propose, you want to run 3 MPI ranks over 3 SMP machines (1 MPI rank per machine) + 16 OMP threads per rank.  Here's the command line for that:

$ mpirun -f mpi-hosts-list -perhost 1 -genv OMP_NUM_THREADS 16 -n 3 ./hybrid.exe

The -perhost option will tell Intel MPI how many ranks you want per SMP machine.  The OMP_NUM_THREADS env variable is applicable to the OMP runtimes and you can easily set it here via the -genv option.  Finally, you're running 3 MPI ranks which is set by the -n option.

Please note you need to either provide a hosts file here (mpi-hosts-list) that lists your 3 SMP hosts, one hostname per line.  If you're running under a job scheduler, you can omit that option.

I encourage you to take a look at the OpenMP interoperability section in our Intel MPI Library Reference Manual.  That has more details on pinning the processes.  This article on Beginning Hybrid MPI/OpenMP Development is also very helpful.

Hope this will do the trick.  Let us know if you have further questions.

Regards,
~Gergana