Re:Questions about Intel MPI 2021

RuiXi · ‎05-17-2021

Hello,

Here is my environment information

Intel one API 2021.2

CentOS 7.6

MLNX_OFED_LINUX-5.3-1.0.0.1

UCX 1.10

I have two HCAs per node, but only one Active.

When I run a test of 18 nodes, I get results in 0.9 seconds.

[root@n00001 mpi-test]# time mpirun -np 18 -ppn 1 -f hostfile2 hostname
n07061
n07068
n07069
n07063
n07064
n07074
n07073
n07072
n07062
n07067
n07066
n07070
n07065
n07071
n07059
n07060
n07057
n07058

real 0m0.939s
user 0m0.841s
sys 0m0.871s

When I run tests with more than 18 nodes, I need 41 seconds to get results.

[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2 hostname
n07066
n07070
n07073
n07067
n07075
n07068
n07065
n07069
n07074
n07071
n07072
n07076
n07059
n07057
n07058
n07060
n07064
n07063
n07061
n07062

real 0m41.026s
user 0m0.790s
sys 0m0.875s

Do you have any suggestions on this question? Thank you

ShivaniK_Intel · ‎05-19-2021

Hi,

Thanks for reaching out to us.

Could you please let us know about your cluster topology, whether all nodes are connected to the same switch or different?

Thanks & Regards

Shivani

RuiXi · ‎05-19-2021

Hi Shivani,

Thank you for your reply.

My network is a fat tree non-blocking network, and these nodes are connected to the same switch.

BTW.

I tested Intel MPI 2018 and Openmpi did not have this problem.

Intel MPI 2018:

[root@n00001 mpi-test]# mpirun -V
Intel(R) MPI Library for Linux* OS, Version 2018 Update 1 Build 20171011 (id: 17941)
Copyright (C) 2003-2017, Intel Corporation. All rights reserved.
[root@n00001 mpi-test]#
[root@n00001 mpi-test]#
[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2 hostname
n07058
n07057
n07065
n07059
n07063
n07060
n07061
n07064
n07069
n07062
n07067
n07068
n07066
n07075
n07073
n07072
n07076
n07071
n07070
n07074

real 0m0.496s
user 0m1.093s
sys 0m0.242s

OpenMPI:

[root@n00001 mpi-test]# time /usr/mpi/gcc/openmpi-4.1.0rc5/bin/mpirun --allow-run-as-root -np 20 --hostfile hostfile2 -N 1 hostname
n07064
n07063
n07074
n07067
n07072
n07071
n07057
n07075
n07059
n07061
n07068
n07076
n07069
n07065
n07070
n07062
n07073
n07060
n07066
n07058

real 0m0.938s
user 0m0.306s
sys 0m0.094s

Thanks & Regards

Rui

ShivaniK_Intel · ‎05-21-2021

Hi,

Thanks for providing the details. We are working on it and will get back to you soon.

Thanks & Regards

Shivani

James_T_Intel · ‎05-24-2021

Please repeat the 18 and 20 node runs with I_MPI_HYDRA_DEBUG=1 and provide the results.

RuiXi · ‎05-24-2021

Hi James,

Results for 18 nodes

[root@n00001 mpi-test]# time mpirun -np 18 -ppn 1 -f hostfile2 -genv I_MPI_HYDRA_DEBUG=1 hostname
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07057 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07059 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 2 --node-id 2 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07061 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 4 --node-id 4 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07062 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 5 --node-id 5 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07063 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 6 --node-id 6 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07064 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 7 --node-id 7 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07065 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 8 --node-id 8 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07066 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 9 --node-id 9 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07067 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 10 --node-id 10 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07068 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 11 --node-id 11 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07069 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 12 --node-id 12 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07070 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 13 --node-id 13 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07071 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 14 --node-id 14 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07072 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 15 --node-id 15 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07073 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 16 --node-id 16 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07074 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 17 --node-id 17 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:0@n07057] Launch arguments: /bin/ssh -q -x n07058 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07057 --upstream-port 41499 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 1 --node-id 1 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:2@n07059] Launch arguments: /bin/ssh -q -x n07060 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07059 --upstream-port 36338 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 3 --node-id 3 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07072
n07064
n07066
n07069
n07061
n07065
n07071
n07070
n07068
n07063
n07073
n07067
n07074
n07062
n07060
n07057
n07059
n07058

real 0m0.944s
user 0m0.816s
sys 0m0.888s

Results for 20 nodes

[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2 -genv I_MPI_HYDRA_DEBUG=1 hostname
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07057 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07059 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 2 --node-id 2 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07061 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 4 --node-id 4 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07063 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 6 --node-id 6 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07065 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 8 --node-id 8 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07066 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 9 --node-id 9 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07067 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 10 --node-id 10 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07068 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 11 --node-id 11 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07069 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 12 --node-id 12 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07070 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 13 --node-id 13 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07071 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 14 --node-id 14 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07072 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 15 --node-id 15 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07073 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 16 --node-id 16 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07074 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 17 --node-id 17 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07075 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 18 --node-id 18 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07076 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 19 --node-id 19 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:0@n07057] Launch arguments: /bin/ssh -q -x n07058 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07057 --upstream-port 35577 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 1 --node-id 1 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:2@n07059] Launch arguments: /bin/ssh -q -x n07060 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07059 --upstream-port 38702 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 3 --node-id 3 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07070
n07069
n07073
n07071
n07072
n07076
n07067
n07068
n07075
n07066
n07065
n07074
n07059
n07060
n07057
n07058
[bstrap:0:6@n07063] Launch arguments: /bin/ssh -q -x n07064 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07063 --upstream-port 39938 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 7 --node-id 7 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:4@n07061] Launch arguments: /bin/ssh -q -x n07062 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07061 --upstream-port 38796 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 5 --node-id 5 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07063
n07061
n07064
n07062

real 0m40.957s
user 0m0.379s
sys 0m0.357s

Thanks & Regards

Rui

James_T_Intel · ‎05-25-2021

Please try I_MPI_HYDRA_BRANCH_COUNT=0.

James_T_Intel · ‎06-07-2021

Due to lack of reply, I am closing Intel support on this thread. Any further posts on this thread will be community only. If you need further Intel support assistance, please create a new thread.