Intel® oneAPI HPC Toolkit
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.

Questions about Intel MPI 2021

RuiXi
Beginner
611 Views

Hello,

 

Here is my environment information

Intel one API 2021.2

CentOS 7.6

MLNX_OFED_LINUX-5.3-1.0.0.1

UCX 1.10

I have two HCAs per node, but only one Active.

 

When I run a test of 18 nodes, I get results in 0.9 seconds.

[root@n00001 mpi-test]# time mpirun -np 18 -ppn 1 -f hostfile2  hostname
n07061
n07068
n07069
n07063
n07064
n07074
n07073
n07072
n07062
n07067
n07066
n07070
n07065
n07071
n07059
n07060
n07057
n07058

real 0m0.939s
user 0m0.841s
sys 0m0.871s

 

When I run tests with more than 18 nodes, I need 41 seconds to get results.

[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2  hostname
n07066
n07070
n07073
n07067
n07075
n07068
n07065
n07069
n07074
n07071
n07072
n07076
n07059
n07057
n07058
n07060
n07064
n07063
n07061
n07062

real 0m41.026s
user 0m0.790s
sys 0m0.875s

 

 

Do you have any suggestions on this question? Thank you

0 Kudos
7 Replies
ShivaniK_Intel
Moderator
568 Views

Hi,


Thanks for reaching out to us.


Could you please let us know about your cluster topology, whether all nodes are connected to the same switch or different?


Thanks & Regards

Shivani 


RuiXi
Beginner
564 Views

Hi Shivani,

 

Thank you for your reply.

 

My network is a fat tree non-blocking network, and these nodes are connected to the same switch.

 

BTW.

I tested Intel MPI 2018 and Openmpi did not have this problem.

 

Intel MPI 2018:

[root@n00001 mpi-test]# mpirun -V
Intel(R) MPI Library for Linux* OS, Version 2018 Update 1 Build 20171011 (id: 17941)
Copyright (C) 2003-2017, Intel Corporation. All rights reserved.
[root@n00001 mpi-test]#
[root@n00001 mpi-test]#
[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2 hostname
n07058
n07057
n07065
n07059
n07063
n07060
n07061
n07064
n07069
n07062
n07067
n07068
n07066
n07075
n07073
n07072
n07076
n07071
n07070
n07074

real 0m0.496s
user 0m1.093s
sys 0m0.242s

 

OpenMPI:

[root@n00001 mpi-test]# time /usr/mpi/gcc/openmpi-4.1.0rc5/bin/mpirun --allow-run-as-root -np 20 --hostfile hostfile2 -N 1 hostname
n07064
n07063
n07074
n07067
n07072
n07071
n07057
n07075
n07059
n07061
n07068
n07076
n07069
n07065
n07070
n07062
n07073
n07060
n07066
n07058

real 0m0.938s
user 0m0.306s
sys 0m0.094s

 

 

 

Thanks & Regards

Rui

 

ShivaniK_Intel
Moderator
537 Views

Hi,


Thanks for providing the details. We are working on it and will get back to you soon.


Thanks & Regards

Shivani


James_T_Intel
Moderator
517 Views

Please repeat the 18 and 20 node runs with I_MPI_HYDRA_DEBUG=1 and provide the results.


RuiXi
Beginner
504 Views

Hi James,

 

Results for 18 nodes

[root@n00001 mpi-test]# time mpirun -np 18 -ppn 1 -f hostfile2 -genv I_MPI_HYDRA_DEBUG=1 hostname
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07057 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07059 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 2 --node-id 2 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07061 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 4 --node-id 4 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07062 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 5 --node-id 5 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07063 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 6 --node-id 6 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07064 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 7 --node-id 7 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07065 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 8 --node-id 8 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07066 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 9 --node-id 9 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07067 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 10 --node-id 10 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07068 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 11 --node-id 11 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07069 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 12 --node-id 12 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07070 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 13 --node-id 13 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07071 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 14 --node-id 14 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07072 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 15 --node-id 15 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07073 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 16 --node-id 16 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07074 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 43802 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 17 --node-id 17 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:0@n07057] Launch arguments: /bin/ssh -q -x n07058 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07057 --upstream-port 41499 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 1 --node-id 1 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:2@n07059] Launch arguments: /bin/ssh -q -x n07060 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07059 --upstream-port 36338 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 3 --node-id 3 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07072
n07064
n07066
n07069
n07061
n07065
n07071
n07070
n07068
n07063
n07073
n07067
n07074
n07062
n07060
n07057
n07059
n07058

real 0m0.944s
user 0m0.816s
sys 0m0.888s

 

 

Results for 20 nodes

[root@n00001 mpi-test]# time mpirun -np 20 -ppn 1 -f hostfile2 -genv I_MPI_HYDRA_DEBUG=1 hostname
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07057 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07059 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 2 --node-id 2 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07061 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 4 --node-id 4 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07063 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 6 --node-id 6 --subtree-size 2 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07065 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 8 --node-id 8 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07066 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 9 --node-id 9 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07067 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 10 --node-id 10 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07068 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 11 --node-id 11 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07069 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 12 --node-id 12 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07070 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 13 --node-id 13 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07071 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 14 --node-id 14 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07072 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 15 --node-id 15 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07073 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 16 --node-id 16 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07074 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 17 --node-id 17 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07075 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 18 --node-id 18 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@n00001] Launch arguments: /bin/ssh -q -x n07076 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n00001 --upstream-port 45857 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 19 --node-id 19 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:0@n07057] Launch arguments: /bin/ssh -q -x n07058 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07057 --upstream-port 35577 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 1 --node-id 1 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:2@n07059] Launch arguments: /bin/ssh -q -x n07060 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07059 --upstream-port 38702 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 3 --node-id 3 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07070
n07069
n07073
n07071
n07072
n07076
n07067
n07068
n07075
n07066
n07065
n07074
n07059
n07060
n07057
n07058
[bstrap:0:6@n07063] Launch arguments: /bin/ssh -q -x n07064 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07063 --upstream-port 39938 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 7 --node-id 7 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[bstrap:0:4@n07061] Launch arguments: /bin/ssh -q -x n07062 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_bstrap_proxy --upstream-host n07061 --upstream-port 38796 --pgid 0 --launcher ssh --launcher-number 0 --base-path /es01/software/oneapi/mpi/2021.2.0//bin/ --tree-width 16 --tree-level 2 --time-left -1 --launch-type 2 --debug --proxy-id 5 --node-id 5 --subtree-size 1 /es01/software/oneapi/mpi/2021.2.0//bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
n07063
n07061
n07064
n07062

real 0m40.957s
user 0m0.379s
sys 0m0.357s

 

 

 

 

Thanks & Regards

Rui

James_T_Intel
Moderator
496 Views

Please try I_MPI_HYDRA_BRANCH_COUNT=0.


James_T_Intel
Moderator
414 Views

Due to lack of reply, I am closing Intel support on this thread. Any further posts on this thread will be community only. If you need further Intel support assistance, please create a new thread.


Reply