Problem with MPIRUN run on multi-nodes Under LINUX

dingjun_chencmgl_ca — Thu, 28 Nov 2013 20:19:10 GMT

HI, Dear Sir or Madam,

I am testing the Intel MPI benchmar under RedHat Linux with 2 computers: node1-rh(8 cores) and node2-rh(8 cores). After I enter the following command on the local node1-rh:

bash-4.1$ mpirun -hosts node2-rh, node1-rh -np 16 IMB-MPI1
dingjun@node1-rh's password:

*********

IMB-MPI1 did not run. This indicates the IMB-MPI1 did not run on the remote computer node2-rh.

Could you experience such a problem? please tell my your ideas on how to run a executable file on multi-nodes under Linux with the Intel MPI library. Thanks in advance.

Have a good day and I look forward to hearing from you.

You should not be seeing a

James_T_Intel — Tue, 03 Dec 2013 21:35:13 GMT

You should not be seeing a password prompt. Please set up passwordless SSH for the nodes and try again.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Thanks and right now it is OK

dingjun_chencmgl_ca — Wed, 04 Dec 2013 21:40:07 GMT

Thanks and right now it is OK. However, a new problem occurred again. The details are described as follows:

bash-4.1$ ls
hostfile IMB-EXT IMB-IO IMB-MPI1 IMB-NBC IMB-RMA

bash-4.1$ cat hostfile
imex-snb2s-rh
imexsvr4-rh
imexsvr3-rh
(*Note: all above 3 computers are 2-socket 16-core SundyBridge DELL computers.)

The following 3 benchmarkings can be successful to run without any errors.

bash-4.1$ mpirun -genv I_MPI_FABRICS shm:tcp -n 32 -f hostfile IMB-MPI1

bash-4.1$ mpirun -genv I_MPI_FABRICS shm:tcp -n 32 -f hostfile IMB-NBC

bash-4.1$ mpirun -genv I_MPI_FABRICS shm:tcp -n 32 -f hostfile IMB-RMA

BUT, the following 2 benchmarking failed to run and errors occurred as follows:

bash-4.1$ mpirun -genv I_MPI_FABRICS shm:tcp -n 32 -f hostfile IMB-IO

[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-IO (No such file or directory)

bash-4.1$ mpirun -genv I_MPI_FABRICS shm:tcp -n 32 -f hostfile IMB-EXT

[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:0@imex-snb2s-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)
[proxy:0:1@imexsvr4-rh] HYDU_create_process (../../utils/launch/launch.c:572): execvp error on file IMB-EXT (No such file or directory)

Please tell me the reasons why such errors occurred for the benchmarking IMB-IO and IMB-EXT.

Thanks again and I look forward to hearing from you as soon as possible.

dingjun_chencmgl_ca — Wed, 11 Dec 2013 17:01:34 GMT

I look forward to hearing from you. Thanks anyway.

This looks to me like a

James_T_Intel — Tue, 31 Dec 2013 16:56:20 GMT

This looks to me like a problem with the network share on your system. Try logging into each directly and simply running IMB-IO and IMB-EXT. I've seen similar behavior if there are NFS problems, sometimes executables aren't visible on all systems, or sometimes different versions are on different nodes.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Hi,

Garg__Mukul — Mon, 12 Feb 2018 08:50:40 GMT

Hi,

I am facing issue while running below command

mpiexec -np 2 --hosts master slave stress --cpu 8 --timeout 20

getting below error

HYDU_create_process (./utils/launch/launch.c:69): execvp error on file slave (No such file or directory)
HYDU_create_process (./utils/launch/launch.c:69): execvp error on file slave (No such file or directory)

Could you please help me with this.....

Regards,

topic in Intel® MPI Library

Problem with MPIRUN run on multi-nodes Under LINUX

You should not be seeing a

Thanks and right now it is OK

This looks to me like a

Hi,