- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am running a heterogeneous cluster, half the nodes Gbit ethernet and the other half Infiniband. For a year or so everything went well, but recently the Gbit nodes complain about the lack of Infiniband (see below). This phaenomenon is limited to impi code, GNU mpi still runs fine.
The problem appears unrelated to the queuing system, a direct launch fails in the same way as a SGE submitted one.
Any help would be greatly appreciated.
...
compute-0-15.local:19848: open_hca: rdma_bind ERR No such device. Is eth0 configured?
compute-0-15.local:19847: open_hca: rdma_bind ERR No such device. Is eth0 configured?
compute-0-15.local:19845: open_hca: getaddr_netdev ERROR: No such device. Is ib1 configured?
compute-0-15.local:19845: open_hca: device mthca0 not found
compute-0-15.local:19845: open_hca: device mthca0 not found
compute-0-15.local:19845: open_hca: device mlx4_0 not found
compute-0-15.local:19845: open_hca: device mlx4_0 not found
compute-0-15.local:19845: open_hca: device ipath0 not found
compute-0-15.local:19845: open_hca: device ipath0 not found
compute-0-15.local:19845: open_hca: device ehca0 not found
compute-0-15.local:19845: open_hca: rdma_bind ERR No such device. Is eth0 configured?
[cli_0]: got unexpected response to put :cmd=unparseable_msg rc=-1
:
[cli_0]: aborting job:
Fatal error in MPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(283): Initialization failed
MPIDD_Init(98).......: channel initialization failed
MPIDI_CH3_Init(163)..: generic failure with errno = 336068751
(unknown)(): Other MPI error
rank 0 in job 1 xxxxl_51508 caused collective abort of all ranks
exit status of rank 0: return code 13
I am running a heterogeneous cluster, half the nodes Gbit ethernet and the other half Infiniband. For a year or so everything went well, but recently the Gbit nodes complain about the lack of Infiniband (see below). This phaenomenon is limited to impi code, GNU mpi still runs fine.
The problem appears unrelated to the queuing system, a direct launch fails in the same way as a SGE submitted one.
Any help would be greatly appreciated.
...
compute-0-15.local:19848: open_hca: rdma_bind ERR No such device. Is eth0 configured?
compute-0-15.local:19847: open_hca: rdma_bind ERR No such device. Is eth0 configured?
compute-0-15.local:19845: open_hca: getaddr_netdev ERROR: No such device. Is ib1 configured?
compute-0-15.local:19845: open_hca: device mthca0 not found
compute-0-15.local:19845: open_hca: device mthca0 not found
compute-0-15.local:19845: open_hca: device mlx4_0 not found
compute-0-15.local:19845: open_hca: device mlx4_0 not found
compute-0-15.local:19845: open_hca: device ipath0 not found
compute-0-15.local:19845: open_hca: device ipath0 not found
compute-0-15.local:19845: open_hca: device ehca0 not found
compute-0-15.local:19845: open_hca: rdma_bind ERR No such device. Is eth0 configured?
[cli_0]: got unexpected response to put :cmd=unparseable_msg rc=-1
:
[cli_0]: aborting job:
Fatal error in MPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(283): Initialization failed
MPIDD_Init(98).......: channel initialization failed
MPIDI_CH3_Init(163)..: generic failure with errno = 336068751
(unknown)(): Other MPI error
rank 0 in job 1 xxxxl_51508 caused collective abort of all ranks
exit status of rank 0: return code 13
Link Copied
5 Replies
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Hennes,
Intel MPI Library does not support heterogeneous environment. You need to add I_MPI_FABRICS=shm:tcp to the list of enviroment variables if you are using 4.x
Regards!
Dmitry
Intel MPI Library does not support heterogeneous environment. You need to add I_MPI_FABRICS=shm:tcp to the list of enviroment variables if you are using 4.x
Regards!
Dmitry
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Dmitry,
Thanks for your reply. I should clarify that the different fabrics have their own queues. The error posted above shows up when a job is submitted to Gbit-only nodes. Strangely everything worked well for one year. A recent reboot of the head node broke it, but now it seems impossible to figure which update in particular is causing this. Is there a point in trying softiwarp on the gbit nodes?
-env I_MPI_FABRICS shm:tcp seems not to work with -r ssh. Is there a way to make it work?
Thanks,
Hennes
Thanks for your reply. I should clarify that the different fabrics have their own queues. The error posted above shows up when a job is submitted to Gbit-only nodes. Strangely everything worked well for one year. A recent reboot of the head node broke it, but now it seems impossible to figure which update in particular is causing this. Is there a point in trying softiwarp on the gbit nodes?
-env I_MPI_FABRICS shm:tcp seems not to work with -r ssh. Is there a way to make it work?
Thanks,
Hennes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hennes,
Have you changed anything on the cluster? Have you changed Intel MPI?
Intel MPI library should work with ssh, but you need to have passwordless connection. So, from node1 you need to be able to run 'mpiexec -n 1 -host node2 hostname'
Is it reproducable on other nodes?
I'm not sure that the issue is related to the library - it seems to me that you need to check eth0 settings.
Regards!
Dmitry
Have you changed anything on the cluster? Have you changed Intel MPI?
Intel MPI library should work with ssh, but you need to have passwordless connection. So, from node1 you need to be able to run 'mpiexec -n 1 -host node2 hostname'
Is it reproducable on other nodes?
I'm not sure that the issue is related to the library - it seems to me that you need to check eth0 settings.
Regards!
Dmitry
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dmitry,
The cluster remained unchanged (except RHEL5.5 updates on the head), Intel MPI is 4.0.0.028, installed about 12 months ago and left untouched since then. Passwordless connection works from and to all nodes. The eth0 settings appear valid to me and gnu MPI is running fine on all nodes. ssh as such works, just with "-env I_MPI_FABRICS shm:tcp" it breaks. I noticed this last already year when I did some unrelated tests.
Regards,
Hennes
The cluster remained unchanged (except RHEL5.5 updates on the head), Intel MPI is 4.0.0.028, installed about 12 months ago and left untouched since then. Passwordless connection works from and to all nodes. The eth0 settings appear valid to me and gnu MPI is running fine on all nodes. ssh as such works, just with "-env I_MPI_FABRICS shm:tcp" it breaks. I noticed this last already year when I did some unrelated tests.
Regards,
Hennes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Hennes,
Could you submit a ticket at premier.intel.com and please attach the output of a run with I_MPI_DEBUG=20
Regards!
Dmitry
Could you submit a ticket at premier.intel.com and please attach the output of a run with I_MPI_DEBUG=20
Regards!
Dmitry
Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page