topic Azure multi-agent failure on MPI_Init in Intel® MPI Library https://community.intel.com/t5/Intel-MPI-Library/Azure-multi-agent-failure-on-MPI-Init/m-p/1016245#M4012 <P>Hey Forum,<BR /> <BR /> I'm trying to run on the Azure cloud using Intel's MPI implementation, but there is&nbsp;a problem. Everything works as expected when run on one Agent (8 processors), however anything with 2 or more Agents fails on MPI_Init() roughly 25% of the time. The failure is instantaneous (see output&nbsp;below). I was also able to reproduce this crash with a simple point to point send between all processors.&nbsp;I'm unable to reproduce the issue on my local system.<BR /> <BR /> Are there any known issues with Intel MPI on Azure's virtual machines? Any idea on why it may crash on initialization only&nbsp;some of the time?<BR /> <BR /> The current solution has been simply to use microsoft's MPI library, but I would really like to figure out what the source of the problem is.<BR /> <BR /> Thank you kindly.<BR /> <BR /> <BR /> Error output:</P> <TABLE border="0" cellpadding="0"> <TBODY> <TR> <TD> <P>Master Agent: 10</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 2</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 16</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 12</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>job aborted:</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>rank: node: exit code[: error message]</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)......................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).............................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>0: Agent10: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)........................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>1: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>2: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102)......:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>3: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>recv_id_or_tmpvc_info_success_handler(1330): read from socket failed - No error</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>4: Agent10: 1: process 4 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>5: Agent10: 1: process 5 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>6: Agent10: 1: process 6 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>7: Agent10: 1: process 7 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>8: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>9: Agent2: 1: process 9 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>10: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>11: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>12: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>13: Agent2: 1: process 13 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>14: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>15: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>16: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>17: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>18: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>19: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>20: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>21: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>22: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>23: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>24: Agent16: 1: process 24 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>25: Agent16: 1: process 25 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>26: Agent16: 1: process 26 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>27: Agent16: 1: process 27 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>28: Agent16: 1: process 28 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>29: Agent16: 1: process 29 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>30: Agent16: 1: process 30 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>31: Agent16: 1: process 31 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in PMPI_Isend: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>PMPI_Isend(161).................: MPI_Isend(buf=000000D8B7D7F89C, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD, request=000000D8B7D7F8A0) failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_EagerContigIsend(554).: failure occurred while attempting to send an eager message</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_iSendContig(440):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIU_SOCKW_Writev(454)..........: &nbsp;Unable to write to a socket, An existing connection was forcibly closed by the remote host.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>(errno 10054)</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in PMPI_Isend: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>PMPI_Isend(161).................: MPI_Isend(buf=0000001C452DFA5C, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD, request=0000001C452DFA60) failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_EagerContigIsend(554).: failure occurred while attempting to send an eager message</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_iSendContig(440):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIU_SOCKW_Writev(454)..........: &nbsp;Unable to write to a socket, An existing connection was forcibly closed by the remote host.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>(errno 10054)</P> </TD> <TD> <P>Error</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P><BR /> <BR /> &nbsp;</P> Thu, 15 May 2014 20:23:25 GMT Ben_E_ 2014-05-15T20:23:25Z Azure multi-agent failure on MPI_Init https://community.intel.com/t5/Intel-MPI-Library/Azure-multi-agent-failure-on-MPI-Init/m-p/1016245#M4012 <P>Hey Forum,<BR /> <BR /> I'm trying to run on the Azure cloud using Intel's MPI implementation, but there is&nbsp;a problem. Everything works as expected when run on one Agent (8 processors), however anything with 2 or more Agents fails on MPI_Init() roughly 25% of the time. The failure is instantaneous (see output&nbsp;below). I was also able to reproduce this crash with a simple point to point send between all processors.&nbsp;I'm unable to reproduce the issue on my local system.<BR /> <BR /> Are there any known issues with Intel MPI on Azure's virtual machines? Any idea on why it may crash on initialization only&nbsp;some of the time?<BR /> <BR /> The current solution has been simply to use microsoft's MPI library, but I would really like to figure out what the source of the problem is.<BR /> <BR /> Thank you kindly.<BR /> <BR /> <BR /> Error output:</P> <TABLE border="0" cellpadding="0"> <TBODY> <TR> <TD> <P>Master Agent: 10</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 2</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 16</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Secondary Agent: 12</P> </TD> <TD> <P>Information</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>job aborted:</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>rank: node: exit code[: error message]</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)......................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).............................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>0: Agent10: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)........................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>1: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>2: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102)......:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>3: Agent10: 1</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>recv_id_or_tmpvc_info_success_handler(1330): read from socket failed - No error</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>4: Agent10: 1: process 4 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>5: Agent10: 1: process 5 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>6: Agent10: 1: process 6 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>7: Agent10: 1: process 7 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>8: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>9: Agent2: 1: process 9 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>10: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>11: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>12: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>13: Agent2: 1: process 13 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>14: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>15: Agent2: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>16: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>17: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>18: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>19: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>20: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>21: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>22: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>23: Agent12: 123</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>24: Agent16: 1: process 24 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>25: Agent16: 1: process 25 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>26: Agent16: 1: process 26 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>27: Agent16: 1: process 27 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>28: Agent16: 1: process 28 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>29: Agent16: 1: process 29 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>30: Agent16: 1: process 30 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>31: Agent16: 1: process 31 exited without calling finalize</P> </TD> <TD> <P>Debug</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in MPI_Init: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIR_Init_thread(658)................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_Init(195).......................: channel initialization failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_Init(104)..................:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_tcp_post_init(345)..........:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_module_connpoll(3102):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>gen_read_fail_handler(1196)..........: read from socket failed - The specified network name is no longer available.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in PMPI_Isend: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>PMPI_Isend(161).................: MPI_Isend(buf=000000D8B7D7F89C, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD, request=000000D8B7D7F8A0) failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_EagerContigIsend(554).: failure occurred while attempting to send an eager message</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_iSendContig(440):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIU_SOCKW_Writev(454)..........: &nbsp;Unable to write to a socket, An existing connection was forcibly closed by the remote host.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>(errno 10054)</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>Fatal error in PMPI_Isend: Other MPI error, error stack:</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>PMPI_Isend(161).................: MPI_Isend(buf=0000001C452DFA5C, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD, request=0000001C452DFA60) failed</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIDI_CH3_EagerContigIsend(554).: failure occurred while attempting to send an eager message</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPID_nem_newtcp_iSendContig(440):</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>MPIU_SOCKW_Writev(454)..........: &nbsp;Unable to write to a socket, An existing connection was forcibly closed by the remote host.</P> </TD> <TD> <P>Error</P> </TD> </TR> <TR> <TD> <P>(errno 10054)</P> </TD> <TD> <P>Error</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P><BR /> <BR /> &nbsp;</P> Thu, 15 May 2014 20:23:25 GMT https://community.intel.com/t5/Intel-MPI-Library/Azure-multi-agent-failure-on-MPI-Init/m-p/1016245#M4012 Ben_E_ 2014-05-15T20:23:25Z