Intel MPI fatal error - Intel Community

Intel® MPI Library

Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.

Intel MPI fatal error

579 Views

Hi,

We compiled a code which is able to performe atomistic simulations.
The code fail with the folowing error.
I'll be thankful ifyou can help me in fixing this prolem.

Thank you in advance,
Fouad

[0:node44][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [2:node58] completed with error. status=0x1. cookie=0x40002

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 0

[2:node58][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [0:node44] completed with error. status=0x6. cookie=0x40000

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 2

[1:node44] unexpected disconnect completion event from [2:node58]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 1

[3:node58] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 3

[6:node28] unexpected disconnect completion event from [0:node44]

[7:node28] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 7

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 6

[5:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

[4:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 4

internal ABORT - process 5

rank 0 in job 1 node44_33470 caused collective abort of all ranks

exit status of rank 0: return code 1

[Ending Job 36715]

Link Copied

1 Reply

579 Views

Hi Fouad,

It can be useful to know Intel MPI version, DAPL version, Fabric (or Device) used in that run, the whole command line. What fast fabric do use (Infiniband)?

Regards!
Dmitry

Copy link

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.