Intel MPI fatal error

[atrash]

Hi,

We compiled a code which is able to performe atomistic simulations.
The code fail with the folowing error.
I'll be thankful ifyou can help me in fixing this prolem.

Thank you in advance,
Fouad

[0:node44][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [2:node58] completed with error. status=0x1. cookie=0x40002

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 0

[2:node58][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [0:node44] completed with error. status=0x6. cookie=0x40000

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 2

[1:node44] unexpected disconnect completion event from [2:node58]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 1

[3:node58] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 3

[6:node28] unexpected disconnect completion event from [0:node44]

[7:node28] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 7

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 6

[5:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

[4:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 4

internal ABORT - process 5

rank 0 in job 1 node44_33470 caused collective abort of all ranks

exit status of rank 0: return code 1

[Ending Job 36715]

[Dmitry_K_Intel2]

Hi Fouad,

It can be useful to know Intel MPI version, DAPL version, Fabric (or Device) used in that run, the whole command line. What fast fabric do use (Infiniband)?

Regards!
Dmitry