Intel® oneAPI HPC Toolkit
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.

Intel MPI fatal error

atrash
Beginner
158 Views
Hi,

We compiled a code which is able to performe atomistic simulations.
The code fail with the folowing error.
I'll be thankful ifyou can help me in fixing this prolem.

Thank you in advance,
Fouad


[0:node44][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [2:node58] completed with error. status=0x1. cookie=0x40002

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 0

[2:node58][../../dapl_module_poll.c:3972] Intel MPI fatal error: OpenIB-cma DTO operation posted for [0:node44] completed with error. status=0x6. cookie=0x40000

Assertion failed in file ../../dapl_module_poll.c at line 3973: 0

internal ABORT - process 2

[1:node44] unexpected disconnect completion event from [2:node58]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 1

[3:node58] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 3

[6:node28] unexpected disconnect completion event from [0:node44]

[7:node28] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 7

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 6

[5:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

[4:node59] unexpected disconnect completion event from [0:node44]

Assertion failed in file ../../dapl_module_util.c at line 1593: 0

internal ABORT - process 4

internal ABORT - process 5

rank 0 in job 1 node44_33470 caused collective abort of all ranks

exit status of rank 0: return code 1

[Ending Job 36715]

0 Kudos
1 Reply
Dmitry_K_Intel2
Employee
158 Views
Hi Fouad,

It can be useful to know Intel MPI version, DAPL version, Fabric (or Device) used in that run, the whole command line. What fast fabric do use (Infiniband)?

Regards!
Dmitry
Reply