Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2159 Discussions

Intel MPI ''helloworld" program not work on the machine

xulisdu
Beginner
‎09-24-2022 11:16 AM
6,363 Views

Hi,

 

I am reading the article "Essential Guide to Distributed Memory Coarray Fortran with the Intel Fortran Compiler for Linux“ (Essential Guide to Distributed Memory Coarray Fortran with the Intel...), following which I should try if the MPI 'hello world' program from that article can be run on the machine successfully first. The MPI 'hello world' program (MPI_hello.f90) reads:

  Program hello_world
   use mpi

   integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)
   
   call MPI_INIT(ierror)
   call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
   call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
   print*, 'node', rank, ': Hello world'
   call MPI_FINALIZE(ierror)

   end

 I compile it by:

  mpiifort -c helloworld.f90

  mpiifort -o hello helloworld.o

 

But when I run ./hello I get some errors like

[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:pid109993.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:pid109993.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
leonhard-15:pid109993: PSM3 can't open nic unit: 0 (err=23)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:109993:0:109993] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1488e236155c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x1488e2361712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1488e63d6b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1488e656b5a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1488e7a0a85b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1488e53dc3ea]
8 ./hello() [0x402faa]
===================
Segmentation fault (core dumped)

 

My ifort and mpiifort versions are

mpiifort for the Intel(R) MPI Library 2021.6 for Linux*
Copyright Intel Corporation.
ifort version 2021.6.0

 

And the intel processor for this machine is Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz.

This program can be run successfully on another machine with the processor Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz. I am new for linux machine. And I don't know what to do about this. Could you give me some suggestions? 

 

0 Kudos

Link Copied

15 Replies
VarshaS_Intel
Moderator
‎09-26-2022 03:48 AM
6,320 Views

Hi,

 

Thanks for posting in Intel Communities.

 

Could you please let us know the OS details, and hardware details you are using? 

 

Could you please confirm if you can run the below command without any error?

mpirun -n 3 hostname

 

Could you please provide us with the outputs for the below commands:

1) I_MPI_DEBUG=30 FI_LOG_LEVEL=debug mpirun -n 4 ./hello
2) lscpu

Could you please let us know on how many nodes you are running MPI code(single or multi-node)?

And also, we recommend you use 2 or more ranks to launch your MPI Applications. So, use the below commands for compiling and running the MPI code: 

mpiifort helloworld.f90 -o hello
mpirun -n 4 ./hello

 

Thanks & Regards,

Varsha

 

0 Kudos
Copy link
xulisdu
Beginner
‎09-26-2022 11:47 AM
6,279 Views
In response to VarshaS_Intel

Hi Varsha,

 

thank you very much for your help!

 

The OS is

"Linux version 5.3.18-150300.59.43-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux))"；

The results after "mpirun -n 3 hostname": leonhard-15 leonhard-15 leonhard-15;

The results after " I_MPI_DEBUG=30 FI_LOG_LEVEL=debug mpirun -n 4 ./hello":

MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.
MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.
[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.
MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.
[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
libfabric:88637:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88637:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88637:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88637:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88637:core:mr:ofi_default_cache_size():78<info> default cache size=2816918428
MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.
[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
libfabric:88638:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88638:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88638:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88638:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88638:core:mr:ofi_default_cache_size():78<info> default cache size=2816918428
libfabric:88637:core:core:ofi_register_provider():474<info> registering provider: verbs (113.20)
libfabric:88637:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:88637:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:88637:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88637:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88637:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ZE not supported
libfabric:88637:core:core:ofi_register_provider():474<info> registering provider: shm (114.0)
MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.
libfabric:88637:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88637:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88637:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88637:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88637:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.
[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
libfabric:88638:core:core:ofi_register_provider():474<info> registering provider: verbs (113.20)
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
libfabric:88638:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:88639:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88639:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88639:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88639:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88639:core:mr:ofi_default_cache_size():78<info> default cache size=2816918428
libfabric:88638:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:88638:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88638:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88638:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ZE not supported
libfabric:88638:core:core:ofi_register_provider():474<info> registering provider: shm (114.0)
libfabric:88638:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88638:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88638:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88638:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88638:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:88639:core:core:ofi_register_provider():474<info> registering provider: verbs (113.20)
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
libfabric:88639:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:88640:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88640:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88639:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:88640:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88640:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88640:core:mr:ofi_default_cache_size():78<info> default cache size=2816918428
libfabric:88639:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88639:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88639:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ZE not supported
libfabric:88639:core:core:ofi_register_provider():474<info> registering provider: shm (114.0)
libfabric:88639:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88639:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:88639:core:core:ze_hmem_dl_init():422<warn> Failed to dlopen libze_loader.so
libfabric:88639:core:core:ofi_hmem_init():214<warn> Failed to initialize hmem iface FI_HMEM_ZE: No data available
libfabric:88639:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:88640:core:core:ofi_register_provider():474<info> registering provider: verbs (113.20)
libfabric:88640:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:88640:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:88640:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:88640:core:core:ofi_hmem_init():222<info> Hmem iface FI_HMEM_ROCR not supported

............................

system msg for write_line failure : Bad file descriptor
[leonhard-15:88695:0:88695] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14cf33f1755c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x14cf33f17712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14cf37f8cb2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14cf381215a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14cf395c085b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14cf36f923ea]
8 ./hello() [0x402faa]
===================
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).

 

 

 

The results after ”Iscpu":

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 144
On-line CPU(s) list: 0-143
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 4
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
Stepping: 7
CPU MHz: 1200.051
CPU max MHz: 4000.0000
CPU min MHz: 1200.0000
BogoMIPS: 6200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-17,72-89
NUMA node1 CPU(s): 18-35,90-107
NUMA node2 CPU(s): 36-53,108-125
NUMA node3 CPU(s): 54-71,126-143
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

 

 

 

Could you please let us know on how many nodes you are running MPI code(single or multi-node)?

To be honest, I am new for this. How can I know how many nodes I used? I just type "./hello", so I guess only single node?

 

I also try mpiifort helloworld.f90 -o hello; mpirun -n 4 ./hello. But I still get a lot of errors like before with the final sentence 

"mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault)."

 

Thank you again!

 

Best regards,

 

Xu

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎10-04-2022 04:21 AM
6,253 Views

Hi,

 

Thanks for providing the details.

 

Could you please please let us know the commands you are using for the slurm(to take nodes)?

 

Could you please try following the below steps:

export I_MPI_PIN_RESPECT_CPUSET=0
unset I_MPI_PMI_LIBRARY
mpiifort helloworld.f90 -o hello
mpirun -n 4 ./hello

 

Could you please let us know if you are still facing any issues?

 

Thanks & Regards,

Varsha

 

 

0 Kudos
Copy link
VarshaS_Intel
Moderator
‎10-11-2022 04:10 AM
6,206 Views

Hi,


We have not heard back from you. Could you please provide an update on your issue?


Thanks & Regards,

Varsha


0 Kudos
Copy link
xulisdu
Beginner
‎10-11-2022 04:34 AM
6,200 Views
In response to VarshaS_Intel

Hi,

 

I am sorry for my late reply. Currently it is still not working.

 

I try  following commands, and the error report you could find below.

 

PS: if I switch off the coarray option in my individual code (not helloworld.f90), it can be run normally. 

 

Thank you very much and best regards,

 

Xu

 

============================ error report =================================================

 

[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank0: PSM3 can't open nic unit: 0 (err=23)
leonhard-15:rank0.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank0.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7590 :0:7590] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x148fd5adf55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x148fd5adf712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x148fd9b54b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x148fd9ce95a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x148fdb18885b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x148fd8b5a3ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank3: PSM3 can't open nic unit: 0 (err=23)
qelr_alloc_context: Failed to allocate context for device.
leonhard-15:rank3.hello: Unable to open qedr0: Invalid argument
leonhard-15:rank3.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7593 :0:7593] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14bac3b6f55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x14bac3b6f712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14bac7be4b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14bac7d795a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14bac921885b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14bac6bea3ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank2: PSM3 can't open nic unit: 0 (err=23)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7592 :0:7592] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x145b809cb55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x145b809cb712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x145b84a40b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x145b84bd55a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x145b8607485b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x145b83a463ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank1: PSM3 can't open nic unit: 0 (err=23)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7591 :0:7591] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1490c9b3055c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x1490c9b30712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1490cdba5b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1490cdd3a5a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1490cf1d985b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1490ccbab3ea]
8 ./hello() [0x402faa]
===================
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank1.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank1.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank2.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank2.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank0: PSM3 can't open nic unit: 0 (err=23)
leonhard-15:rank0.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank0.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7590 :0:7590] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x148fd5adf55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x148fd5adf712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x148fd9b54b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x148fd9ce95a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x148fdb18885b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x148fd8b5a3ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank3: PSM3 can't open nic unit: 0 (err=23)
qelr_alloc_context: Failed to allocate context for device.
leonhard-15:rank3.hello: Unable to open qedr0: Invalid argument
leonhard-15:rank3.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7593 :0:7593] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14bac3b6f55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x14bac3b6f712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14bac7be4b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14bac7d795a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14bac921885b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14bac6bea3ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank2: PSM3 can't open nic unit: 0 (err=23)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7592 :0:7592] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x145b809cb55c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x145b809cb712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x145b84a40b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x145b84bd55a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x145b8607485b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x145b83a463ea]
8 ./hello() [0x402faa]
===================
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95
[qelr_create_cq:256]create cq: failed with rc = 22
leonhard-15:rank1: PSM3 can't open nic unit: 0 (err=23)
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(178)........:
MPID_Init(1532)..............:
MPIDI_OFI_mpi_init_hook(1552):
create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
:
system msg for write_line failure : Bad file descriptor
[leonhard-15:7591 :0:7591] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1490c9b3055c]
1 /usr/lib64/libucs.so.0(+0x1b712) [0x1490c9b30712]
2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1490cdba5b2d]
3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1490cdd3a5a0]
4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1490cf1d985b]
5 ./hello() [0x4030ee]
6 ./hello() [0x403092]
7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1490ccbab3ea]
8 ./hello() [0x402faa]
===================
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank1.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank1.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes
leonhard-15:rank2.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument
leonhard-15:rank2.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).

In response to VarshaS_Intel
0 Kudos
Copy link
xulisdu
Beginner
‎10-11-2022 04:35 AM
6,198 Views
In response to VarshaS_Intel

Hi,

 

the "following commands" in the last letter means:

 

export I_MPI_PIN_RESPECT_CPUSET=0

unset I_MPI_PMI_LIBRARY

mpiifort helloworld.f90 -o hello

mpirun -n 4 ./hello

 

Best,

 

Xu

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎10-17-2022 04:02 AM
6,111 Views

Hi,

 

Thanks for providing the information.

 

Could you please provide us with the output of the cluster checker, so that we can confirm whether you are running on a single node or multiple nodes?

clck -F health_user

 

Thanks & Regards,

Varsha

0 Kudos
Copy link
xulisdu
Beginner
‎10-19-2022 12:31 AM
6,036 Views
In response to VarshaS_Intel

Hi,

 

thanks for you reply!

 

When I run clck -F health_user, I get

 

Intel(R) Cluster Checker 2021 Update 6 (build 20220318)

No nodes specified or discovered. Please refer to the Intel(R) Cluster Checker User Guide for information about how to specify nodes to include in testing.

 

Best regards,

 

Xu

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎10-26-2022 12:17 AM
5,981 Views

Hi,

 

Could you please try using the below command and let us know if it works without any errors? Please try the below command:

 

For Compiling the code: mpiifort helloworld.f90 -o hellofortran
For Running the code: I_MPI_DEBUG=30 mpirun -genv I_MPI_FABRICS=shm -genv FI_PROVIDER=shm -n 2 ./hellofortran

 

Thanks & Regards,

Varsha

 

0 Kudos
Copy link
xulisdu
Beginner
‎10-30-2022 11:07 AM
5,944 Views
In response to VarshaS_Intel

Hi,

 

I get

 

mpirun: Error: unknown option "-genv"
Type 'mpirun --help' for usage.

 

It seems there is none such a command -genv.

 

Thanks and best regards,

 

Xu

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎10-31-2022 04:40 AM
5,935 Views

Hi,

 

We are able to get the results without any errors using the above command:

I_MPI_DEBUG=30 mpirun -genv I_MPI_FABRICS=shm -genv FI_PROVIDER=shm -n 2 ./hellofortran

Please find the below screenshot:

VarshaS_Intel_0-1667216516193.png

 

But, you can also try using the below command, and please do let us know if you are able to get the expected results:

I_MPI_DEBUG=30 I_MPI_FABRICS=shm FI_PROVIDER=shm mpirun -n 2 ./hellofortran

 

Thanks & Regards,

Varsha

 

 

0 Kudos
Copy link
xulisdu
Beginner
‎10-31-2022 06:41 AM
5,929 Views
In response to VarshaS_Intel

Hi, 

 

please find the screenshot below

 

fig 1fig 1

 

fig2fig2

 

 

Is this the correct results? There are still some sentences like 'Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.' and ' PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.'

 

Thanks and best regards,

 

Xu 

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎11-01-2022 03:25 AM
5,917 Views

Hi,

 

Thanks for providing the information.

 

Could you please provide the operating system you are using with the help of below command?

command: cat /etc/os-release

And also, could you please let us know how you are running the MPI application?

Are you using a terminal to run the application manually? or using any other job schedulers mentioned in link ?

 

Thanks & Regards,

Varsha

 

0 Kudos
Copy link
xulisdu
Beginner
‎11-01-2022 09:28 AM
5,899 Views
In response to VarshaS_Intel

Hi,

 

below is the answer of you first question:

 

NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

 

For the second question, i guess I am using a terminal to run the application manually. But, I am not sure, because the workstation is 

not configured by me. Meanwhile, I plan to close this discussion because our project is close to end and maybe we don't need to use this 

workstation anymore. Anyway, really thanks for your kindly help, which helps me a lot!

 

Thanks and best regards,

 

Xu

In response to VarshaS_Intel
0 Kudos
Copy link
VarshaS_Intel
Moderator
‎11-08-2022 09:28 AM
5,840 Views

Hi,


>>But, I am not sure, because the workstation is not configured by me.

The warning you are facing might be the configuration in the workstation


>>Is this the correct results?

Yes, from the screenshot(fig2), we are able to see the correct results of the MPI code.


>> I plan to close this discussion.

Thanks for the confirmation. We are closing this thread from our end. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.


Thanks & Regards,

Varsha


0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.