https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426804#M9987 <P>Hi,</P> <P>&nbsp;</P> <P>below is the answer of you first question:</P> <P>&nbsp;</P> <P>NAME="openSUSE Leap"<BR />VERSION="15.2"<BR />ID="opensuse-leap"<BR />ID_LIKE="suse opensuse"<BR />VERSION_ID="15.2"<BR />PRETTY_NAME="openSUSE Leap 15.2"<BR />ANSI_COLOR="0;32"<BR />CPE_NAME="cpe:/o:opensuse:leap:15.2"<BR />BUG_REPORT_URL="<A href="https://bugs.opensuse.org" target="_blank">https://bugs.opensuse.org</A>"<BR />HOME_URL="<A href="https://www.opensuse.org/" target="_blank">https://www.opensuse.org/</A>"</P> <P>&nbsp;</P> <P>For the second question, i guess I am&nbsp;<SPAN>using a terminal to run the application manually. But, I am not sure, because the workstation is&nbsp;</SPAN></P> <P><SPAN>not&nbsp;configured by me. Meanwhile, I plan to close this discussion because our project is close to end and maybe we don't need to use this&nbsp;</SPAN></P> <P><SPAN>workstation anymore. Anyway, really thanks for your kindly help, which helps me a lot!</SPAN></P> <P>&nbsp;</P> <P><SPAN>Thanks and best regards,</SPAN></P> <P>&nbsp;</P> <P><SPAN>Xu</SPAN></P> Tue, 01 Nov 2022 16:28:11 GMT xulisdu 2022-11-01T16:28:11Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417089#M9860 <P>Hi,</P> <P>&nbsp;</P> <P>I am reading the article "Essential Guide to Distributed Memory Coarray Fortran with the Intel Fortran Compiler for Linux“ (<A href="https://www.intel.com/content/www/us/en/developer/articles/technical/distributed-memory-coarray-fortran-with-the-intel-fortran-compiler-for-linux-essential.html" target="_blank" rel="noopener">Essential Guide to Distributed Memory Coarray Fortran with the Intel...</A>), following which I should try if the&nbsp;<SPAN>MPI 'hello world' program from that article can be run on the machine&nbsp;</SPAN><SPAN>successfully first. The&nbsp;MPI 'hello world' program (MPI_hello.f90) reads:</SPAN></P> <PRE> Program hello_world use mpi integer rank, size, ierror, tag, status(MPI_STATUS_SIZE) call MPI_INIT(ierror) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror) print*, 'node', rank, ': Hello world' call MPI_FINALIZE(ierror) end</PRE> <P>&nbsp;I compile it by:</P> <P>&nbsp; mpiifort -c helloworld.f90</P> <P>&nbsp; mpiifort -o hello helloworld.o</P> <P>&nbsp;</P> <P>But when I run ./hello I get some errors like</P> <P>[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:pid109993.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:pid109993.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />leonhard-15:pid109993: PSM3 can't open nic unit: 0 (err=23)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:109993:0:109993] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1488e236155c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x1488e2361712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1488e63d6b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1488e656b5a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1488e7a0a85b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1488e53dc3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />Segmentation fault (core dumped)</P> <P>&nbsp;</P> <P>My ifort and mpiifort versions are</P> <P>mpiifort for the Intel(R) MPI Library 2021.6 for Linux*<BR />Copyright Intel Corporation.<BR />ifort version 2021.6.0</P> <P>&nbsp;</P> <P>And the intel processor for this machine is&nbsp;Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz.</P> <P>This program can be run successfully on another machine with the processor&nbsp;Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz. I am new for linux machine. And I don't know what to do about this. Could you give me some suggestions?&nbsp;</P> <P>&nbsp;</P> Sat, 24 Sep 2022 18:34:41 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417089#M9860 xulisdu 2022-09-24T18:34:41Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417292#M9866 <P>Hi,</P> <P>&nbsp;</P> <P>Thanks for posting in Intel Communities.</P> <P>&nbsp;</P> <P>Could you please let us know the OS details, and hardware details you are using?&nbsp;</P> <P>&nbsp;</P> <P>Could you please confirm if you can run the below command without any error?</P> <LI-CODE lang="markup">mpirun -n 3 hostname</LI-CODE> <P>&nbsp;</P> <P>Could you please provide us with the outputs for the below commands:</P> <LI-CODE lang="markup">1) I_MPI_DEBUG=30 FI_LOG_LEVEL=debug mpirun -n 4 ./hello 2) lscpu</LI-CODE> <P>Could you please let us know on how many nodes you are running MPI code(single or multi-node)?</P> <P>And also, we recommend you use 2 or more ranks to launch your MPI Applications. So, use the below commands for compiling and running the MPI code:&nbsp;</P> <LI-CODE lang="markup">mpiifort helloworld.f90 -o hello mpirun -n 4 ./hello</LI-CODE> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> <P>&nbsp;</P> Mon, 26 Sep 2022 10:56:11 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417292#M9866 VarshaS_Intel 2022-09-26T10:56:11Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417390#M9868 <P>Hi Varsha,</P> <P>&nbsp;</P> <P>thank you very much for your help!</P> <P>&nbsp;</P> <P>The OS is</P> <P>"Linux version 5.3.18-150300.59.43-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux))"；</P> <P>The results after "mpirun -n 3 hostname":&nbsp;leonhard-15 leonhard-15 leonhard-15;</P> <P>The results after " I_MPI_DEBUG=30 FI_LOG_LEVEL=debug mpirun -n 4 ./hello":</P> <P>MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.<BR />Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.<BR />MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.<BR />[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)<BR />[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.<BR />[0] MPI startup(): library kind: release<BR />MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.<BR />Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.<BR />MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.<BR />[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)<BR />[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.<BR />[0] MPI startup(): library kind: release<BR />[0] MPI startup(): libfabric version: 1.13.2rc1-impi<BR />libfabric:88637:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88637:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88637:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88637:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88637:core:mr:ofi_default_cache_size():78&lt;info&gt; default cache size=2816918428<BR />MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.<BR />Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.<BR />[0] MPI startup(): libfabric version: 1.13.2rc1-impi<BR />MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.<BR />[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)<BR />[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.<BR />[0] MPI startup(): library kind: release<BR />libfabric:88638:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88638:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88638:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88638:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88638:core:mr:ofi_default_cache_size():78&lt;info&gt; default cache size=2816918428<BR />libfabric:88637:core:core:ofi_register_provider():474&lt;info&gt; registering provider: verbs (113.20)<BR />libfabric:88637:core:core:ofi_register_provider():474&lt;info&gt; registering provider: tcp (113.20)<BR />libfabric:88637:core:core:ofi_register_provider():474&lt;info&gt; registering provider: sockets (113.20)<BR />libfabric:88637:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88637:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88637:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ZE not supported<BR />libfabric:88637:core:core:ofi_register_provider():474&lt;info&gt; registering provider: shm (114.0)<BR />MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.<BR />Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.<BR />libfabric:88637:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88637:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88637:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88637:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88637:core:core:ofi_register_provider():474&lt;info&gt; registering provider: ofi_rxm (113.20)<BR />MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.<BR />[0] MPI startup(): Intel(R) MPI Library, Version 2021.6 Build 20220227 (id: 28877f3f32)<BR />[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.<BR />[0] MPI startup(): library kind: release<BR />libfabric:88638:core:core:ofi_register_provider():474&lt;info&gt; registering provider: verbs (113.20)<BR />[0] MPI startup(): libfabric version: 1.13.2rc1-impi<BR />libfabric:88638:core:core:ofi_register_provider():474&lt;info&gt; registering provider: tcp (113.20)<BR />libfabric:88639:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88639:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88639:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88639:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88639:core:mr:ofi_default_cache_size():78&lt;info&gt; default cache size=2816918428<BR />libfabric:88638:core:core:ofi_register_provider():474&lt;info&gt; registering provider: sockets (113.20)<BR />libfabric:88638:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88638:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88638:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ZE not supported<BR />libfabric:88638:core:core:ofi_register_provider():474&lt;info&gt; registering provider: shm (114.0)<BR />libfabric:88638:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88638:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88638:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88638:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88638:core:core:ofi_register_provider():474&lt;info&gt; registering provider: ofi_rxm (113.20)<BR />libfabric:88639:core:core:ofi_register_provider():474&lt;info&gt; registering provider: verbs (113.20)<BR />[0] MPI startup(): libfabric version: 1.13.2rc1-impi<BR />libfabric:88639:core:core:ofi_register_provider():474&lt;info&gt; registering provider: tcp (113.20)<BR />libfabric:88640:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88640:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88639:core:core:ofi_register_provider():474&lt;info&gt; registering provider: sockets (113.20)<BR />libfabric:88640:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88640:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88640:core:mr:ofi_default_cache_size():78&lt;info&gt; default cache size=2816918428<BR />libfabric:88639:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88639:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88639:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ZE not supported<BR />libfabric:88639:core:core:ofi_register_provider():474&lt;info&gt; registering provider: shm (114.0)<BR />libfabric:88639:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88639:core:core:ofi_hmem_init():209&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported<BR />libfabric:88639:core:core:ze_hmem_dl_init():422&lt;warn&gt; Failed to dlopen libze_loader.so<BR />libfabric:88639:core:core:ofi_hmem_init():214&lt;warn&gt; Failed to initialize hmem iface FI_HMEM_ZE: No data available<BR />libfabric:88639:core:core:ofi_register_provider():474&lt;info&gt; registering provider: ofi_rxm (113.20)<BR />libfabric:88640:core:core:ofi_register_provider():474&lt;info&gt; registering provider: verbs (113.20)<BR />libfabric:88640:core:core:ofi_register_provider():474&lt;info&gt; registering provider: tcp (113.20)<BR />libfabric:88640:core:core:ofi_register_provider():474&lt;info&gt; registering provider: sockets (113.20)<BR />libfabric:88640:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_CUDA not supported<BR />libfabric:88640:core:core:ofi_hmem_init():222&lt;info&gt; Hmem iface FI_HMEM_ROCR not supported</P> <P>............................</P> <P>system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:88695:0:88695] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14cf33f1755c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x14cf33f17712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14cf37f8cb2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14cf381215a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14cf395c085b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14cf36f923ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />--------------------------------------------------------------------------<BR />mpirun noticed that process rank 3 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>The results after ”Iscpu":</P> <P>Architecture: x86_64<BR />CPU op-mode(s): 32-bit, 64-bit<BR />Byte Order: Little Endian<BR />Address sizes: 46 bits physical, 48 bits virtual<BR />CPU(s): 144<BR />On-line CPU(s) list: 0-143<BR />Thread(s) per core: 2<BR />Core(s) per socket: 18<BR />Socket(s): 4<BR />NUMA node(s): 4<BR />Vendor ID: GenuineIntel<BR />CPU family: 6<BR />Model: 85<BR />Model name: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz<BR />Stepping: 7<BR />CPU MHz: 1200.051<BR />CPU max MHz: 4000.0000<BR />CPU min MHz: 1200.0000<BR />BogoMIPS: 6200.00<BR />Virtualization: VT-x<BR />L1d cache: 32K<BR />L1i cache: 32K<BR />L2 cache: 1024K<BR />L3 cache: 25344K<BR />NUMA node0 CPU(s): 0-17,72-89<BR />NUMA node1 CPU(s): 18-35,90-107<BR />NUMA node2 CPU(s): 36-53,108-125<BR />NUMA node3 CPU(s): 54-71,126-143<BR />Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><SPAN>Could you please let us know on how many nodes you are running MPI code(single or multi-node)?</SPAN></P> <P><SPAN>To be honest, I am new for this. How can I know how many nodes I used? I just type "./hello", so I guess only single node?</SPAN></P> <P>&nbsp;</P> <P><SPAN>I also try&nbsp;mpiifort helloworld.f90 -o hello; mpirun -n 4 ./hello. But I still get a lot of errors like before with the final sentence&nbsp;</SPAN></P> <P><SPAN>"mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault)."<BR /></SPAN></P> <P>&nbsp;</P> <P><SPAN>Thank you again!</SPAN></P> <P>&nbsp;</P> <P><SPAN>Best regards,</SPAN></P> <P>&nbsp;</P> <P><SPAN>Xu</SPAN></P> Mon, 26 Sep 2022 18:47:58 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1417390#M9868 xulisdu 2022-09-26T18:47:58Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1419347#M9890 <P>Hi,</P> <P>&nbsp;</P> <P>Thanks for providing the details.</P> <P>&nbsp;</P> <P>Could you please please let us know the commands you are using for the slurm(to take nodes)?</P> <P>&nbsp;</P> <P>Could you please try following the below steps:</P> <LI-CODE lang="markup">export I_MPI_PIN_RESPECT_CPUSET=0 unset I_MPI_PMI_LIBRARY mpiifort helloworld.f90 -o hello mpirun -n 4 ./hello</LI-CODE> <P>&nbsp;</P> <P>Could you please let us know if you are still facing any issues?</P> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> <P>&nbsp;</P> <P>&nbsp;</P> Tue, 04 Oct 2022 11:22:16 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1419347#M9890 VarshaS_Intel 2022-10-04T11:22:16Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421047#M9912 <P>Hi,</P><P><BR /></P><P>We have not heard back from you. Could you please provide an update on your issue?</P><P><BR /></P><P>Thanks &amp; Regards,</P><P>Varsha</P><BR /> Tue, 11 Oct 2022 11:10:18 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421047#M9912 VarshaS_Intel 2022-10-11T11:10:18Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421052#M9913 <P>Hi,</P> <P>&nbsp;</P> <P>I am sorry for my late reply. Currently it is still not working.</P> <P>&nbsp;</P> <P>I try&nbsp; following commands, and the error report you could find below.</P> <P>&nbsp;</P> <P>PS: if I switch off the coarray option in my individual code (not helloworld.f90), it can be run normally.&nbsp;</P> <P>&nbsp;</P> <P>Thank you very much and best regards,</P> <P>&nbsp;</P> <P>Xu</P> <P>&nbsp;</P> <P>============================ error report =================================================</P> <P>&nbsp;</P> <P>[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank0: PSM3 can't open nic unit: 0 (err=23)<BR />leonhard-15:rank0.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank0.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7590 :0:7590] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x148fd5adf55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x148fd5adf712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x148fd9b54b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x148fd9ce95a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x148fdb18885b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x148fd8b5a3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank3: PSM3 can't open nic unit: 0 (err=23)<BR />qelr_alloc_context: Failed to allocate context for device.<BR />leonhard-15:rank3.hello: Unable to open qedr0: Invalid argument<BR />leonhard-15:rank3.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7593 :0:7593] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14bac3b6f55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x14bac3b6f712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14bac7be4b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14bac7d795a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14bac921885b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14bac6bea3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank2: PSM3 can't open nic unit: 0 (err=23)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7592 :0:7592] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x145b809cb55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x145b809cb712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x145b84a40b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x145b84bd55a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x145b8607485b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x145b83a463ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank1: PSM3 can't open nic unit: 0 (err=23)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7591 :0:7591] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1490c9b3055c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x1490c9b30712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1490cdba5b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1490cdd3a5a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1490cf1d985b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1490ccbab3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank1.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank1.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank2.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank2.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />--------------------------------------------------------------------------<BR />mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank0: PSM3 can't open nic unit: 0 (err=23)<BR />leonhard-15:rank0.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank0.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7590 :0:7590] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x148fd5adf55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x148fd5adf712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x148fd9b54b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x148fd9ce95a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x148fdb18885b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x148fd8b5a3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank3: PSM3 can't open nic unit: 0 (err=23)<BR />qelr_alloc_context: Failed to allocate context for device.<BR />leonhard-15:rank3.hello: Unable to open qedr0: Invalid argument<BR />leonhard-15:rank3.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7593 :0:7593] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x14bac3b6f55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x14bac3b6f712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x14bac7be4b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x14bac7d795a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x14bac921885b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x14bac6bea3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank2: PSM3 can't open nic unit: 0 (err=23)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7592 :0:7592] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x145b809cb55c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x145b809cb712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x145b84a40b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x145b84bd55a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x145b8607485b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x145b83a463ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 22<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 14<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_qp:749]create qp: failed on ibv_cmd_create_qp with 95<BR />[qelr_create_cq:256]create cq: failed with rc = 22<BR />leonhard-15:rank1: PSM3 can't open nic unit: 0 (err=23)<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:<BR />MPIR_Init_thread(178)........:<BR />MPID_Init(1532)..............:<BR />MPIDI_OFI_mpi_init_hook(1552):<BR />create_vni_context(2131).....: OFI endpoint open failed (ofi_init.c:2131:create_vni_context:Invalid argument)<BR />[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247<BR />:<BR />system msg for write_line failure : Bad file descriptor<BR />[leonhard-15:7591 :0:7591] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))<BR />==== backtrace ====<BR />0 /usr/lib64/libucs.so.0(+0x1b55c) [0x1490c9b3055c]<BR />1 /usr/lib64/libucs.so.0(+0x1b712) [0x1490c9b30712]<BR />2 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPIR_Err_return_comm+0x15d) [0x1490cdba5b2d]<BR />3 /opt/intel/oneapi/mpi/2021.6.0//lib/release/libmpi.so.12(MPI_Init+0x150) [0x1490cdd3a5a0]<BR />4 /opt/intel/oneapi/mpi/2021.6.0//lib/libmpifort.so.12(PMPI_INIT+0x1b) [0x1490cf1d985b]<BR />5 ./hello() [0x4030ee]<BR />6 ./hello() [0x403092]<BR />7 /lib64/libc.so.6(__libc_start_main+0xea) [0x1490ccbab3ea]<BR />8 ./hello() [0x402faa]<BR />===================<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank1.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank1.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />[0] MPI startup(): FI_PSM3_UUID was not generated, please set it to avoid possible resources ownership conflicts between MPI processes<BR />leonhard-15:rank2.hello: Unable to create send CQ of size 5080 on qedr0: Invalid argument<BR />leonhard-15:rank2.hello: Unable to initialize verbs NIC /sys/class/infiniband/qedr0 (unit 0:0)<BR />--------------------------------------------------------------------------<BR />mpirun noticed that process rank 0 with PID 0 on node leonhard-15 exited on signal 11 (Segmentation fault).<BR /><BR /></P> Tue, 11 Oct 2022 11:34:03 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421052#M9913 xulisdu 2022-10-11T11:34:03Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421053#M9914 <P>Hi,</P> <P>&nbsp;</P> <P>the "following commands" in the last letter means:</P> <P>&nbsp;</P> <P><SPAN>export I_MPI_PIN_RESPECT_CPUSET=0 </SPAN></P> <P><SPAN>unset I_MPI_PMI_LIBRARY </SPAN></P> <P><SPAN>mpiifort helloworld.f90 -o hello </SPAN></P> <P><SPAN>mpirun -n 4 ./hello</SPAN></P> <P>&nbsp;</P> <P><SPAN>Best,</SPAN></P> <P>&nbsp;</P> <P><SPAN>Xu</SPAN></P> Tue, 11 Oct 2022 11:35:55 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1421053#M9914 xulisdu 2022-10-11T11:35:55Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1422534#M9944 <P>Hi,</P> <P>&nbsp;</P> <P>Thanks for providing the information.</P> <P>&nbsp;</P> <P>Could you please provide us with the output of the cluster checker, so that we can confirm whether you are running on a single node or multiple nodes?</P> <LI-CODE lang="markup">clck -F health_user</LI-CODE> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> Mon, 17 Oct 2022 11:08:28 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1422534#M9944 VarshaS_Intel 2022-10-17T11:08:28Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1423248#M9948 <P>Hi,</P> <P>&nbsp;</P> <P>thanks for you reply!</P> <P>&nbsp;</P> <P>When I run&nbsp;<SPAN>clck -F health_user, I get</SPAN></P> <P>&nbsp;</P> <P>Intel(R) Cluster Checker 2021 Update 6 (build 20220318)</P> <P>No nodes specified or discovered. Please refer to the Intel(R) Cluster Checker User Guide for information about how to specify nodes to include in testing.</P> <P>&nbsp;</P> <P>Best regards,</P> <P>&nbsp;</P> <P>Xu</P> Wed, 19 Oct 2022 07:31:45 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1423248#M9948 xulisdu 2022-10-19T07:31:45Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1425066#M9962 <P>Hi,</P> <P>&nbsp;</P> <P>Could you please try using the below command and let us know if it works without any errors? Please try the below command:</P> <P>&nbsp;</P> <LI-CODE lang="markup">For Compiling the code: mpiifort helloworld.f90 -o hellofortran For Running the code: I_MPI_DEBUG=30 mpirun -genv I_MPI_FABRICS=shm -genv FI_PROVIDER=shm -n 2 ./hellofortran</LI-CODE> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> <P>&nbsp;</P> Wed, 26 Oct 2022 07:18:42 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1425066#M9962 VarshaS_Intel 2022-10-26T07:18:42Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426229#M9980 <P>Hi,</P> <P>&nbsp;</P> <P>I get</P> <P>&nbsp;</P> <P>mpirun: Error: unknown option "-genv"<BR />Type 'mpirun --help' for usage.</P> <P>&nbsp;</P> <P>It seems there is none such a command -genv.</P> <P>&nbsp;</P> <P>Thanks and best regards,</P> <P>&nbsp;</P> <P>Xu</P> Sun, 30 Oct 2022 18:07:24 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426229#M9980 xulisdu 2022-10-30T18:07:24Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426400#M9983 <P>Hi,</P> <P>&nbsp;</P> <P>We are able to get the results without any errors using the above command:</P> <LI-CODE lang="markup">I_MPI_DEBUG=30 mpirun -genv I_MPI_FABRICS=shm -genv FI_PROVIDER=shm -n 2 ./hellofortran</LI-CODE> <P>Please find the below screenshot:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="VarshaS_Intel_0-1667216516193.png" style="width: 400px;"><img src="https://community.intel.com/t5/image/serverpage/image-id/34692i41C4D36749BBC4CA/image-size/medium?v=v2&amp;px=400&amp;whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright" role="button" title="VarshaS_Intel_0-1667216516193.png" alt="VarshaS_Intel_0-1667216516193.png" /></span></P> <P>&nbsp;</P> <P>But, you can also try using the below command, and please do let us know if you are able to get the expected results:</P> <LI-CODE lang="markup">I_MPI_DEBUG=30 I_MPI_FABRICS=shm FI_PROVIDER=shm mpirun -n 2 ./hellofortran</LI-CODE> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> <P>&nbsp;</P> <P>&nbsp;</P> Mon, 31 Oct 2022 11:42:29 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426400#M9983 VarshaS_Intel 2022-10-31T11:42:29Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426428#M9984 <P>Hi,&nbsp;</P> <P>&nbsp;</P> <P>please find the screenshot below</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="fig 1" style="width: 747px;"><img src="https://community.intel.com/t5/image/serverpage/image-id/34695iEA9D61BD54EBF4B6/image-size/large/is-moderation-mode/true?v=v2&amp;px=999&amp;whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright" role="button" title="1667223289302.png" alt="fig 1" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">fig 1</span></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="fig2" style="width: 772px;"><img src="https://community.intel.com/t5/image/serverpage/image-id/34696i67E12EA6B977E7FB/image-size/large/is-moderation-mode/true?v=v2&amp;px=999&amp;whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright" role="button" title="1667223379939.png" alt="fig2" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">fig2</span></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Is this the correct results? There are still some sentences like 'Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.' and ' PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.'</P> <P>&nbsp;</P> <P>Thanks and best regards,</P> <P>&nbsp;</P> <P>Xu&nbsp;</P> Mon, 31 Oct 2022 13:41:30 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426428#M9984 xulisdu 2022-10-31T13:41:30Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426726#M9986 <P>Hi,</P> <P>&nbsp;</P> <P>Thanks for providing the information.</P> <P>&nbsp;</P> <P>Could you please provide the operating system you are using with the help of below command?</P> <LI-CODE lang="markup">command: cat /etc/os-release</LI-CODE> <P>And also, could you please let us know how you are running the MPI application?</P> <P>Are you using a terminal to run the application manually? or using any other job schedulers mentioned in <A href="http://&nbsp;https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/running-applications/job-schedulers-support.html&nbsp;" target="_self">link&nbsp;</A>?</P> <P>&nbsp;</P> <P>Thanks &amp; Regards,</P> <P>Varsha</P> <P>&nbsp;</P> Tue, 01 Nov 2022 10:28:14 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426726#M9986 VarshaS_Intel 2022-11-01T10:28:14Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426804#M9987 <P>Hi,</P> <P>&nbsp;</P> <P>below is the answer of you first question:</P> <P>&nbsp;</P> <P>NAME="openSUSE Leap"<BR />VERSION="15.2"<BR />ID="opensuse-leap"<BR />ID_LIKE="suse opensuse"<BR />VERSION_ID="15.2"<BR />PRETTY_NAME="openSUSE Leap 15.2"<BR />ANSI_COLOR="0;32"<BR />CPE_NAME="cpe:/o:opensuse:leap:15.2"<BR />BUG_REPORT_URL="<A href="https://bugs.opensuse.org" target="_blank">https://bugs.opensuse.org</A>"<BR />HOME_URL="<A href="https://www.opensuse.org/" target="_blank">https://www.opensuse.org/</A>"</P> <P>&nbsp;</P> <P>For the second question, i guess I am&nbsp;<SPAN>using a terminal to run the application manually. But, I am not sure, because the workstation is&nbsp;</SPAN></P> <P><SPAN>not&nbsp;configured by me. Meanwhile, I plan to close this discussion because our project is close to end and maybe we don't need to use this&nbsp;</SPAN></P> <P><SPAN>workstation anymore. Anyway, really thanks for your kindly help, which helps me a lot!</SPAN></P> <P>&nbsp;</P> <P><SPAN>Thanks and best regards,</SPAN></P> <P>&nbsp;</P> <P><SPAN>Xu</SPAN></P> Tue, 01 Nov 2022 16:28:11 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1426804#M9987 xulisdu 2022-11-01T16:28:11Z https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1428519#M10011 <P>Hi,</P><P><BR /></P><P>&gt;&gt;<EM>But, I am not sure, because the workstation is not configured by me.</EM></P><P>The warning you are facing might be the configuration in the workstation</P><P><BR /></P><P>&gt;&gt;<EM>Is this the correct results?</EM></P><P>Yes, from the screenshot(fig2), we are able to see the correct results of the MPI code.</P><P><BR /></P><P>&gt;&gt;<EM> I plan to close this discussion.</EM></P><P>Thanks for the confirmation. We are closing this thread from our end. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.</P><P><BR /></P><P>Thanks &amp; Regards,</P><P>Varsha</P><BR /> Tue, 08 Nov 2022 17:28:48 GMT https://community.intel.com/t5/Intel-MPI-Library/Intel-MPI-helloworld-quot-program-not-work-on-the-machine/m-p/1428519#M10011 VarshaS_Intel 2022-11-08T17:28:48Z