topic Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user. in Intel® MPI Library

Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

haoyahao3 — Mon, 09 Dec 2024 02:45:34 GMT

Here is a strange error when using IntelMPI: After successful install and using IntelMPI runtime and complier, I have complied the fowllowing codes for mpi in hello.f90 :

PROGRAM hello_world_mpi include 'mpif.h' integer process_Rank, size_Of_Cluster, ierror, tag call MPI_INIT(ierror) call MPI_COMM_SIZE(MPI_COMM_WORLD, size_Of_Cluster, ierror) call MPI_COMM_RANK(MPI_COMM_WORLD, process_Rank, ierror) print *, 'Hello World from process: ', process_Rank, 'of ', size_Of_Cluster call MPI_FINALIZE(ierror) END PROGRAM

I have complied it by using intel ifork complier by

mpiifort hello.f90 -o hello

Then when I try to run such program as root user:

> mpirun -n 2 ./hello Hello World from process: 1 of 2 Hello World from process: 0 of 2

Everything goes right in root user. When I try to run this in any other user, it get:

> mpirun -n 2 ./hello [1733578910.431461221] RFRLServer7:rank89.hello: Unable to create send CQ of size 5080 on mlx5_bond_0: Cannot allocate memory [1733578910.433054057] RFRLServer7:rank89.hello: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_bond_0 (unit 0:0) RFRLServer7:rank89: PSM3 can't open nic unit: 0 (err=23) Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack: MPIR_Init_thread(193)........: MPID_Init(1715)..............: MPIDI_OFI_mpi_init_hook(1673): create_vni_context(2242).....: OFI endpoint open failed (ofi_init.c:2242:create_vni_context:Invalid argument)

It seems a wrong with this.

The ifork information: ifort (IFORT) 2021.13.1 20240703

The complier is 2024.2.

The intel MPI is Intel(R) MPI Library for Linux* OS, Version 2021.14 Build 20241121 (id: e7829d6).

My cpu is Intel(R) Xeon(R) Platinum 8360H CPU @ 3.00GHz.

The sysmtem is rocky linux 8.10.

Is there anyone could help me with this problem?

The same problem occur when I use the latest complier ifx.

Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

TobiasK — Mon, 09 Dec 2024 09:09:27 GMT

@haoyahao3
Please show the full output of

I_MPI_DEBUG=10 I_MPI_HYDRA_DEBUG=1 mpirun -np 2 IMB-MPI1

Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

haoyahao3 — Wed, 11 Dec 2024 02:53:01 GMT

Here is the ouput for your information:

[user01@RFRLServer7 ~]$ mpirun -np 2 IMB-MPI1 [mpiexec@RFRLServer7] Launch arguments: /opt/intel/oneapi/mpi/2021.14/bin//hydra_bstrap_proxy --upstream-host RFRLServer7 --upstream-port 43083 --pgid 0 --launcher ssh --launcher-number 0 --base-path /opt/intel/oneapi/mpi/2021.14/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /opt/intel/oneapi/mpi/2021.14/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9 [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@RFRLServer7] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=get_maxes [proxy:0:0@RFRLServer7] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=get_appnum [proxy:0:0@RFRLServer7] PMI response: cmd=appnum appnum=0 [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=get_my_kvsname [proxy:0:0@RFRLServer7] PMI response: cmd=my_kvsname kvsname=kvs_3502253_0 [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=get kvsname=kvs_3502253_0 key=PMI_process_mapping [proxy:0:0@RFRLServer7] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2)) [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@RFRLServer7] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=get_maxes [proxy:0:0@RFRLServer7] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=get_appnum [proxy:0:0@RFRLServer7] PMI response: cmd=appnum appnum=0 [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@RFRLServer7] PMI response: cmd=my_kvsname kvsname=kvs_3502253_0 [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=get kvsname=kvs_3502253_0 key=PMI_process_mapping [proxy:0:0@RFRLServer7] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2)) [0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20241121 (id: e7829d6) [0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=barrier_in [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=put kvsname=kvs_3502253_0 key=-bcast-1-0 value=2F6465762F73686D2F496E74656C5F4D50495F6C4E676D597A [proxy:0:0@RFRLServer7] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@RFRLServer7] PMI response: cmd=barrier_out [proxy:0:0@RFRLServer7] PMI response: cmd=barrier_out [proxy:0:0@RFRLServer7] pmi cmd from fd 9: cmd=get kvsname=kvs_3502253_0 key=-bcast-1-0 [proxy:0:0@RFRLServer7] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F6C4E676D597A [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.21.0-impi [0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1) [0] MPI startup(): libfabric provider: psm3 [1733885461.826111329] RFRLServer7:rank0.IMB-MPI1: Unable to create send CQ of size 5080 on mlx5_bond_0: Cannot allocate memory [1733885461.826473879] RFRLServer7:rank0.IMB-MPI1: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_bond_0 (unit 0:0) RFRLServer7:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Unknown error class, error stack: MPIR_Init_thread(193)........: MPID_Init(1715)..............: MPIDI_OFI_mpi_init_hook(1673): create_vni_context(2242).....: OFI endpoint open failed (ofi_init.c:2242:create_vni_context:Invalid argument) [proxy:0:0@RFRLServer7] pmi cmd from fd 6: cmd=abort exitcode=1615247

Is there anything to help you solve this problem?

Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

TobiasK — Thu, 02 Jan 2025 12:11:19 GMT

@haoyahao3 did you configure ulimits correctly?
e.g. follow this description:
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html-single/configuring_infiniband_and_rdma_networks/index#increasing-the-amount-of-memory-that-users-are-allowed-to-pin-in-the-system_configuring-infiniband-and-rdma-networks

Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

haoyahao3 — Mon, 28 Apr 2025 01:31:39 GMT

I have configured the ulimited as the link. But the problem does not seem to be solved

Re: Rocky Linux 8 using IntelMPI occur error when normal users, good in root user.

TobiasK — Mon, 28 Apr 2025 15:35:01 GMT

@haoyahao3 have you verified the settings with

ulimit -l

for the user account in question?

Usually those settings should be set by the MOFED installer, which IB stack are you using?