Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
162 Views

I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

Hi everyone,

On a node with single E5-2667v3 (8 phys. cores),  single ConnectX-4 board (81:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]), CentOS 7, UCX version 1.6.0, I'm experiencing an unexpected program action with I_MPI_ASYNC_PROGRESS=1 with "release_mt" and "debug_mt" library kind.

Intel MPI library version is 2019 Update 5, also re-checked with Update 9.

The program code is a simple MPI_init()/MPI_Finalize() pair:

 

#include <iostream>
#include <mpi.h>
#include <assert.h>


int main(int argc, char **argv)
{
    int thrlev;
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &thrlev);
    assert(thrlev == MPI_THREAD_MULTIPLE);
    MPI_Finalize();
    return 0;
}

 

 

Built simply with: mpiicpc -o myapp myapp.cpp

Ran it as:

 

. /xxx/compilers_and_libraries_2020.0.166/linux/bin/compilervars.sh intel64 
. /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/mpivars.sh debug_mt
module load gcc/7.2.0

export I_MPI_DEBUG=1000
export FI_PROVIDER=verbs
export I_MPI_ASYNC_PROGRESS=1

mpiexec.hydra -machinefile hostfile -np 2 -ppn 2 --errfile-pattern=err.%r --outfile-pattern=out.%r ./myapp

 

I have in "out.0": 

 

MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.9.0a1-impi
[0] MPI startup(): libfabric provider: verbs;ofi_rxm
[0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1
[0] MPI startup(): addrname_len: 16, addrname_firstlen: 16
[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1
[0] MPI startup(): selected platform: unknown
[0] MPI startup(): Platform was not configured (platform=auto will be ignored).
[0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat"
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 8
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       30047    n944011    {8,9}
[0] MPI startup(): 1       30048    n944011    {8,9}
[0] MPI startup(): I_MPI_ROOT=/xxx/compilers_and_libraries_2020.0.166/linux/mpi
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0
[0] MPI startup(): I_MPI_PIN_UNIT=-1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_ASYNC_PROGRESS=1
[0] MPI startup(): I_MPI_DEBUG=1000
[0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2
[0] MPI startup(): threading: thread: 0, processor: 15
[0] MPI startup(): threading: thread: 1, processor: 14

 

without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only this difference in output, the action of program is the same:

 

@@ -1,9 +1,10 @@
 MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
 [0] MPI startup(): libfabric version: 1.9.0a1-impi
-[0] MPI startup(): libfabric provider: verbs;ofi_rxm
+[0] MPI startup(): libfabric provider: mlx
+[0] MPI startup(): detected mlx provider, set device name to "mlx"
 [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1
-[0] MPI startup(): addrname_len: 16, addrname_firstlen: 16
-[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1
+[0] MPI startup(): addrname_len: 512, addrname_firstlen: 512
+[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 1030, num_parts: 1
 [0] MPI startup(): selected platform: unknown
 [0] MPI startup(): Platform was not configured (platform=auto will be ignored).
 [0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat"

 

The program is checked on another system with IMPI 2019u9, and it works there fine.

Could you suggest any actions to do to isolate and fix this?

--
Regards,
Alexey

0 Kudos
8 Replies
Highlighted
Moderator
146 Views

Hi Alexey,


Could you please elaborate the unexpected program action that you are referring to? We haven't observed any error messages in the output of both the versions.


Regards

Prasanth


0 Kudos
Highlighted
139 Views

Hi Prasanth,

as I noticed with a debugger attached, program exits inside MPI_Init_thread(), doesn't seem to get any signal, didn't elaborate any further details yet.

I changed the reproducer code like this:

#include <iostream>
#include <mpi.h>
#include <assert.h>


int main(int argc, char **argv)
{
    int thrlev;
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &thrlev);
    assert(thrlev == MPI_THREAD_MULTIPLE);
    std::cout << "Hello" << std::endl;
    MPI_Finalize();
    return 0;
}


The problem is: it doesn't print "Hello".

--
Regards,
Alexey

 

0 Kudos
Highlighted
117 Views

With I_MPI_HYDRA_DEBUG=1 i've got the output:

[mpiexec@n944004] Launch arguments: /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hyd
ra_bstrap_proxy --upstream-host n944004 --upstream-port 44997 --pgid 0 --launcher ssh --launcher-number 0 --base-pa
th /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --tim
e-left -1 --collective-launch 1 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /xxx
/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-sign
al 15 
[proxy:0:0@n944004] Warning - oversubscription detected: 2 processes will be placed on 0 cores
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_maxes
[proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_maxes
[proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_appnum
[proxy:0:0@n944004] PMI response: cmd=appnum appnum=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_appnum
[proxy:0:0@n944004] PMI response: cmd=appnum appnum=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-0 value=/dev/shm/Intel_MPI_RRkf1r
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_RRkf1r
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=bc-0 value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_20759_0 key=bc-1 value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_O5oWwX
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_O5oWwX
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=bc-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=bc-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719

 

--
Regards,
Alexey

0 Kudos
Highlighted
119 Views

With I_MPI_HYDRA_DEBUG=1, the last lines of output are:

[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=bc-0 value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_23009_0 key=bc-1 value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_SIF710
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=sharedFilename[0]-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_SIF710
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_23009_0 key=bc-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=bc-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719

 

 

With attached gdb, made a breakpoint on MPI_Init_thread(), reduced I_MPI_DEBUG=5, going step-by-step with gdb's "next" command:

Breakpoint 3, PMPI_Init_thread (argc=0x7fffffffc1b0, argv=0x7fffffffc1b8, required=3, provided=0x7fffffffc1a0) at ../../src/mpi/init/initthread.c:893
900     in ../../src/mpi/init/initthread.c
901     in ../../src/mpi/init/initthread.c
902     in ../../src/mpi/init/initthread.c
910     in ../../src/mpi/init/initthread.c
917     in ../../src/mpi/init/initthread.c
938     in ../../src/mpi/init/initthread.c
941     in ../../src/mpi/init/initthread.c
944     in ../../src/mpi/init/initthread.c
945     in ../../src/mpi/init/initthread.c
946     in ../../src/mpi/init/initthread.c
949     in ../../src/mpi/init/initthread.c
955     in ../../src/mpi/init/initthread.c
956     in ../../src/mpi/init/initthread.c
958     in ../../src/mpi/init/initthread.c
MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.9.0a1-impi
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       23026    n944004    {10,11}
[0] MPI startup(): 1       23019    n944004    {10,11}
[0] MPI startup(): I_MPI_ROOT=/uls/7/intel-2020/compilers_and_libraries_2020.0.166/linux/mpi
[0] MPI startup(): I_MPI_HYDRA_DEBUG=1
[0] MPI startup(): I_MPI_JOB_ABORT_SIGNAL=15
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_PIN_UNIT=-1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_ASYNC_PROGRESS=1
[0] MPI startup(): I_MPI_DEBUG=5
[New Thread 0x2aaad4488700 (LWP 23061)]
959     in ../../src/mpi/init/initthread.c
962     in ../../src/mpi/init/initthread.c
967     in ../../src/mpi/init/initthread.c
968     in ../../src/mpi/init/initthread.c
[0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2
[0] MPI startup(): threading: thread: 0, processor: 15
[0] MPI startup(): threading: thread: 1, processor: 14
[New Thread 0x2aaad58e5700 (LWP 23063)]
969     in ../../src/mpi/init/initthread.c
1000    in ../../src/mpi/init/initthread.c

 

Hope this helps a bit.

--
Regards,
Alexey

0 Kudos
Highlighted
124 Views

Hi Prasanth,

1) last lines of output with I_MPI_HYDRA_DEBUG=1 is here in the attached file "async_issue_hydra.txt.gz"
2) I've attached gdb to both ranks, set breakpoint at MPI_Init_thread(), then went on with "next" gdb commands until the program stopped. (debug_mt library kind, I_MPI_DEBUG=5). The output from gdb (rank=0) is here in the attached file "async_issue_gdb.txt.gz". For rank=1 reported program lines are the same.

Hope this would help.

--
Regards,
Alexey

0 Kudos
Highlighted
Moderator
92 Views

Hi Alexey,


Could you please try with mlx provider? For InfiniBand, mlx is recommended over verbs.

You can set the provider using -: export FI_PROVIDER=mlx to use mlx provider.

I have tried with mlx in my environment on InfiniBand and found no errors (i can see the stdout messages)

Let us know if you observe any error with mlx too.


Regards

Prasanth


0 Kudos
Highlighted
83 Views

Hi Prasanth,

please look at my original post, where I wrote:

>> without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only
>> this difference in output, the action of program is the same:

So: yes, I can confirm that the error persists on this system with "mlx" as well. I don't see such an error on another machine where these modes (release_mt and I_MPI_ASYNC_PROGRESS) work fine on 2019u9, so the problem seems to be machine/environment specific. The question is: how to debug this and how to find the root cause?

--
Regards,
Alexey

 

0 Kudos
Highlighted
Moderator
34 Views

Hi Alexey,


Sorry for the late response.

I suggested mlx since I thought you were using mlx only in 2019u9.

I am escalating this issue to the Subject Matter experts for better support.

Could you please provide the output of the following command:

ucx_info -d | grep Transport


Regards

Prasanth


0 Kudos