topic Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread() in Intel® MPI Library

I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Thu, 05 Nov 2020 19:18:07 GMT

Hi everyone,

On a node with single E5-2667v3 (8 phys. cores), single ConnectX-4 board (81:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]), CentOS 7, UCX version 1.6.0, I'm experiencing an unexpected program action with I_MPI_ASYNC_PROGRESS=1 with "release_mt" and "debug_mt" library kind.

Intel MPI library version is 2019 Update 5, also re-checked with Update 9.

The program code is a simple MPI_init()/MPI_Finalize() pair:

#include <iostream> #include <mpi.h> #include <assert.h> int main(int argc, char **argv) { int thrlev; MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &thrlev); assert(thrlev == MPI_THREAD_MULTIPLE); MPI_Finalize(); return 0; }

Built simply with: mpiicpc -o myapp myapp.cpp

Ran it as:

. /xxx/compilers_and_libraries_2020.0.166/linux/bin/compilervars.sh intel64 . /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/mpivars.sh debug_mt module load gcc/7.2.0 export I_MPI_DEBUG=1000 export FI_PROVIDER=verbs export I_MPI_ASYNC_PROGRESS=1 mpiexec.hydra -machinefile hostfile -np 2 -ppn 2 --errfile-pattern=err.%r --outfile-pattern=out.%r ./myapp

I have in "out.0":

MPI startup(): Imported environment partly inaccesible. Map=0 Info=0 [0] MPI startup(): libfabric version: 1.9.0a1-impi [0] MPI startup(): libfabric provider: verbs;ofi_rxm [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1 [0] MPI startup(): addrname_len: 16, addrname_firstlen: 16 [0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1 [0] MPI startup(): selected platform: unknown [0] MPI startup(): Platform was not configured (platform=auto will be ignored). [0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat" MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 8 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 1 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV MPIU_Selection_find_entry(): Layer: PPN, Key: 2 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456 [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 30047 n944011 {8,9} [0] MPI startup(): 1 30048 n944011 {8,9} [0] MPI startup(): I_MPI_ROOT=/xxx/compilers_and_libraries_2020.0.166/linux/mpi [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0 [0] MPI startup(): I_MPI_PIN_UNIT=-1 [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_ASYNC_PROGRESS=1 [0] MPI startup(): I_MPI_DEBUG=1000 [0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2 [0] MPI startup(): threading: thread: 0, processor: 15 [0] MPI startup(): threading: thread: 1, processor: 14

without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only this difference in output, the action of program is the same:

@@ -1,9 +1,10 @@ MPI startup(): Imported environment partly inaccesible. Map=0 Info=0 [0] MPI startup(): libfabric version: 1.9.0a1-impi -[0] MPI startup(): libfabric provider: verbs;ofi_rxm +[0] MPI startup(): libfabric provider: mlx +[0] MPI startup(): detected mlx provider, set device name to "mlx" [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1 -[0] MPI startup(): addrname_len: 16, addrname_firstlen: 16 -[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1 +[0] MPI startup(): addrname_len: 512, addrname_firstlen: 512 +[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 1030, num_parts: 1 [0] MPI startup(): selected platform: unknown [0] MPI startup(): Platform was not configured (platform=auto will be ignored). [0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat"

The program is checked on another system with IMPI 2019u9, and it works there fine.

Could you suggest any actions to do to isolate and fix this?

--
Regards,
Alexey

Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

PrasanthD_intel — Fri, 06 Nov 2020 11:08:11 GMT

Hi Alexey,

Could you please elaborate the unexpected program action that you are referring to? We haven't observed any error messages in the output of both the versions.

Regards

Prasanth

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Fri, 06 Nov 2020 16:02:38 GMT

Hi Prasanth,

as I noticed with a debugger attached, program exits inside MPI_Init_thread(), doesn't seem to get any signal, didn't elaborate any further details yet.

I changed the reproducer code like this:

The problem is: it doesn't print "Hello".

--
Regards,
Alexey

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Fri, 06 Nov 2020 16:40:06 GMT

With I_MPI_HYDRA_DEBUG=1 i've got the output:

[mpiexec@n944004] Launch arguments: /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hyd ra_bstrap_proxy --upstream-host n944004 --upstream-port 44997 --pgid 0 --launcher ssh --launcher-number 0 --base-pa th /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --tim e-left -1 --collective-launch 1 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /xxx /compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-sign al 15 [proxy:0:0@n944004] Warning - oversubscription detected: 2 processes will be placed on 0 cores [proxy:0:0@n944004] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_maxes [proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_maxes [proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_appnum [proxy:0:0@n944004] PMI response: cmd=appnum appnum=0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_appnum [proxy:0:0@n944004] PMI response: cmd=appnum appnum=0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2)) [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2)) [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in [proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-0 value=/dev/shm/Intel_MPI_RRkf1r [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-0 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_RRkf1r [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=bc-0 value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_20759_0 key=bc-1 value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in [proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_O5oWwX [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-1 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_O5oWwX [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=bc-0 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=bc-1 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719

--
Regards,
Alexey

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Fri, 06 Nov 2020 17:01:04 GMT

With I_MPI_HYDRA_DEBUG=1, the last lines of output are:

[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=bc-0 value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_23009_0 key=bc-1 value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0 [proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_SIF710 [proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] PMI response: cmd=barrier_out [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=sharedFilename[0]-1 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_SIF710 [proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_23009_0 key=bc-0 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=bc-1 [proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719

With attached gdb, made a breakpoint on MPI_Init_thread(), reduced I_MPI_DEBUG=5, going step-by-step with gdb's "next" command:

Breakpoint 3, PMPI_Init_thread (argc=0x7fffffffc1b0, argv=0x7fffffffc1b8, required=3, provided=0x7fffffffc1a0) at ../../src/mpi/init/initthread.c:893 900 in ../../src/mpi/init/initthread.c 901 in ../../src/mpi/init/initthread.c 902 in ../../src/mpi/init/initthread.c 910 in ../../src/mpi/init/initthread.c 917 in ../../src/mpi/init/initthread.c 938 in ../../src/mpi/init/initthread.c 941 in ../../src/mpi/init/initthread.c 944 in ../../src/mpi/init/initthread.c 945 in ../../src/mpi/init/initthread.c 946 in ../../src/mpi/init/initthread.c 949 in ../../src/mpi/init/initthread.c 955 in ../../src/mpi/init/initthread.c 956 in ../../src/mpi/init/initthread.c 958 in ../../src/mpi/init/initthread.c MPI startup(): Imported environment partly inaccesible. Map=0 Info=0 [0] MPI startup(): libfabric version: 1.9.0a1-impi [0] MPI startup(): libfabric provider: mlx [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23026 n944004 {10,11} [0] MPI startup(): 1 23019 n944004 {10,11} [0] MPI startup(): I_MPI_ROOT=/uls/7/intel-2020/compilers_and_libraries_2020.0.166/linux/mpi [0] MPI startup(): I_MPI_HYDRA_DEBUG=1 [0] MPI startup(): I_MPI_JOB_ABORT_SIGNAL=15 [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_PIN_UNIT=-1 [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_ASYNC_PROGRESS=1 [0] MPI startup(): I_MPI_DEBUG=5 [New Thread 0x2aaad4488700 (LWP 23061)] 959 in ../../src/mpi/init/initthread.c 962 in ../../src/mpi/init/initthread.c 967 in ../../src/mpi/init/initthread.c 968 in ../../src/mpi/init/initthread.c [0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2 [0] MPI startup(): threading: thread: 0, processor: 15 [0] MPI startup(): threading: thread: 1, processor: 14 [New Thread 0x2aaad58e5700 (LWP 23063)] 969 in ../../src/mpi/init/initthread.c 1000 in ../../src/mpi/init/initthread.c

Hope this helps a bit.

--
Regards,
Alexey

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Fri, 06 Nov 2020 18:01:09 GMT

Hi Prasanth,

1) last lines of output with I_MPI_HYDRA_DEBUG=1 is here in the attached file "async_issue_hydra.txt.gz"
2) I've attached gdb to both ranks, set breakpoint at MPI_Init_thread(), then went on with "next" gdb commands until the program stopped. (debug_mt library kind, I_MPI_DEBUG=5). The output from gdb (rank=0) is here in the attached file "async_issue_gdb.txt.gz". For rank=1 reported program lines are the same.

Hope this would help.

--
Regards,
Alexey

Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

PrasanthD_intel — Tue, 10 Nov 2020 11:28:45 GMT

Hi Alexey,

Could you please try with mlx provider? For InfiniBand, mlx is recommended over verbs.

You can set the provider using -: export FI_PROVIDER=mlx to use mlx provider.

I have tried with mlx in my environment on InfiniBand and found no errors (i can see the stdout messages)

Let us know if you observe any error with mlx too.

Regards

Prasanth

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Wed, 11 Nov 2020 10:29:36 GMT

Hi Prasanth,

please look at my original post, where I wrote:

>> without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only
>> this difference in output, the action of program is the same:

So: yes, I can confirm that the error persists on this system with "mlx" as well. I don't see such an error on another machine where these modes (release_mt and I_MPI_ASYNC_PROGRESS) work fine on 2019u9, so the problem seems to be machine/environment specific. The question is: how to debug this and how to find the root cause?

--
Regards,
Alexey

Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

PrasanthD_intel — Wed, 18 Nov 2020 11:53:33 GMT

Hi Alexey,

Sorry for the late response.

I suggested mlx since I thought you were using mlx only in 2019u9.

I am escalating this issue to the Subject Matter experts for better support.

Could you please provide the output of the following command:

ucx_info -d | grep Transport

Regards

Prasanth

Re: Re:I_MPI_ASYNC_PROGRESS: program exits in MPI_Init_thread()

alexey-medvedev-MSU — Wed, 02 Dec 2020 17:04:26 GMT

Hi Prasanth,

I'm currently on vacation, will return wirg this info on the next week. Thanks.

--
Regards,
Alexey