Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2159 Discussions

IMPI numa_num Assertion Failed

Zach_C
Beginner
‎04-21-2020 07:34 PM
2,876 Views

Hi All - 

I'm getting an error and not quite sure where to begin tracking it down. I'm running a model known to run on our system using:

Intel(R) MPI Library for Linux* OS, Version 2019 Update 6 Build 20191024 (id: 082ae5608)

The code runs for certain core counts(generally smaller processor counts) but errors for some counts with:

Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
  8 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
  9 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f01f66321d4]
 10 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f01f5dba031]
 11 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f01f5f34c5d]
 12 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f01f5e465e4]
 13 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f01f5e1ad8e]
 14 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f01f5f376a7]
 15 /apps/applications/development/compilers/intel/1
 16 Abort(1) on node 2: Internal error
 17 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f983aed31d4]
 18 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f983a65b031]
 19 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f983a7d5c5d]
 20 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f983a6e75e4]
 21 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f983a6bbd8e]
 22 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f983a7d86a7]
 23 /apps/applications/development/compilers/intel/1
 24 Abort(1) on node 3: Internal error
 25 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
 26 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
 27 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f9ede95a1d4]
 28 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f9ede0e2031]
 29 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f9ede25cc5d]
 30 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f9ede16e5e4]
 31 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f9ede142d8e]
 32 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f9ede25f6a7]
 33 /apps/applications/development/compilers/intel/1
 34 Abort(1) on node 1: Internal error
 35 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
 36 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff6ff7ae1d4]
 37 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff6fef36031]
 38 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff6ff0b0c5d]
 39 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff6fefc25e4]
 40 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff6fef96d8e]
 41 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff6ff0b36a7]
 42 /apps/applications/development/compilers/intel/1
 43 Abort(1) on node 84: Internal error

The code does run correctly with this core configuration when started through SLURM using "srun --mpi=pmi2". Can you provide any guidance? 

The machine this is running on is a dual-socket AMD Epyc 7702 with hyperthreading disabled and Ubuntu 18.04 server

Thanks

0 Kudos

Link Copied

6 Replies
PrasanthD_intel
Moderator
‎04-22-2020 03:57 AM
2,876 Views

Hi Zach,

Can you please provide the following:

1) The command line you are following and the output.

2)The output with hostname as parameter(mpirun/mpiexec -np <processes from which error starts showing> hostname)

3)The log report after running the mpi command with I_MPI_DEBUG=5  (I_MPI_DEBUG=5  mpirun/mpiexec -np<> ./Executable)

We need this information to debug the issue.

 

Thanks

Prasanth

 

0 Kudos
Copy link
Zach_C
Beginner
‎04-22-2020 07:20 AM
2,876 Views

Hi Prasanth - 

Thanks for your response. 

(1) The script which starts the code is (note the memory request @ 64G is far beyond what is required, but I've cranked this up to ensure it is not part of the issue):

#!/bin/bash
#SBATCH -p research
#SBATCH -A research
#SBATCH -t 12:00:00
#SBATCH --ntasks=128
#SBATCH --mem=64G

module load adcirc/54.00/intel/19.1

export I_MPI_DEBUG=5

date > run.begin
#srun --mpi=pmi2 -n $SLURM_NTASKS padcirc -W 1
mpirun -np $SLURM_NTASKS padcirc -W 1
date > run.end

(2) The output from the hostname command  is:

Loading adcirc/54.00/intel/19.1
  Loading requirement: compilers/intel/19.1 mpi/impi/19.1/intel/19.1
    szip/2.1.1/intel/19.1 hdf5/1.8.18/intel/19.1/serial
    netcdf/4.5.0/intel/19.1/serial
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01
roux-rome-01

(3) The output from the run with verbose debugging is:

Loading adcirc/54.00/intel/19.1
  Loading requirement: compilers/intel/19.1 mpi/impi/19.1/intel/19.1
    szip/2.1.1/intel/19.1 hdf5/1.8.18/intel/19.1/serial
    netcdf/4.5.0/intel/19.1/serial
[0] MPI startup(): libfabric version: 1.9.0a1-impi
[0] MPI startup(): libfabric provider: tcp;ofi_rxm
[0] MPI startup(): Rank    Pid      Node name     Pin cpu
[0] MPI startup(): 0       7450     roux-rome-01  0
[0] MPI startup(): 1       7451     roux-rome-01  1
[0] MPI startup(): 2       7452     roux-rome-01  2
[0] MPI startup(): 3       7453     roux-rome-01  3
[0] MPI startup(): 4       7454     roux-rome-01  4
[0] MPI startup(): 5       7455     roux-rome-01  5
[0] MPI startup(): 6       7456     roux-rome-01  6
[0] MPI startup(): 7       7457     roux-rome-01  7
[0] MPI startup(): 8       7458     roux-rome-01  8
[0] MPI startup(): 9       7459     roux-rome-01  9
[0] MPI startup(): 10      7460     roux-rome-01  10
[0] MPI startup(): 11      7461     roux-rome-01  11
[0] MPI startup(): 12      7462     roux-rome-01  12
[0] MPI startup(): 13      7463     roux-rome-01  13
[0] MPI startup(): 14      7464     roux-rome-01  14
[0] MPI startup(): 15      7465     roux-rome-01  15
[0] MPI startup(): 16      7466     roux-rome-01  16
[0] MPI startup(): 17      7467     roux-rome-01  17
[0] MPI startup(): 18      7468     roux-rome-01  18
[0] MPI startup(): 19      7469     roux-rome-01  19
[0] MPI startup(): 20      7471     roux-rome-01  20
[0] MPI startup(): 21      7472     roux-rome-01  21
[0] MPI startup(): 22      7473     roux-rome-01  22
[0] MPI startup(): 23      7474     roux-rome-01  23
[0] MPI startup(): 24      7475     roux-rome-01  24
[0] MPI startup(): 25      7476     roux-rome-01  25
[0] MPI startup(): 26      7477     roux-rome-01  26
[0] MPI startup(): 27      7478     roux-rome-01  27
[0] MPI startup(): 28      7479     roux-rome-01  28
[0] MPI startup(): 29      7480     roux-rome-01  29
[0] MPI startup(): 30      7481     roux-rome-01  30
[0] MPI startup(): 31      7482     roux-rome-01  31
[0] MPI startup(): 32      7483     roux-rome-01  32
[0] MPI startup(): 33      7484     roux-rome-01  33
[0] MPI startup(): 34      7485     roux-rome-01  34
[0] MPI startup(): 35      7486     roux-rome-01  35
[0] MPI startup(): 36      7487     roux-rome-01  36
[0] MPI startup(): 37      7488     roux-rome-01  37
[0] MPI startup(): 38      7489     roux-rome-01  38
[0] MPI startup(): 39      7490     roux-rome-01  39
[0] MPI startup(): 40      7491     roux-rome-01  40
[0] MPI startup(): 41      7492     roux-rome-01  41
[0] MPI startup(): 42      7494     roux-rome-01  42
[0] MPI startup(): 43      7495     roux-rome-01  43
[0] MPI startup(): 44      7497     roux-rome-01  44
[0] MPI startup(): 45      7498     roux-rome-01  45
[0] MPI startup(): 46      7499     roux-rome-01  46
[0] MPI startup(): 47      7500     roux-rome-01  47
[0] MPI startup(): 48      7501     roux-rome-01  48
[0] MPI startup(): 49      7502     roux-rome-01  49
[0] MPI startup(): 50      7503     roux-rome-01  50
[0] MPI startup(): 51      7504     roux-rome-01  51
[0] MPI startup(): 52      7505     roux-rome-01  52
[0] MPI startup(): 53      7506     roux-rome-01  53
[0] MPI startup(): 54      7507     roux-rome-01  54
[0] MPI startup(): 55      7508     roux-rome-01  55
[0] MPI startup(): 56      7509     roux-rome-01  56
[0] MPI startup(): 57      7510     roux-rome-01  57
[0] MPI startup(): 58      7511     roux-rome-01  58
[0] MPI startup(): 59      7512     roux-rome-01  59
[0] MPI startup(): 60      7513     roux-rome-01  60
[0] MPI startup(): 61      7514     roux-rome-01  61
[0] MPI startup(): 62      7515     roux-rome-01  62
[0] MPI startup(): 63      7516     roux-rome-01  63
[0] MPI startup(): 64      7517     roux-rome-01  64
[0] MPI startup(): 65      7518     roux-rome-01  65
[0] MPI startup(): 66      7519     roux-rome-01  66
[0] MPI startup(): 67      7520     roux-rome-01  67
[0] MPI startup(): 68      7521     roux-rome-01  68
[0] MPI startup(): 69      7522     roux-rome-01  69
[0] MPI startup(): 70      7523     roux-rome-01  70
[0] MPI startup(): 71      7524     roux-rome-01  71
[0] MPI startup(): 72      7525     roux-rome-01  72
[0] MPI startup(): 73      7526     roux-rome-01  73
[0] MPI startup(): 74      7527     roux-rome-01  74
[0] MPI startup(): 75      7528     roux-rome-01  75
[0] MPI startup(): 76      7529     roux-rome-01  76
[0] MPI startup(): 77      7530     roux-rome-01  77
[0] MPI startup(): 78      7531     roux-rome-01  78
[0] MPI startup(): 79      7532     roux-rome-01  79
[0] MPI startup(): 80      7533     roux-rome-01  80
[0] MPI startup(): 81      7534     roux-rome-01  81
[0] MPI startup(): 82      7535     roux-rome-01  82
[0] MPI startup(): 83      7536     roux-rome-01  83
[0] MPI startup(): 84      7538     roux-rome-01  84
[0] MPI startup(): 85      7539     roux-rome-01  85
[0] MPI startup(): 86      7540     roux-rome-01  86
[0] MPI startup(): 87      7541     roux-rome-01  87
[0] MPI startup(): 88      7542     roux-rome-01  88
[0] MPI startup(): 89      7543     roux-rome-01  89
[0] MPI startup(): 90      7544     roux-rome-01  90
[0] MPI startup(): 91      7545     roux-rome-01  91
[0] MPI startup(): 92      7546     roux-rome-01  92
[0] MPI startup(): 93      7547     roux-rome-01  93
[0] MPI startup(): 94      7548     roux-rome-01  94
[0] MPI startup(): 95      7549     roux-rome-01  95
[0] MPI startup(): 96      7550     roux-rome-01  96
[0] MPI startup(): 97      7551     roux-rome-01  97
[0] MPI startup(): 98      7552     roux-rome-01  98
[0] MPI startup(): 99      7553     roux-rome-01  99
[0] MPI startup(): 100     7554     roux-rome-01  100
[0] MPI startup(): 101     7555     roux-rome-01  101
[0] MPI startup(): 102     7556     roux-rome-01  102
[0] MPI startup(): 103     7557     roux-rome-01  103
[0] MPI startup(): 104     7558     roux-rome-01  104
[0] MPI startup(): 105     7559     roux-rome-01  105
[0] MPI startup(): 106     7560     roux-rome-01  106
[0] MPI startup(): 107     7561     roux-rome-01  107
[0] MPI startup(): 108     7562     roux-rome-01  108
[0] MPI startup(): 109     7563     roux-rome-01  109
[0] MPI startup(): 110     7564     roux-rome-01  110
[0] MPI startup(): 111     7565     roux-rome-01  111
[0] MPI startup(): 112     7566     roux-rome-01  112
[0] MPI startup(): 113     7567     roux-rome-01  113
[0] MPI startup(): 114     7568     roux-rome-01  114
[0] MPI startup(): 115     7569     roux-rome-01  115
[0] MPI startup(): 116     7570     roux-rome-01  116
[0] MPI startup(): 117     7571     roux-rome-01  117
[0] MPI startup(): 118     7572     roux-rome-01  118
[0] MPI startup(): 119     7573     roux-rome-01  119
[0] MPI startup(): 120     7574     roux-rome-01  120
[0] MPI startup(): 121     7575     roux-rome-01  121
[0] MPI startup(): 122     7576     roux-rome-01  122
[0] MPI startup(): 123     7577     roux-rome-01  123
[0] MPI startup(): 124     7578     roux-rome-01  124
[0] MPI startup(): 125     7579     roux-rome-01  125
[0] MPI startup(): 126     7580     roux-rome-01  126
[0] MPI startup(): 127     7581     roux-rome-01  127
[0] MPI startup(): I_MPI_ROOT=/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_DEBUG=5
ECHO: Application launched with: padcirc -W
 INFO: Searching for ADCIRC subdomain directories:
 INFO: Looking for './PE0000/fort.14' ...
 INFO: File './PE0000/fort.14' was found!
 INFO: The search for the subdomain directory was completed successfully.
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f08459e11d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f0845169031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f08452e3c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f08451f55e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f08451c9d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f08452e66a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 2: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f4fd11c51d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f4fd094d031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f4fd0ac7c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f4fd09d95e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f4fd09add8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f4fd0aca6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 23: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f4eaa9b81d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f4eaa140031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f4eaa2bac5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f4eaa1cc5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f4eaa1a0d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f4eaa2bd6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 3: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7fa0bb4c81d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7fa0bac50031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7fa0badcac5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7fa0bacdc5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7fa0bacb0d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7fa0badcd6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 20: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f7fac68f1d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f7fabe17031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7fabf91c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f7fabea35e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f7fabe77d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7fabf946a7]
/apps/applications/development/compilers/intel/1
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Abort(1) on node 22: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f6f8e2161d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f6f8d99e031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f6f8db18c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f6f8da2a5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f6f8d9fed8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f6f8db1b6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 71: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f98266891d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f9825e11031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f9825f8bc5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f9825e9d5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f9825e71d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f9825f8e6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 70: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f579685d1d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f5795fe5031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f579615fc5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f57960715e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f5796045d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f57961626a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 76: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f38293241d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f3828aac031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f3828c26c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f3828b385e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f3828b0cd8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f3828c296a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 77: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f53aff911d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f53af719031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f53af893c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f53af7a55e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f53af779d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f53af8966a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 86: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f36ae8f11d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f36ae079031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f36ae1f3c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f36ae1055e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f36ae0d9d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f36ae1f66a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 68: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff79ed621d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff79e4ea031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff79e664c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff79e5765e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff79e54ad8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff79e6676a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 1: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7fc3d6f0a1d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7fc3d6692031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7fc3d680cc5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7fc3d671e5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7fc3d66f2d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7fc3d680f6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 84: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f244dc141d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f244d39c031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f244d516c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f244d4285e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f244d3fcd8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f244d5196a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 79: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f36308f51d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f363007d031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f36301f7c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f36301095e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f36300ddd8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f36301fa6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 87: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff7b2ba61d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff7b232e031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff7b24a8c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff7b23ba5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff7b238ed8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff7b24ab6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 85: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f41a45a81d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f41a3d30031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f41a3eaac5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f41a3dbc5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f41a3d90d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f41a3ead6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 74: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f7a5dc331d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f7a5d3bb031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7a5d535c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f7a5d4475e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f7a5d41bd8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7a5d5386a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 72: Internal error
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f78081c61d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f780794e031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7807ac8c5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f78079da5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f78079aed8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7807acb6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 73: Internal error
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1)
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f24fe79a1d4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f24fdf22031]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f24fe09cc5d]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f24fdfae5e4]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f24fdf82d8e]
/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f24fe09f6a7]
/apps/applications/development/compilers/intel/1
Abort(1) on node 21: Internal error

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 7450 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 4 PID 7454 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 5 PID 7455 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 6 PID 7456 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 7 PID 7457 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 8 PID 7458 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 9 PID 7459 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 10 PID 7460 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 11 PID 7461 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 12 PID 7462 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 13 PID 7463 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 14 PID 7464 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 15 PID 7465 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 16 PID 7466 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 17 PID 7467 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 18 PID 7468 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 19 PID 7469 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 21 PID 7472 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 24 PID 7475 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 25 PID 7476 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 26 PID 7477 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 27 PID 7478 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 28 PID 7479 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 29 PID 7480 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 30 PID 7481 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 31 PID 7482 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 32 PID 7483 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 33 PID 7484 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 34 PID 7485 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 35 PID 7486 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 36 PID 7487 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 37 PID 7488 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 38 PID 7489 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 39 PID 7490 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 40 PID 7491 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 41 PID 7492 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 42 PID 7494 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 43 PID 7495 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 44 PID 7497 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 45 PID 7498 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 46 PID 7499 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 47 PID 7500 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 48 PID 7501 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 49 PID 7502 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 50 PID 7503 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 51 PID 7504 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 52 PID 7505 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 53 PID 7506 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 54 PID 7507 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 55 PID 7508 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 56 PID 7509 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 57 PID 7510 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 58 PID 7511 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 59 PID 7512 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 60 PID 7513 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 61 PID 7514 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 62 PID 7515 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 63 PID 7516 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 64 PID 7517 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 65 PID 7518 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 66 PID 7519 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 67 PID 7520 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 69 PID 7522 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 75 PID 7528 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 78 PID 7531 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 80 PID 7533 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 81 PID 7534 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 82 PID 7535 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 83 PID 7536 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 88 PID 7542 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 89 PID 7543 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 90 PID 7544 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 91 PID 7545 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 92 PID 7546 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 93 PID 7547 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 94 PID 7548 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 95 PID 7549 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 96 PID 7550 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 97 PID 7551 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 98 PID 7552 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 99 PID 7553 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 100 PID 7554 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 101 PID 7555 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 102 PID 7556 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 103 PID 7557 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 104 PID 7558 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 105 PID 7559 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 106 PID 7560 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 107 PID 7561 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 108 PID 7562 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 109 PID 7563 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 110 PID 7564 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 111 PID 7565 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 112 PID 7566 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 113 PID 7567 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 114 PID 7568 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 115 PID 7569 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 116 PID 7570 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 117 PID 7571 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 118 PID 7572 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 119 PID 7573 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 120 PID 7574 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 121 PID 7575 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 122 PID 7576 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 123 PID 7577 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 124 PID 7578 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 125 PID 7579 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 126 PID 7580 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 127 PID 7581 RUNNING AT roux-rome-01
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

Thanks for your help

 

Zach

0 Kudos
Copy link
PrasanthD_intel
Moderator
‎04-27-2020 04:30 AM
2,876 Views

Hi Zach,

We are forwarding this issue to the concerned engineering team.

We will get back to you soon.

 

Regards

Prasanth

0 Kudos
Copy link
Jennifer_D_Intel
Moderator
‎10-14-2020 08:13 AM
2,764 Views

We have released an update to the oneAPI beta HPC toolkit with better MPI support. Can you try the latest update and see if the problem is resolved?


0 Kudos
Copy link
Jennifer_D_Intel
Moderator
‎02-25-2021 03:11 PM
2,574 Views

Any luck trying the more recent version of IMPI? OneAPI has been officially released and IMPI is part of the HPC toolkit.


0 Kudos
Copy link
GouthamK_Intel
Moderator
‎06-18-2021 01:37 AM
2,320 Views

Hi,

As we have not heard back from you, we are considering that your issue has been resolved and we have answered all your queries. So we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only 

Have a Good day!


Thanks & Regards

Goutham


0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.