- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello.
I recently installed Intel oneAPI HPC Toolkit in my Linux Mint box.
(Intel(R) MPI Library, Version 2021.13 Build 20240701 (id: 179630a))
Sourced the environmental variables like:
source /opt/intel/oneapi/setvars.shAnd then compiled the GULP program.
Run an example using:
mpirun -n 2 gulp example1and got the errors:
Abort(2139023) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(192)........:
MPID_Init(1665)..............:
MPIDI_OFI_mpi_init_hook(1625):
open_fabric(2726)............:
find_provider(2904)..........: OFI fi_getinfo() failed (ofi_init.c:2904:find_provider:No data available)
Abort(2139023) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(192)........:
MPID_Init(1665)..............:
MPIDI_OFI_mpi_init_hook(1625):
open_fabric(2726)............:
find_provider(2904)..........: OFI fi_getinfo() failed (ofi_init.c:2904:find_provider:No data available)Then run as:
mpirun -np 2 gulp example1and the error was:
Abort(2139023) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(192)........:
MPID_Init(1665)..............:
MPIDI_OFI_mpi_init_hook(1625):
open_fabric(2726)............:
find_provider(2904)..........: OFI fi_getinfo() failed (ofi_init.c:2904:find_provider:No data available)
The command mpirun --version returned:
[mpiexec@VARADERO-LIN] match_arg (../../../../../src/pm/i_hydra/libhydra/arg/hydra_arg.c:82): unrecognized argument version
[mpiexec@VARADERO-LIN] Similar arguments:
[mpiexec@VARADERO-LIN] version
[mpiexec@VARADERO-LIN] HYD_arg_parse_array (../../../../../src/pm/i_hydra/libhydra/arg/hydra_arg.c:106): argument matching returned error
[mpiexec@VARADERO-LIN] mpiexec_get_parameters (../../../../../src/pm/i_hydra/mpiexec/mpiexec_params.c:1190): error parsing input array
[mpiexec@VARADERO-LIN] main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:1725): error parsing parametersand mpirun -info returned:
HYDRA build details:
Version: 2021.13
Release Date: 20240701 (id: 179630a)
Process Manager: pmi
Bootstrap servers available: ssh slurm rsh ll sge pbs pbsdsh pdsh srun lsf blaunch qrsh fork
Resource management kernels available: slurm ll lsf sge pbs cobaltAll the calculations are run in my notebook:
OS: Linux Mint 22 x86_64
Host: Dell G15 5530
Kernel: 6.8.0-40-generic
Uptime: 2 hours, 21 mins
Packages: 3115 (dpkg), 7 (flatpak)
Shell: bash 5.2.21
Resolution: 1920x1080
DE: Cinnamon 6.2.9
WM: Mutter (Muffin)
WM Theme: Mint-L-Dark (Mint-Y)
Theme: Mint-L-Dark [GTK2/3]
Icons: Mint-L [GTK2/3]
Terminal: gnome-terminal
CPU: 13th Gen Intel i7-13650HX (20) @ 4.700GHz
GPU: Intel Raptor Lake-S UHD Graphics
GPU: NVIDIA GeForce RTX 4050 Max-Q / Mobile
Memory: 11651MiB / 31778MiB lscpu returned:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 20
On-line CPU(s) list: 0-19
Vendor ID: GenuineIntel
Model name: 13th Gen Intel(R) Core(TM) i7-13650HX
CPU family: 6
Model: 183
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 1
Stepping: 1
CPU(s) scaling MHz: 28%
CPU max MHz: 4900.0000
CPU min MHz: 800.0000
BogoMIPS: 5606.40
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 544 KiB (14 instances)
L1i cache: 704 KiB (14 instances)
L2 cache: 11.5 MiB (8 instances)
L3 cache: 24 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-19
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Mitigation; Clear Register File
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affectedand fi_info -l returned:
psm2:
version: 120.10
mlx:
version: 1.4
psm3:
version: 706.0
verbs:
version: 120.10
verbs:
version: 120.10
ofi_rxm:
version: 120.10
tcp:
version: 120.10
shm:
version: 120.10
ofi_hook_noop:
version: 120.10
off_coll:
version: 120.10
Any help is highly appreciated.
Link Copied
8 Replies
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please provide the output of
I_MPI_DEBUG=10 I_MPI_HYDRA_DEBUG=1 mpirun -np 2 IMB-MPI1?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here it is:
[mpiexec@VARADERO-LIN] Launch arguments: /opt/intel/oneapi/mpi/2021.13/bin//hydra_bstrap_proxy --upstream-host VARADERO-LIN --upstream-port 41533 --pgid 0 --launcher ssh --launcher-number 0 --base-path /opt/intel/oneapi/mpi/2021.13/bin/ --topolib hwloc --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /opt/intel/oneapi/mpi/2021.13/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@VARADERO-LIN] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_maxes
[proxy:0:0@VARADERO-LIN] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_appnum
[proxy:0:0@VARADERO-LIN] PMI response: cmd=appnum appnum=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_my_kvsname
[proxy:0:0@VARADERO-LIN] PMI response: cmd=my_kvsname kvsname=kvs_26985_0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@VARADERO-LIN] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_maxes
[proxy:0:0@VARADERO-LIN] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_appnum
[proxy:0:0@VARADERO-LIN] PMI response: cmd=appnum appnum=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@VARADERO-LIN] PMI response: cmd=my_kvsname kvsname=kvs_26985_0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get kvsname=kvs_26985_0 key=PMI_process_mapping
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get kvsname=kvs_26985_0 key=PMI_process_mapping
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[0] MPI startup(): Intel(R) MPI Library, Version 2021.13 Build 20240701 (id: 179630a)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=barrier_in
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=put kvsname=kvs_26985_0 key=-bcast-1-0 value=2F6465762F73686D2F496E74656C5F4D50495F716D514C4B66
[proxy:0:0@VARADERO-LIN] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@VARADERO-LIN] PMI response: cmd=barrier_out
[proxy:0:0@VARADERO-LIN] PMI response: cmd=barrier_out
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get kvsname=kvs_26985_0 key=-bcast-1-0
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F716D514C4B66
[0] MPI startup(): libfabric loaded: libfabric.so.1
[0] MPI startup(): libfabric version: 1.20.1-impi
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=abort exitcode=2139023
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=abort exitcode=2139023
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @TobiasK .
I did a test compiling/running a simple hello_word example, and the errors running it are the same, so, I am afraid this is not a GULP problem.
My hello_world code:
PROGRAM hello_world_mpi
include 'mpif.h'
integer process_Rank, size_Of_Cluster, ierror, tag
call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size_Of_Cluster, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, process_Rank, ierror)
print *, 'Hello World from process: ', process_Rank, 'of ', size_Of_Cluster
call MPI_FINALIZE(ierror)
END PROGRAM
Compile it with
mpiifx hello_world_mpi.f90 -o hello_world_mpi.exe
and got:
Abort(2139535) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(193)........:
MPID_Init(1715)..............:
MPIDI_OFI_mpi_init_hook(1633):
open_fabric(2739)............:
find_provider(2924)..........: OFI fi_getinfo() failed (ofi_init.c:2924:find_provider:No data available)
Abort(2139535) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(193)........:
MPID_Init(1715)..............:
MPIDI_OFI_mpi_init_hook(1633):
open_fabric(2739)............:
find_provider(2924)..........: OFI fi_getinfo() failed (ofi_init.c:2924:find_provider:No data available)- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please try with:
I_MPI_FABRICS=shm mpirun ...- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
icamps@VARADERO-LIN:Examples$ I_MPI_FABRICS=shm mpirun -np 2 gulp example1
Abort(134828037) on node 1 (rank 1 in comm 0): Fatal error in PMPI_Comm_rank: Unknown error class, error stack:
PMPI_Comm_rank(106): MPI_Comm_rank(comm=0x0, rank=0x7ffe819c927c) failed
PMPI_Comm_rank(63).: Invalid communicator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Camps I am afraid I cannont help with GULP:
GULP is available for free for academic use by anyone with a valid University email account. To access the code please go to the appropriate tab and "register". The code is provided "as is" and without support. Any questions regarding the use of the code can be posted at the user forum if not already covered in previous topics. Bug reports and registration queries can be sent to gulpcode at curtin.edu.au, and we will try to respond as soon as is possible.
you may ask in the forum mentioned.
Please provide the output of:
I_MPI_FABRICS=shm I_MPI_DEBUG=10 I_MPI_HYDRA_DEBUG=1 mpirun -np 2 IMB-MPI1
if that works you need to ask someone with access to GULP
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @TobiasK
Below is the output.
The first thing I did was to make a post on the GULP forum. Unfortunately, Julian (the GULP developer) cannot help with this issue.
[mpiexec@VARADERO-LIN] Launch arguments: /opt/intel/oneapi/mpi/2021.14/bin//hydra_bstrap_proxy --upstream-host VARADERO-LIN --upstream-port 35289 --pgid 0 --launcher ssh --launcher-number 0 --base-path /opt/intel/oneapi/mpi/2021.14/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /opt/intel/oneapi/mpi/2021.14/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@VARADERO-LIN] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@VARADERO-LIN] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_maxes
[proxy:0:0@VARADERO-LIN] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_maxes
[proxy:0:0@VARADERO-LIN] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_appnum
[proxy:0:0@VARADERO-LIN] PMI response: cmd=appnum appnum=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_appnum
[proxy:0:0@VARADERO-LIN] PMI response: cmd=appnum appnum=0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@VARADERO-LIN] PMI response: cmd=my_kvsname kvsname=kvs_8807_0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get_my_kvsname
[proxy:0:0@VARADERO-LIN] PMI response: cmd=my_kvsname kvsname=kvs_8807_0
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=get kvsname=kvs_8807_0 key=PMI_process_mapping
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get kvsname=kvs_8807_0 key=PMI_process_mapping
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[0] MPI startup(): Intel(R) MPI Library, Version 2021.14 Build 20240911 (id: b3fc682)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=barrier_in
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=put kvsname=kvs_8807_0 key=-bcast-1-0 value=2F6465762F73686D2F496E74656C5F4D50495F353952764A43
[proxy:0:0@VARADERO-LIN] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@VARADERO-LIN] PMI response: cmd=barrier_out
[proxy:0:0@VARADERO-LIN] PMI response: cmd=barrier_out
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=get kvsname=kvs_8807_0 key=-bcast-1-0
[proxy:0:0@VARADERO-LIN] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F353952764A43
[0] MPI startup(): shm segment size (1068 MB per rank) * (2 local ranks) = 2136 MB total
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.14/opt/mpi/etc/tuning_skx_shm.dat"
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: async_progress coll split: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): tag bits available: 30 (TAG_UB value: 1073741823)
[0] MPI startup(): source bits available: 0 (Maximal number of rank: 0)
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 8812 VARADERO-LIN {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}
[0] MPI startup(): 1 8813 VARADERO-LIN {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}
[0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.14
[0] MPI startup(): ONEAPI_ROOT=/opt/intel/oneapi
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_DEBUG=1
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=-1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_FABRICS=shm
[0] MPI startup(): I_MPI_DEBUG=10
#----------------------------------------------------------------
# Intel(R) MPI Benchmarks 2021.8, MPI-1 part
#----------------------------------------------------------------
# Date : Tue Nov 5 09:55:55 2024
# Machine : x86_64
# System : Linux
# Release : 6.8.0-40-generic
# Version : #40-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 10:34:03 UTC 2024
# MPI Version : 3.1
# MPI Thread Environment:
# Calling sequence was:
# IMB-MPI1
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# PingPong
# PingPing
# Sendrecv
# Exchange
# Allreduce
# Reduce
# Reduce_local
# Reduce_scatter
# Reduce_scatter_block
# Allgather
# Allgatherv
# Gather
# Gatherv
# Scatter
# Scatterv
# Alltoall
# Alltoallv
# Bcast
# Barrier
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.36 0.00
1 1000 0.37 2.71
2 1000 0.32 6.34
4 1000 0.27 14.77
8 1000 0.25 32.51
16 1000 0.22 74.25
32 1000 0.19 165.05
64 1000 0.18 352.45
128 1000 0.17 772.28
256 1000 0.16 1607.62
512 1000 0.22 2310.16
1024 1000 0.29 3553.83
2048 1000 0.38 5404.71
4096 1000 0.52 7807.74
8192 1000 0.89 9169.67
16384 1000 1.56 10532.23
32768 1000 2.57 12743.93
65536 640 4.04 16238.29
131072 320 6.77 19374.62
262144 160 9.43 27805.87
524288 80 20.63 25419.79
1048576 40 49.24 21297.16
2097152 20 95.06 22062.18
4194304 10 181.56 23101.08
#---------------------------------------------------
# Benchmarking PingPing
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 0.17 0.00
1 1000 0.18 5.59
2 1000 0.16 12.36
4 1000 0.16 25.02
8 1000 0.16 49.88
16 1000 0.16 97.19
32 1000 0.16 198.68
64 1000 0.17 386.28
128 1000 0.18 718.09
256 1000 0.19 1335.45
512 1000 0.26 1938.36
1024 1000 0.32 3242.76
2048 1000 0.40 5122.62
4096 1000 0.54 7550.11
8192 1000 0.93 8831.94
16384 1000 1.59 10329.53
32768 1000 2.18 14998.42
65536 640 4.10 16003.13
131072 320 6.54 20036.65
262144 160 12.67 20684.85
524288 80 36.43 14391.15
1048576 40 83.19 12604.03
2097152 20 170.84 12275.77
4194304 10 308.29 13604.87
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 0.15 0.15 0.15 0.00
1 1000 0.16 0.16 0.16 12.70
2 1000 0.16 0.16 0.16 24.67
4 1000 0.16 0.16 0.16 48.53
8 1000 0.16 0.16 0.16 100.61
16 1000 0.16 0.16 0.16 200.93
32 1000 0.16 0.16 0.16 402.49
64 1000 0.17 0.17 0.17 759.61
128 1000 0.17 0.17 0.17 1487.36
256 1000 0.19 0.19 0.19 2726.17
512 1000 0.24 0.24 0.24 4261.52
1024 1000 0.30 0.30 0.30 6921.19
2048 1000 0.38 0.38 0.38 10853.89
4096 1000 0.55 0.55 0.55 14950.89
8192 1000 0.91 0.91 0.91 18094.06
16384 1000 1.59 1.59 1.59 20591.82
32768 1000 2.15 2.15 2.15 30497.38
65536 640 4.03 4.04 4.03 32483.56
131072 320 6.67 6.68 6.67 39261.98
262144 160 12.58 12.58 12.58 41684.46
524288 80 38.13 38.37 38.25 27328.95
1048576 40 82.88 82.88 82.88 25303.14
2097152 20 192.53 192.62 192.58 21774.88
4194304 10 564.02 564.05 564.04 14872.08
#-----------------------------------------------------------------------------
# Benchmarking Exchange
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 0.46 0.46 0.46 0.00
1 1000 0.45 0.45 0.45 8.85
2 1000 0.47 0.47 0.47 16.85
4 1000 0.49 0.49 0.49 32.91
8 1000 0.49 0.49 0.49 65.82
16 1000 0.49 0.49 0.49 131.73
32 1000 0.48 0.48 0.48 265.34
64 1000 0.51 0.51 0.51 501.87
128 1000 0.52 0.52 0.52 983.99
256 1000 0.52 0.52 0.52 1955.01
512 1000 0.70 0.70 0.70 2908.69
1024 1000 0.91 0.91 0.91 4523.90
2048 1000 1.11 1.11 1.11 7405.37
4096 1000 1.61 1.61 1.61 10148.14
8192 1000 2.22 2.22 2.22 14742.76
16384 1000 3.58 3.58 3.58 18329.77
32768 1000 4.55 4.55 4.55 28799.56
65536 640 8.14 8.14 8.14 32213.36
131072 320 13.22 13.22 13.22 39656.49
262144 160 29.84 29.87 29.85 35110.22
524288 80 80.84 80.97 80.90 25900.15
1048576 40 169.03 169.04 169.03 24812.95
2097152 20 346.37 346.47 346.42 24211.36
4194304 10 648.29 648.81 648.55 25858.29
#----------------------------------------------------------------
# Benchmarking Allreduce
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 0.23 0.24 0.23
8 1000 0.24 0.24 0.24
16 1000 0.23 0.23 0.23
32 1000 0.23 0.24 0.23
64 1000 0.27 0.33 0.30
128 1000 0.26 0.33 0.30
256 1000 0.28 0.34 0.31
512 1000 0.33 0.40 0.37
1024 1000 0.38 0.46 0.42
2048 1000 0.49 0.57 0.53
4096 1000 0.71 0.79 0.75
8192 1000 1.17 1.25 1.21
16384 1000 2.22 2.26 2.24
32768 1000 3.73 3.78 3.75
65536 640 5.09 5.11 5.10
131072 320 9.48 9.49 9.49
262144 160 15.70 15.78 15.74
524288 80 35.77 36.50 36.14
1048576 40 89.30 89.88 89.59
2097152 20 205.96 205.99 205.97
4194304 10 569.25 576.56 572.91
#----------------------------------------------------------------
# Benchmarking Reduce
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 0.08 0.24 0.16
8 1000 0.08 0.21 0.14
16 1000 0.12 0.26 0.19
32 1000 0.08 0.18 0.13
64 1000 0.12 0.27 0.19
128 1000 0.13 0.27 0.20
256 1000 0.14 0.32 0.23
512 1000 0.08 0.33 0.21
1024 1000 0.09 0.41 0.25
2048 1000 0.09 0.53 0.31
4096 1000 0.18 0.76 0.47
8192 1000 0.58 1.18 0.88
16384 1000 1.15 2.49 1.82
32768 1000 2.94 4.25 3.59
65536 640 2.88 6.14 4.51
131072 320 3.57 8.54 6.06
262144 160 13.28 16.72 15.00
524288 80 13.20 31.36 22.28
1048576 40 33.55 72.56 53.05
2097152 20 81.28 143.82 112.55
4194304 10 167.45 286.28 226.87
#----------------------------------------------------------------
# Benchmarking Reduce_local
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 0.02 0.02 0.02
8 1000 0.02 0.02 0.02
16 1000 0.02 0.02 0.02
32 1000 0.02 0.02 0.02
64 1000 0.02 0.02 0.02
128 1000 0.02 0.02 0.02
256 1000 0.02 0.02 0.02
512 1000 0.02 0.02 0.02
1024 1000 0.03 0.04 0.03
2048 1000 0.05 0.06 0.06
4096 1000 0.07 0.10 0.09
8192 1000 0.13 0.15 0.14
16384 1000 0.23 0.25 0.24
32768 1000 0.61 0.62 0.62
65536 640 1.20 1.21 1.20
131072 320 2.35 2.36 2.36
262144 160 4.88 4.88 4.88
524288 80 9.55 9.57 9.56
1048576 40 39.26 39.59 39.43
2097152 20 81.46 81.51 81.49
4194304 10 162.40 162.58 162.49
#----------------------------------------------------------------
# Benchmarking Reduce_scatter
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.06 0.08 0.07
4 1000 0.30 0.36 0.33
8 1000 0.29 0.36 0.33
16 1000 0.31 0.38 0.34
32 1000 0.32 0.38 0.35
64 1000 0.32 0.38 0.35
128 1000 0.34 0.40 0.37
256 1000 0.36 0.41 0.39
512 1000 0.41 0.47 0.44
1024 1000 0.46 0.53 0.50
2048 1000 0.57 0.65 0.61
4096 1000 0.85 0.92 0.88
8192 1000 1.57 1.65 1.61
16384 1000 2.86 2.97 2.92
32768 1000 4.38 4.47 4.42
65536 640 17.29 18.41 17.85
131072 320 10.99 11.92 11.46
262144 160 26.46 26.93 26.70
524288 80 71.03 73.72 72.37
1048576 40 163.52 169.76 166.64
2097152 20 342.67 347.95 345.31
4194304 10 815.82 837.96 826.89
#----------------------------------------------------------------
# Benchmarking Reduce_scatter_block
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 0.28 0.34 0.31
8 1000 0.28 0.34 0.31
16 1000 0.28 0.34 0.31
32 1000 0.28 0.34 0.31
64 1000 0.29 0.35 0.32
128 1000 0.30 0.36 0.33
256 1000 0.32 0.38 0.35
512 1000 0.42 0.47 0.44
1024 1000 0.44 0.51 0.47
2048 1000 0.54 0.62 0.58
4096 1000 0.82 0.89 0.85
8192 1000 1.60 1.66 1.63
16384 1000 3.00 3.08 3.04
32768 1000 4.31 4.47 4.39
65536 640 8.21 8.60 8.40
131072 320 15.10 15.14 15.12
262144 160 26.11 26.12 26.11
524288 80 71.25 71.47 71.36
1048576 40 166.80 170.55 168.68
2097152 20 335.26 341.19 338.23
4194304 10 790.99 791.86 791.42
#----------------------------------------------------------------
# Benchmarking Allgather
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 0.22 0.29 0.26
2 1000 0.22 0.28 0.25
4 1000 0.23 0.30 0.27
8 1000 0.21 0.28 0.25
16 1000 0.22 0.28 0.25
32 1000 0.22 0.28 0.25
64 1000 0.23 0.28 0.26
128 1000 0.23 0.29 0.26
256 1000 0.25 0.32 0.28
512 1000 0.30 0.38 0.34
1024 1000 0.37 0.40 0.39
2048 1000 0.45 0.49 0.47
4096 1000 0.64 0.68 0.66
8192 1000 1.35 1.42 1.38
16384 1000 1.96 2.01 1.98
32768 1000 2.75 2.83 2.79
65536 640 5.09 5.09 5.09
131072 320 8.37 8.43 8.40
262144 160 17.31 17.46 17.39
524288 80 55.58 55.81 55.70
1048576 40 124.00 124.18 124.09
2097152 20 266.23 266.41 266.32
4194304 10 480.44 480.53 480.49
#----------------------------------------------------------------
# Benchmarking Allgatherv
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.05 0.06 0.06
1 1000 0.27 0.33 0.30
2 1000 0.26 0.33 0.29
4 1000 0.26 0.32 0.29
8 1000 0.26 0.33 0.29
16 1000 0.27 0.33 0.30
32 1000 0.27 0.33 0.30
64 1000 0.28 0.34 0.31
128 1000 0.31 0.37 0.34
256 1000 0.31 0.37 0.34
512 1000 0.37 0.43 0.40
1024 1000 0.41 0.48 0.44
2048 1000 0.51 0.58 0.54
4096 1000 0.80 0.85 0.83
8192 1000 1.41 1.48 1.45
16384 1000 2.46 2.55 2.50
32768 1000 3.82 3.84 3.83
65536 640 7.06 7.10 7.08
131072 320 12.49 12.50 12.49
262144 160 17.99 18.16 18.08
524288 80 55.38 55.62 55.50
1048576 40 123.08 123.44 123.26
2097152 20 247.70 247.80 247.75
4194304 10 475.62 475.70 475.66
#----------------------------------------------------------------
# Benchmarking Gather
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 0.12 0.26 0.19
2 1000 0.12 0.28 0.20
4 1000 0.12 0.29 0.21
8 1000 0.12 0.26 0.19
16 1000 0.12 0.29 0.21
32 1000 0.12 0.26 0.19
64 1000 0.12 0.27 0.20
128 1000 0.13 0.27 0.20
256 1000 0.13 0.29 0.21
512 1000 0.20 0.42 0.31
1024 1000 0.21 0.50 0.35
2048 1000 0.28 0.65 0.47
4096 1000 0.35 0.90 0.63
8192 1000 0.49 1.55 1.02
16384 1000 0.79 2.06 1.42
32768 1000 1.31 3.10 2.20
65536 640 2.66 4.79 3.73
131072 320 3.03 7.21 5.12
262144 160 5.90 13.10 9.50
524288 80 11.94 41.18 26.56
1048576 40 32.61 82.53 57.57
2097152 20 80.98 163.79 122.38
4194304 10 244.19 281.78 262.99
#----------------------------------------------------------------
# Benchmarking Gatherv
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.16 0.17 0.17
1 1000 0.38 0.42 0.40
2 1000 0.38 0.43 0.40
4 1000 0.38 0.43 0.40
8 1000 0.39 0.44 0.41
16 1000 0.38 0.43 0.40
32 1000 0.39 0.44 0.41
64 1000 0.40 0.44 0.42
128 1000 0.39 0.44 0.42
256 1000 0.40 0.44 0.42
512 1000 0.55 0.58 0.57
1024 1000 0.63 0.67 0.65
2048 1000 0.75 0.78 0.77
4096 1000 0.96 1.00 0.98
8192 1000 1.38 1.40 1.39
16384 1000 2.04 2.05 2.05
32768 1000 3.06 3.09 3.08
65536 640 4.58 4.60 4.59
131072 320 7.38 7.40 7.39
262144 160 13.47 13.49 13.48
524288 80 40.88 40.97 40.93
1048576 40 83.08 83.13 83.10
2097152 20 164.98 165.03 165.01
4194304 10 282.31 282.47 282.39
#----------------------------------------------------------------
# Benchmarking Scatter
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 0.12 0.15 0.13
2 1000 0.12 0.14 0.13
4 1000 0.12 0.14 0.13
8 1000 0.12 0.15 0.14
16 1000 0.12 0.14 0.13
32 1000 0.12 0.15 0.14
64 1000 0.12 0.16 0.14
128 1000 0.12 0.16 0.14
256 1000 0.12 0.16 0.14
512 1000 0.22 0.29 0.26
1024 1000 0.23 0.34 0.28
2048 1000 0.28 0.45 0.36
4096 1000 0.35 0.68 0.51
8192 1000 0.51 1.16 0.84
16384 1000 0.97 1.51 1.24
32768 1000 1.75 3.46 2.60
65536 640 3.55 5.44 4.50
131072 320 7.12 9.18 8.15
262144 160 11.42 14.45 12.94
524288 80 38.30 40.21 39.25
1048576 40 85.24 87.36 86.30
2097152 20 169.80 172.92 171.36
4194304 10 324.24 326.30 325.27
#----------------------------------------------------------------
# Benchmarking Scatterv
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.21 0.27 0.24
1 1000 0.22 0.28 0.25
2 1000 0.22 0.26 0.24
4 1000 0.22 0.27 0.24
8 1000 0.23 0.28 0.26
16 1000 0.23 0.27 0.25
32 1000 0.23 0.27 0.25
64 1000 0.23 0.27 0.25
128 1000 0.23 0.28 0.25
256 1000 0.24 0.29 0.27
512 1000 0.33 0.36 0.35
1024 1000 0.33 0.41 0.37
2048 1000 0.39 0.55 0.47
4096 1000 0.47 0.77 0.62
8192 1000 0.66 1.22 0.94
16384 1000 1.16 1.63 1.39
32768 1000 2.02 2.48 2.25
65536 640 3.92 3.98 3.95
131072 320 6.69 7.46 7.07
262144 160 9.73 11.67 10.70
524288 80 30.72 43.10 36.91
1048576 40 49.33 85.53 67.43
2097152 20 93.93 170.73 132.33
4194304 10 176.21 326.69 251.45
#----------------------------------------------------------------
# Benchmarking Alltoall
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 0.41 0.42 0.42
2 1000 0.42 0.43 0.42
4 1000 0.41 0.42 0.41
8 1000 0.41 0.42 0.41
16 1000 0.42 0.43 0.42
32 1000 0.41 0.42 0.41
64 1000 0.44 0.45 0.44
128 1000 0.45 0.48 0.46
256 1000 0.45 0.47 0.46
512 1000 0.52 0.58 0.55
1024 1000 0.57 0.62 0.59
2048 1000 0.67 0.72 0.70
4096 1000 0.84 0.89 0.87
8192 1000 1.28 1.32 1.30
16384 1000 1.65 1.66 1.66
32768 1000 2.88 2.90 2.89
65536 640 5.39 5.40 5.39
131072 320 8.65 8.98 8.82
262144 160 20.96 20.98 20.97
524288 80 56.93 57.13 57.03
1048576 40 123.72 123.85 123.79
2097152 20 258.11 260.91 259.51
4194304 10 535.03 535.15 535.09
#----------------------------------------------------------------
# Benchmarking Alltoallv
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.40 0.41 0.41
1 1000 0.83 0.84 0.83
2 1000 0.82 0.83 0.82
4 1000 0.83 0.83 0.83
8 1000 0.84 0.85 0.84
16 1000 0.81 0.82 0.81
32 1000 0.83 0.84 0.84
64 1000 0.84 0.84 0.84
128 1000 0.85 0.85 0.85
256 1000 0.88 0.88 0.88
512 1000 0.90 0.96 0.93
1024 1000 0.96 1.03 0.99
2048 1000 1.17 1.24 1.20
4096 1000 1.21 1.28 1.25
8192 1000 1.69 1.74 1.71
16384 1000 2.05 2.07 2.06
32768 1000 3.26 3.29 3.28
65536 640 5.58 5.61 5.59
131072 320 8.92 8.93 8.92
262144 160 21.56 21.65 21.61
524288 80 59.73 90.06 74.89
1048576 40 131.82 195.50 163.66
2097152 20 259.41 261.86 260.63
4194304 10 535.77 535.85 535.81
#----------------------------------------------------------------
# Benchmarking Bcast
# #processes = 2
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.01 0.01 0.01
1 1000 0.10 0.11 0.11
2 1000 0.11 0.11 0.11
4 1000 0.11 0.11 0.11
8 1000 0.11 0.11 0.11
16 1000 0.12 0.12 0.12
32 1000 0.11 0.11 0.11
64 1000 0.11 0.13 0.12
128 1000 0.11 0.13 0.12
256 1000 0.11 0.13 0.12
512 1000 0.11 0.15 0.13
1024 1000 0.12 0.25 0.18
2048 1000 0.12 0.33 0.23
4096 1000 0.18 0.52 0.35
8192 1000 0.27 0.90 0.58
16384 1000 0.42 1.66 1.04
32768 1000 0.79 2.68 1.73
65536 640 1.71 3.81 2.76
131072 320 3.30 6.40 4.85
262144 160 6.92 10.33 8.62
524288 80 14.23 18.66 16.45
1048576 40 33.27 42.50 37.88
2097152 20 89.77 93.12 91.44
4194304 10 176.36 179.56 177.96
#---------------------------------------------------
# Benchmarking Barrier
# #processes = 2
#---------------------------------------------------
#repetitions t_min[usec] t_max[usec] t_avg[usec]
1000 0.18 0.18 0.18
# All processes entering MPI_Finalize
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 6: cmd=finalize
[proxy:0:0@VARADERO-LIN] PMI response: cmd=finalize_ack
[proxy:0:0@VARADERO-LIN] pmi cmd from fd 9: cmd=finalize
[proxy:0:0@VARADERO-LIN] PMI response: cmd=finalize_ack
Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page