numberOfProcessor=20 , hostfile=hosts_64Node , imblistpath=eximb.txt , repeattime=1 20 hosts_64Node eximb.txt --------ClusterNode List-------- prod-0001 prod-0003 prod-0004 prod-0005 prod-0006 prod-0007 prod-0008 prod-0009 prod-0010 prod-0011 prod-0012 prod-0013 prod-0014 prod-0015 prod-0016 prod-0017 prod-0018 prod-0019 prod-0020 prod-0021 prod-0022 prod-0023 prod-0024 prod-0025 prod-0026 prod-0027 prod-0028 prod-0029 prod-0030 prod-0031 prod-0032 prod-0033 prod-0034 prod-0035 prod-0036 prod-0037 prod-0038 prod-0039 prod-0040 prod-0041 prod-0042 prod-0043 prod-0044 prod-0045 prod-0046 prod-0047 prod-0048 prod-0049 prod-0050 prod-0051 prod-0052 prod-0053 prod-0054 prod-0055 prod-0056 prod-0057 prod-0058 prod-0059 prod-0060 prod-0061 prod-0062 prod-0063 prod-0064 prod-0065 -------------------------------- 1280 test 1 , 1280 Processes , 64 asgc nodes Round at 1 of IMB-MPI1= Allgatherv MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f24e4e81161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f24e4df0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f249ee9617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f24e4df19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f501904a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5018fb9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4fd305f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5018fba9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa342faa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa342f19fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa2fcfbf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa342f1a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f909a2dd161] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f681cae0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f681ca4ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f67d6af517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f681ca509c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f90f2105161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f90f2074fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f90ac11a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f90f20759c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f909a24cfff] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f90542f217d] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f909a24d9c0] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7c247e8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7c24757fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7bde7fd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7c247589c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3be5f22161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3be5e91fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3b9ff3717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3be5e929c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8af8187161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8af80f6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8ab219c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8af80f79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1ca193d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1ca18acfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1c5b95217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1ca18ad9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f19db96a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f19db8d9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f199597f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f19db8da9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd7c6e33161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd7c6da2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd780e4817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd7c6da39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== ==== backtrace ==== ==== backtrace ==== 8 IMB-MPI1(main+0x4b7) [0x404667] ==== backtrace ==== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9fd32a3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9fd3212fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9f8d2b817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9fd32139c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc1916f1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc191660fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc14b70617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc1916619c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6b9daba161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6b9da29fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6b57acf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6b9da2a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== MXM: Got signal 11 (Segmentation fault) ==== backtrace ==== ==== backtrace ==== 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f502ecb2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f502ec21fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4fe8cc717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f502ec229c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5cee615161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5cee584fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5ca862a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5cee5859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6138b1a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6138a89fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f60f2b2f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6138a8a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6ed9742161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6ed96b1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6e9375717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6ed96b29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faf76b2d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faf76a9cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faf30b4217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faf76a9d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faf900da161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faf90049fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faf4a0ef17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faf9004a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7b58dbe161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7b58d2dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7b12dd317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7b58d2e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7499f02161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7499e71fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7453f1717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7499e729c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f05271eb161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f052715afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f04e120017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f052715b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc3a1f65161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc3a1ed4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc35bf7a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc3a1ed59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f83784dc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f837844bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f83324f117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f837844c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9126bd6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9126b45fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f90e0beb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9126b469c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc92f49c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc92f40bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc8e94b117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc92f40c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6da87f3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6da8762fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6d6280817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6da87639c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3ba62cd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3ba623cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3b602e217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3ba623d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fea42e63161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fea42dd2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe9fce7817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fea42dd39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f00f75cc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f00f753bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f00b15e117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f00f753c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1be841b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1be838afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1ba243017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1be838b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6bfc9b9161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6bfc928fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6bb69ce17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6bfc9299c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff847f49161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff847eb8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff801f5e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff847eb99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fca18762161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fca186d1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc9d277717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fca186d29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== ==== backtrace ==== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb74eb8c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb74eafbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb708ba117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb74eafc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff60afc7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff60af36fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff5c4fdc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff60af379c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0aa249c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0aa240bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0a5c4b117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0aa240c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2ebf9e3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2ebf952fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2e799f817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2ebf9539c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc1834a3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc183412fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc13d4b817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc1834139c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1c0c2f0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1c0c25ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1bc630517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1c0c2609c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7abd907161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7abd876fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7a7791c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7abd8779c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f433cccf161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f433cc3efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f42f6ce417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f433cc3f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f66cb3e4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f66cb353fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f66853f917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f66cb3549c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8a974e8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8a97457fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8a514fd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8a974589c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3c3774f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3c376befff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3bf176417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3c376bf9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7b300e3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7b30052fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7aea0f817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7b300539c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff8be802161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff8be771fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff87881717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff8be7729c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7febf4784161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7febf46f3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7febae79917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7febf46f49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb90be77161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb90bde6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb8c5e8c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb90bde79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f25bba56161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f25bb9c5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2575a6b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f25bb9c69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3e99c3d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3e99bacfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3e53c5217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3e99bad9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f68103c3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6810332fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f67ca3d817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f68103339c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0fda4da161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0fda449fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0f944ef17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0fda44a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8935a0d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f893597cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f88efa2217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f893597d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc0fa8fd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc0fa86cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc0b491217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc0fa86d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f189fa0f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f189f97efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1859a2417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f189f97f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa7f3db1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa7f3d20fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa7addc617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa7f3d219c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 10 IMB-MPI1() [0x4040e9] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f175ef54161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f175eec3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1718f6917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f175eec49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f27a401f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f27a3f8efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f275e03417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f27a3f8f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f90cecfd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f90cec6cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9088d1217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f90cec6d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb18d2aa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb18d219fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb1472bf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb18d21a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe81b4df161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe81b44efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe7d54f417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe81b44f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6f3ae3e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6f3adadfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6ef4e5317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6f3adae9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f553ba03161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f553b972fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f54f5a1817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f553b9739c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd4ead6e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd4eacddfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd4a4d8317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd4eacde9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd65a754161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd65a6c3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd61476917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd65a6c49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fdcfa1a2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fdcfa111fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdcb41b717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fdcfa1129c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f150b8d4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f150b843fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f14c58e917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f150b8449c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff974563161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff9744d2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff92e57817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff9744d39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f65a928d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f65a91fcfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f65632a217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f65a91fd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9258884161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f92587f3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f921289917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f92587f49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f51e06c7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f51e0636fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f519a6dc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f51e06379c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faf4650c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faf4647bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faf0052117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faf4647c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fda01394161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fda01303fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd9bb3a917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fda013049c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0f8d33b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0f8d2aafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0f4735017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0f8d2ab9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3226657161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f32265c6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f31e066c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f32265c79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd601cde161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd601c4dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd5bbcf317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd601c4e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f16c15b0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f16c151ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f167b5c517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f16c15209c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0d98caa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0d98c19fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0d52cbf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0d98c1a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe54c942161] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f14915a7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1491516fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f144b5bc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f14915179c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc669e6c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc669ddbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc623e8117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc669ddc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f15e3596161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f15e3505fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f159d5ab17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f15e35069c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb41dd8f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb41dcfefff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb3d7da417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb41dcff9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f726e834161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f726e7a3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f722884917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f726e7a49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f911f33b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f911f2aafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f90d935017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f911f2ab9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f930f758161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f930f6c7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f92c976d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f930f6c89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f681eb69161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f681ead8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f67d8b7e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f681ead99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f96bf09a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f96bf009fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f96790af17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f96bf00a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0bbafa6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0bbaf15fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0b74fbb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0bbaf169c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fdb6b369161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fdb6b2d8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdb2537e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fdb6b2d99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f194d32b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f194d29afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f190734017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f194d29b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd57ab15161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd57aa84fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd534b2a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd57aa859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcc02c4a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcc02bb9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcbbcc5f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcc02bba9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0115a30161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f011599ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f00cfa4517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f01159a09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f968f304161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f968f273fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f964931917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f968f2749c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc5bab8d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc5baafcfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc574ba217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc5baafd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb2dd4ab161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb2dd41afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb2974c017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb2dd41b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc964cbe161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc964c2dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc91ecd317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc964c2e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f791077b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f79106eafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f78ca79017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f79106eb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fab737ef161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fab7375efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fab2d80417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fab7375f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8089b80161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8089aeffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8043b9517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8089af09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb72ad2b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb72ac9afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb6e4d4017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb72ac9b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fabf0764161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fabf06d3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fabaa77917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fabf06d49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f94f9ae2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f94f9a51fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f94b3af717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f94f9a529c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2b3fedc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2b3fe4bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2af9ef117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2b3fe4c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0877b0d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0877a7cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0831b2217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0877a7d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5549531161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f55494a0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f550354617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f55494a19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe54c8b1fff] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3021618161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3021587fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2fdb62d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f30215889c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f367c4c3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f367c432fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f36364d817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f367c4339c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f97e9327161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f97e9296fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f97a333c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f97e92979c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f837fb93161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f837fb02fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8339ba817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f837fb039c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f01f15e6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f01f1555fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f01ab5fb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f01f15569c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2c6b834161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2c6b7a3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2c2584917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2c6b7a49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc6062e0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc60624ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc5c02f517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc6062509c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f145f4a0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f145f40ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f14194b517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f145f4109c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f61da9f7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f61da966fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6194a0c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f61da9679c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1f690c4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1f69033fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1f230d917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1f690349c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd2fc640161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd2fc5affff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd2b665517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd2fc5b09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7642adc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7642a4bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f75fcaf117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7642a4c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5a66647161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5a665b6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5a2065c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5a665b79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f97e3479161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f97e33e8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f979d48e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f97e33e99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbefb808161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbefb777fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbeb581d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbefb7789c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3e3b65b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3e3b5cafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3df567017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3e3b5cb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7614bcc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7614b3bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f75cebe117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7614b3c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6e1a734161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6e1a6a3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6dd474917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6e1a6a49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4a230f2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4a23061fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f49dd10717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4a230629c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc067a78161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc0679e7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc021a8d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc0679e89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fda15424161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fda15393fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd9cf43917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fda153949c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7feb7c9eb161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7feb7c95afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7feb36a0017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7feb7c95b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd1bfa53161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd1bf9c2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd179a6817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd1bf9c39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe50695717d] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa9e3a48161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa9e39b7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa99da5d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa9e39b89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa2f861a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa2f8589fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa2b262f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa2f858a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbabe25d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbabe1ccfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fba7827217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbabe1cd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faa030cf161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faa0303efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa9bd0e417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faa0303f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6666508161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6666477fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f662051d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f66664789c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f83b882a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f83b8799fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f837283f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f83b879a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0f83c1e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0f83b8dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0f3dc3317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0f83b8e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faa7718a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faa770f9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faa3119f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faa770fa9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1dc265b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1dc25cafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1d7c67017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1dc25cb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f20251d1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2025140fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1fdf1e617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f20251419c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8efe400161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8efe36ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8eb841517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8efe3709c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff17356c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff1734dbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff12d58117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff1734dc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8db23b5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8db2324fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8d6c3ca17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8db23259c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc44a4ad161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc44a41cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc4044c217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc44a41d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f241c8f5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f241c864fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f23d690a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f241c8659c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa145a2d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa14599cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa0ffa4217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa14599d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe5aa087161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe5a9ff6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe56409c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe5a9ff79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4edf0b1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4edf020fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4e990c617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4edf0219c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0f7fa38161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0f7f9a7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0f39a4d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0f7f9a89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcd9ab3a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcd9aaa9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcd54b4f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcd9aaaa9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3f49629161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3f49598fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3f0363e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3f495999c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7f12ad8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7f12a47fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7eccaed17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7f12a489c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1183269161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f11831d8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f113d27e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f11831d99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbea7f89161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbea7ef8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbe61f9e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbea7ef99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe54c8b29c0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f74abb08161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f74aba77fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7465b1d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f74aba789c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f97e07d9161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f97e0748fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f979a7ee17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f97e07499c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff69019d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff69010cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff64a1b217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff69010d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcd1baaa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcd1ba19fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fccd5abf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcd1ba1a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f28677b4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2867723fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f28217c917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f28677249c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc9372b3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc937222fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc8f12c817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc9372239c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f151529d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f151520cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f14cf2b217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f151520d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f58d3a94161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f58d3a03fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f588daa917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f58d3a049c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7e50a85161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7e509f4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7e0aa9a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7e509f59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f57746fa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5774669fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f572e70f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f577466a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f01ab579161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f01ab4e8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f016558e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f01ab4e99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7090ad7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7090a46fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f704aaec17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7090a479c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fae29340161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fae292affff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fade335517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fae292b09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe3de639161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe3de5a8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe39864e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe3de5a99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1a81418161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1a81387fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1a3b42d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1a813889c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa387c89161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa387bf8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa341c9e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa387bf99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc774586161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc7744f5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc72e59b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc7744f69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f187857d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f18784ecfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f183259217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f18784ed9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f179ac1c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f179ab8bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1754c3117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f179ab8c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f43bea19161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f43be988fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4378a2e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f43be9899c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffecea7b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffece9eafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffe88a9017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffece9eb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb366a22161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb366991fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb320a3717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb3669929c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb3df47f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb3df3eefff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb39949417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb3df3ef9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbec9d27161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbec9c96fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbe83d3c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbec9c979c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffab0bac161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffab0b1bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffa6abc117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffab0b1c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe3ce069161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe3cdfd8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe38807e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe3cdfd99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f13ae981161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f13ae8f0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f136899617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f13ae8f19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe666846161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe6667b5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe62085b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe6667b69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f93325c0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f933252ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f92ec5d517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f93325309c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fda2cb48161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fda2cab7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd9e6b5d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fda2cab89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fefa608b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fefa5ffafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fef600a017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fefa5ffb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe833b37161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe833aa6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe7edb4c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe833aa79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fea7b03c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fea7afabfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fea3505117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fea7afac9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0f0e2d6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0f0e245fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0ec82eb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0f0e2469c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f94acef0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f94ace5ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9466f0517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f94ace609c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9ad94c5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9ad9434fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9a934da17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9ad94359c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff75dd4c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff75dcbbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff717d6117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff75dcbc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f404b39e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f404b30dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f40053b317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f404b30e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6bae415161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6bae384fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6b6842a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6bae3859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe63ee81161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe63edf0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe5f8e9617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe63edf19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f85e3d0d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f85e3c7cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f859dd2217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f85e3c7d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f36a0591161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f36a0500fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f365a5a617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f36a05019c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f261ee7b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f261edeafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f25d8e9017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f261edeb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f44353fc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f443536bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f43ef41117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f443536c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f441e080161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f441dfeffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f43d809517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f441dff09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb2c0bed161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb2c0b5cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb27ac0217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb2c0b5d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd8d5e8d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd8d5dfcfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd88fea217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd8d5dfd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6b7b7ef161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6b7b75efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6b3580417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6b7b75f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6dfb633161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6dfb5a2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6db564817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6dfb5a39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc11b4f4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc11b463fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc0d550917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc11b4649c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f613a4b8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f613a427fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f60f44cd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f613a4289c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe506cf4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe506c63fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe4c0d0917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe506c649c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3496ead161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3496e1cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3450ec217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3496e1d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f732c66c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f732c5dbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f72e668117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f732c5dc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f96c0a8e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f96c09fdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f967aaa317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f96c09fe9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6cdd302161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6cdd271fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6c9731717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6cdd2729c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f814b4ae161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f814b41dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f81054c317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f814b41e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbf50bfd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbf50b6cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbf0ac1217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbf50b6d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f33712f5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3371264fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f332b30a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f33712659c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6b7d73c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6b7d6abfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6b3775117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6b7d6ac9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0ed6717161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0ed6686fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0e9072c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0ed66879c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7feb3d040161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7feb3cfaffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7feaf705517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7feb3cfb09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7bb9633161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7bb95a2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7b7364817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7bb95a39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8278638161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f82785a7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f823264d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f82785a89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f25c5522161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f25c5491fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f257f53717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f25c54929c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f34576bc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f345762bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f34116d117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f345762c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbf88150161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbf880bffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbf4216517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbf880c09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7761e10161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7761d7ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f771be2517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7761d809c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffdb676b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffdb66dafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffd7078017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffdb66db9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4554940161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f45548affff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f450e95517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f45548b09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f84f3ba0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f84f3b0ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f84adbb517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f84f3b109c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2f0288e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2f027fdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2ebc8a317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2f027fe9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fef662ae161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fef6621dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fef202c317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fef6621e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1da3896161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1da3805fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1d5d8ab17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1da38069c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3e3f8db161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3e3f84afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3df98f017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3e3f84b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faafeae0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faafea4ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faab8af517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faafea509c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd2a6494161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd2a6403fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd2604a917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd2a64049c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9a063ec161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9a0635bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f99c040117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9a0635c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff1a2dea161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff1a2d59fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff15cdff17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff1a2d5a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8cbc23f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8cbc1aefff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8c7625417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8cbc1af9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbe50147161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbe500b6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbe0a15c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbe500b79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9e3d9d0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9e3d93ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9df79e517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9e3d9409c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f641802e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6417f9dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f63d204317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6417f9e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f980d1f0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f980d15ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f97c720517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f980d1609c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f32c0f1d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f32c0e8cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f327af3217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f32c0e8d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4d708fd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4d7086cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4d2a91217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4d7086d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fea0b066161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fea0afd5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe9c507b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fea0afd69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd1390c8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd139037fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd0f30dd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd1390389c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f33cdd19161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f33cdc88fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3387d2e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f33cdc899c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f50401f8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5040167fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4ffa20d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f50401689c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa9c1a27161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa9c1996fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa97ba3c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa9c19979c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fde20f64161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fde20ed3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdddaf7917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fde20ed49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f555940c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f555937bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f551342117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f555937c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4a775c0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4a7752ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4a315d517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4a775309c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe4b6afd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe4b6a6cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe470b1217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe4b6a6d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f839aaec161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f839aa5bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8354b0117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f839aa5c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa843bc0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa843b2ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa7fdbd517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa843b309c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa3ff421161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa3ff390fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa3b943617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa3ff3919c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2d06f56161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2d06ec5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2cc0f6b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2d06ec69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f68193e6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6819355fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f67d33fb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f68193569c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f077e594161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f077e503fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f07385a917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f077e5049c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f85507c8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8550737fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f850a7dd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f85507389c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd64a4d2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd64a441fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd6044e717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd64a4429c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc1f43e6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc1f4355fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc1ae3fb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc1f43569c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8b9251f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8b9248efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8b4c53417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8b9248f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f08919a7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0891916fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f084b9bc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f08919179c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa4db27c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa4db1ebfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa49529117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa4db1ec9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f22f92f6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f22f9265fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f22b330b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f22f92669c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f20dbab9161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f20dba28fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2095ace17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f20dba299c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4f667c6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4f66735fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4f207db17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4f667369c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f57d0314161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f57d0283fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f578a32917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f57d02849c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f95e8a6c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f95e89dbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f95a2a8117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f95e89dc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9e77b74161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9e77ae3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9e31b8917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9e77ae49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5ae70dd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5ae704cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5aa10f217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5ae704d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f588d885161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f588d7f4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f584789a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f588d7f59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1bba1ad161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1bba11cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1b741c217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1bba11d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f082cbd0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f082cb3ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f07e6be517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f082cb409c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6810a54161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f68109c3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f67caa6917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f68109c49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f150a85b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f150a7cafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f14c487017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f150a7cb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f98c43f1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f98c4360fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f987e40617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f98c43619c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6d307a8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6d30717fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6cea7bd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6d307189c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f68bc546161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f68bc4b5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f687655b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f68bc4b69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6779ab1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6779a20fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6733ac617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6779a219c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7f0fee7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7f0fe56fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7ec9efc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7f0fe579c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7d7e5d4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7d7e543fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7d385e917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7d7e5449c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9ca59b7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9ca5926fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9c5f9cc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9ca59279c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4845434161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f48453a3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f47ff44917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f48453a49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa079643161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa0795b2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa03365817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa0795b39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f431c0cf161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f431c03efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f42d60e417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f431c03f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f06778ad161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f067781cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f06318c217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f067781d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fac3bbbd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fac3bb2cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fabf5bd217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fac3bb2d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1ca4831161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1ca47a0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1c5e84617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1ca47a19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4f9d085161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4f9cff4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4f5709a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4f9cff59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f546644d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f54663bcfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f542046217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f54663bd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9963604161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9963573fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f991d61917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f99635749c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f206f729161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f206f698fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f202973e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f206f6999c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f50d68cd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f50d683cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f50908e217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f50d683d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff12c29e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff12c20dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff0e62b317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff12c20e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb87aa27161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb87a996fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb834a3c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb87a9979c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd7191ef161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd71915efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd6d320417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd71915f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f913d1ec161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f913d15bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f90f720117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f913d15c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa37532b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa37529afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa32f34017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa37529b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8f7fb32161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8f7faa1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8f39b4717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8f7faa29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9232c43161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9232bb2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f91ecc5817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9232bb39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb68ce66161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb68cdd5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb646e7b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb68cdd69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f89f3a3c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f89f39abfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f89ada5117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f89f39ac9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fedea1c0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fedea12ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7feda41d517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fedea1309c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc121658161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc1215c7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc0db66d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc1215c89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f56810ab161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f568101afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f563b0c017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f568101b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9530daf161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9530d1efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f94eadc417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9530d1f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe5c74a7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe5c7416fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe5814bc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe5c74179c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f573c457161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f573c3c6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f56f646c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f573c3c79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2e80b75161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2e80ae4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f2e3ab8a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f2e80ae59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fae5ab95161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fae5ab04fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fae14baa17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fae5ab059c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fccd6b6a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fccd6ad9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcc90b7f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fccd6ada9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f82c3805161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f82c3774fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f827d81a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f82c37759c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f02ff5be161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f02ff52dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f02b95d317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f02ff52e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fdc28ad7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fdc28a46fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdbe2aec17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fdc28a479c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9b3c565161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9b3c4d4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9af657a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9b3c4d59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9e11fda161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9e11f49fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9dcbfef17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9e11f4a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8e73288161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8e731f7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8e2d29d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8e731f89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f26ab704161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f26ab673fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f266571917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f26ab6749c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa527acd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa527a3cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa4e1ae217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa527a3d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3f8dc9d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3f8dc0cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3f47cb217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3f8dc0d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5c75647161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5c755b6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5c2f65c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5c755b79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f52ef3a0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f52ef30ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f52a93b517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f52ef3109c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc7e8eb2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc7e8e21fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc7a2ec717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc7e8e229c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5aaa91c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5aaa88bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5a6493117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5aaa88c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc8e8456161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc8e83c5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc8a246b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc8e83c69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f824ca5c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f824c9cbfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8206a7117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f824c9cc9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0e2777a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0e276e9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0de178f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0e276ea9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fefc670b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fefc667afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fef8072017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fefc667b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f057b5ef161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f057b55efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f053560417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f057b55f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3fc53f7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3fc5366fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3f7f40c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3fc53679c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f20939c4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f2093933fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f204d9d917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f20939349c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f180f375161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f180f2e4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f17c938a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f180f2e59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f710407d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7103fecfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f70be09217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7103fed9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0c27c65161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0c27bd4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0be1c7a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0c27bd59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fda5b4cd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fda5b43cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fda154e217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fda5b43d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5823e8e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5823dfdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f57ddea317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5823dfe9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7a927d8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7a92747fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7a4c7ed17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7a927489c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f463d900161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f463d86ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f45f791517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f463d8709c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9e9509b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9e9500afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9e4f0b017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9e9500b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fdd098f4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fdd09863fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdcc390917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fdd098649c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1f2f5ac161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1f2f51bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1ee95c117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1f2f51c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7b4d72d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7b4d69cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7b0774217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7b4d69d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc7aeff8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc7aef67fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc76900d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc7aef689c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f12b4b40161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f12b4aaffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f126eb5517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f12b4ab09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe43fd20161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe43fc8ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe3f9d3517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe43fc909c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe02e3bc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe02e32bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdfe83d117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe02e32c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f28e4f53161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f28e4ec2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f289ef6817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f28e4ec39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4786465161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f47863d4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f474047a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f47863d59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa598d9d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa598d0cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa552db217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa598d0d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc1ea75e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc1ea6cdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc1a477317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc1ea6ce9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f47e4b9f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f47e4b0efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f479ebb417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f47e4b0f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fae8e49c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fae8e40bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fae484b117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fae8e40c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd089b15161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd089a84fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd043b2a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd089a859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5162348161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f51622b7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f511c35d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f51622b89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f654f47a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f654f3e9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f650948f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f654f3ea9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f86c980a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f86c9779fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f868381f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f86c977a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fdfaf3c8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fdfaf337fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fdf693dd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fdfaf3389c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc99d6ac161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc99d61bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc9576c117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc99d61c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f26d925f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f26d91cefff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f269327417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f26d91cf9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3b17ef3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3b17e62fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3ad1f0817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3b17e639c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4bf7919161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4bf7888fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4bb192e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4bf78899c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc3e6635161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc3e65a4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc3a064a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc3e65a59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8b7aba2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8b7ab11fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8b34bb717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8b7ab129c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4cb59a4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4cb5913fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4c6f9b917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4cb59149c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7eff07a0a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7eff07979fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7efec1a1f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7eff0797a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff86cdc5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff86cd34fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff826dda17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff86cd359c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f81dea23161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f81de992fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8198a3817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f81de9939c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe423d2f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe423c9efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe3ddd4417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe423c9f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7c05d14161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7c05c83fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7bbfd2917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7c05c849c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f53aebb3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f53aeb22fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5368bc817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f53aeb239c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcd676ff161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcd6766efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcd2171417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcd6766f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb430735161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb4306a4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb3ea74a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb4306a59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffcbe3aa161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffcbe319fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffc783bf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffcbe31a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7faad54c4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7faad5433fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7faa8f4d917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7faad54349c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5d1062b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5d1059afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5cca64017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5d1059b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5fd70c1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5fd7030fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5f910d617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5fd70319c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6da5268161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6da51d7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6d5f27d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6da51d89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe4be68e161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe4be5fdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe4786a317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe4be5fe9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9aaee3d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9aaedacfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9a68e5217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9aaedad9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1b990b6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1b99025fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1b530cb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1b990269c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5b0c425161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5b0c394fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5ac643a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5b0c3959c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0c863ea161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0c86359fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0c403ff17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0c8635a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff0b7bf2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff0b7b61fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff071c0717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff0b7b629c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff5b4452161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff5b43c1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff56e46717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff5b43c29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f041737c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f04172ebfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f03d139117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f04172ec9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f030a7f8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f030a767fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f02c480d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f030a7689c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd3d8523161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd3d8492fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd39253817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd3d84939c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f46161da161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4616149fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f45d01ef17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f461614a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f84708f2161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8470861fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f842a90717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f84708629c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f21da301161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f21da270fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f219431617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f21da2719c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9cc7382161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9cc72f1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9c8139717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9cc72f29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8aea3ee161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8aea35dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8aa440317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8aea35e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbe578eb161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbe5785afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbe1190017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbe5785b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f42a2d12161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f42a2c81fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f425cd2717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f42a2c829c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7ba2365161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7ba22d4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7b5c37a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7ba22d59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f937c0c0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f937c02ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f93360d517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f937c0309c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9c59364161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9c592d3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f9c1337917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9c592d49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f470fb90161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f470fafffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f46c9ba517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f470fb009c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fce24dfc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fce24d6bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcddee1117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fce24d6c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd3efb15161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd3efa84fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd3a9b2a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd3efa859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3ae85ba161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3ae8529fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3aa25cf17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3ae852a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f37c83de161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f37c834dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f37823f317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f37c834e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd927c94161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd927c03fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd8e1ca917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd927c049c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5c7cc78161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5c7cbe7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5c36c8d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5c7cbe89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcb6d013161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcb6cf82fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcb2702817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcb6cf839c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f05d1fb7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f05d1f26fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f058bfcc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f05d1f279c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb51bdd8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb51bd47fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb4d5ded17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb51bd489c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7e74afe161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7e74a6dfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7e2eb1317d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7e74a6e9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffb7410c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffb7407bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffb2e12117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffb7407c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5f742fc161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5f7426bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5f2e31117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5f7426c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f633832a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6338299fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f62f233f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f633829a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd33a685161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd33a5f4fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd2f469a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd33a5f59c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6fadb48161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6fadab7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6f67b5d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6fadab89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffb70316161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffb70285fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffb2a32b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffb702869c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f74c119d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f74c110cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f747b1b217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f74c110d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f66370f4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6637063fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f65f110917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f66370649c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5c76bb0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5c76b1ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5c30bc517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5c76b209c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f46cda82161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f46cd9f1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4687a9717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f46cd9f29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6a84777161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6a846e6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6a3e78c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6a846e79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1216ab0161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1216a1ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f11d0ac517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1216a209c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f88b8b41161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f88b8ab0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8872b5617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f88b8ab19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd76043d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd7603acfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd71a45217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd7603ad9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f736726a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f73671d9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f732127f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f73671da9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6a2a5cb161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6a2a53afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f69e45e017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6a2a53b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9a43089161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9a42ff8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f99fd09e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9a42ff99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f97da698161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f97da607fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f97946ad17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f97da6089c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe616e20161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe616d8ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe5d0e3517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe616d909c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f32b89b8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f32b8927fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f32729cd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f32b89289c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fe4b3717161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fe4b3686fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fe46d72c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fe4b36879c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc00879d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc00870cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbfc27b217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc00870d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9634fbd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9634f2cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f95eefd217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f9634f2d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1aa3731161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1aa36a0fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1a5d74617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1aa36a19c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f543ec1b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f543eb8afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f53f8c3017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f543eb8b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3d7444f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3d743befff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3d2e46417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3d743bf9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f46b24c4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f46b2433fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f466c4d917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f46b24349c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb7c31dd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb7c314cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb77d1f217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb7c314d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8c7d2b8161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8c7d227fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8c372cd17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8c7d2289c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1b8b09f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1b8b00efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1b450b417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1b8b00f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fcca452b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fcca449afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcc5e54017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fcca449b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f305caf4161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f305ca63fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f3016b0917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f305ca649c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f00365e5161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0036554fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7efff05fa17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f00365559c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f240cdb1161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f240cd20fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f23c6dc617d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f240cd219c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc4a7849161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc4a77b8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc46185e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc4a77b99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1569632161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f15695a1fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f152364717d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f15695a29c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff3d6f36161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff3d6ea5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff390f4b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff3d6ea69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f78bfa03161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f78bf972fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7879a1817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f78bf9739c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff6281f3161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff628162fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff5e220817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff6281639c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fb10ea7b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fb10e9eafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fb0c8a9017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fb10e9eb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f97c998b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f97c98fafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f97839a017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f97c98fb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7015733161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f70156a2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6fcf74817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f70156a39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f9397615161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f9397584fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f935162a17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f93975859c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa580397161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa580306fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa53a3ac17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa5803079c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fac0290a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fac02879fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fabbc91f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fac0287a9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f057ae98161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f057ae07fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0534ead17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f057ae089c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1eb4519161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1eb4488fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1e6e52e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1eb44899c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f970a98f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f970a8fefff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f96c49a417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f970a8ff9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f59e9f80161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f59e9eeffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f59a3f9517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f59e9ef09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8d3a94b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8d3a8bafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8cf496017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8d3a8bb9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa3432f7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa343266fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa2fd30c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa3432679c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8f7d3a6161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8f7d315fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8f373bb17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8f7d3169c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f7eedb49161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f7eedab8fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f7ea7b5e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f7eedab99c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f0cb89d9161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f0cb8948fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f0c729ee17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f0cb89499c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5635046161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5634fb5fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f55ef05b17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5634fb69c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5b17ca7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5b17c16fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5ad1cbc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5b17c179c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f813ed1d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f813ec8cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f80f8d3217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f813ec8d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1cafa30161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1caf99ffff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1c69a4517d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1caf9a09c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f760e0bd161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f760e02cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f75c80d217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f760e02d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1e7b69b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1e7b60afff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1e356b017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1e7b60b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff7f6e03161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff7f6d72fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff7b0e1817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff7f6d739c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1647688161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f16475f7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f160169d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f16475f89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fce44903161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fce44872fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcdfe91817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fce448739c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] ==== backtrace ==== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffd827c7161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffd82736fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffd3c7dc17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffd827379c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f952790f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f952787efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f94e192417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f952787f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f73cd313161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f73cd282fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f738732817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f73cd2839c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1acb69f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1acb60efff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1a856b417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1acb60f9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8b18467161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8b183d6fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8ad247c17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8b183d79c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f37a1f58161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f37a1ec7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f375bf6d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f37a1ec89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ff6b6e3b161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ff6b6daafff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ff670e5017d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ff6b6dab9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1b56a4f161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1b569befff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1b10a6417d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1b569bf9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fc16e283161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fc16e1f2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fc12829817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fc16e1f39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f5f21f7a161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f5f21ee9fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f5edbf8f17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f5f21eea9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fec64938161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fec648a7fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fec1e94d17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fec648a89c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa34872c161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa34869bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa30274117d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa34869c9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6851b8d161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6851afcfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f680bba217d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6851afd9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f4c05053161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f4c04fc2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f4bbf06817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f4c04fc39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 0 /lib64/libc.so.6() [0x38fbc329a0] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f36a9878161] 0 /lib64/libc.so.6() [0x38fbc329a0] 0 /lib64/libc.so.6() [0x38fbc329a0] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 10 IMB-MPI1() [0x4040e9] =================== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f396a098161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f854fd29161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fec866e6161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fa26005f161] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fbfb26c8161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fac558ea161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8797506161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f3339d8a161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8f3459c161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7ffc5c307161] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6b5eec3161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f05a4096161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1ee9d20161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fce024cb161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7feffad4e161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fad2bdd2161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f8815b6c161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f6f82b43161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f36a97e7fff] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f2142862161] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7fd0873c0161] 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1061b33161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1061aa2fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f101bb4817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1061aa39c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] =================== 1 /lib64/libc.so.6(memcpy+0x15b) [0x38fbc89aab] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1bcc349161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f396a007fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f854fc98fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fec86655fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fa25ffcefff] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f642ad54161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fbfb2637fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fac55859fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8797475fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f3339cf9fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f21427d1fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fad2bd41fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7ffc5c276fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6b5ee32fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f6f82ab2fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f05a4005fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1ee9c8ffff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fce0243afff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8f3450bfff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f8815adbfff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7fd08732ffff] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f1701bed161] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7feffacbdfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f366388d17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f39240ad17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8509d3e17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fec406fb17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fa21a07417d] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f642acc3fff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fbf6c6dd17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fac0f8ff17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f875151b17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f32f3d9f17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f8eee5b117d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7ffc1631c17d] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1bcc2b8fff] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f1701b5cfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6b18ed817d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f055e0ab17d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1ea3d3517d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fcdbc4e017d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fefb4d6317d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7face5de717d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f87cfb8117d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f6f3cb5817d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f36a97e89c0] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f20fc87717d] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7fd0413d517d] 2 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(+0xfd161) [0x7f74aa41c161] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7feffacbe9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1ee9c909c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fad2bd429c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fd0873309c0] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f16bbc0217d] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f21427d29c0] 3 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(ompi_datatype_sndrcv+0x52f) [0x7f74aa38bfff] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f1b8635e17d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f396a0089c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f854fc999c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fec866569c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8815adc9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fa25ffcf9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fbfb26389c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fac5585a9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f87974769c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f3339cfa9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f8f3450c9c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7ffc5c2779c0] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f63e4d6917d] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6b5ee339c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f6f82ab39c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f05a40069c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7fce0243b9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f642acc49c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1bcc2b99c0] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f1701b5d9c0] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 4 /usr/mpi/gcc/openmpi-1.6.5/lib64/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgatherv_intra_neighborexchange+0x8d) [0x7f746443117d] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 5 /usr/mpi/gcc/openmpi-1.6.5/lib64/libmpi.so.1(PMPI_Allgatherv+0x80) [0x7f74aa38c9c0] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 6 IMB-MPI1(IMB_allgatherv+0x122) [0x40e062] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 8 IMB-MPI1(main+0x4b7) [0x404667] 8 IMB-MPI1(main+0x4b7) [0x404667] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 7 IMB-MPI1(IMB_init_buffers_iter+0x588) [0x408648] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] =================== 10 IMB-MPI1() [0x4040e9] 10 IMB-MPI1() [0x4040e9] 8 IMB-MPI1(main+0x4b7) [0x404667] =================== =================== =================== =================== 10 IMB-MPI1() [0x4040e9] =================== =================== =================== =================== =================== 10 IMB-MPI1() [0x4040e9] =================== =================== =================== =================== =================== =================== =================== =================== =================== 10 IMB-MPI1() [0x4040e9] =================== =================== 9 /lib64/libc.so.6(__libc_start_main+0xfd) [0x38fbc1ed1d] =================== =================== 10 IMB-MPI1() [0x4040e9] =================== =================== MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) -------------------------------------------------------------------------- mpirun noticed that process rank 1021 with PID 14100 on node prod-0053 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) MXM: Got signal 15 (Terminated) Allgatherv is pass Execution Time : 357590 milliseconds