FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

harshal05 — Mon, 03 Jul 2023 08:03:25 GMT

Dear Team,

I am getting errors while running the FVCOM simulation model as I am using more than 1048 processes. But when I am using less than 1048 processes it is running successfully.

I am using a Cray XC40 machine, where I have used Intel 2017 compiler and Cray-MPICH v7.6.3 implementation (which contains MPI_TAG_UB=1681915906) .

1053 27 14 3 15
1054 37 16 4 21
! ELEMENT COMM TEST : PASSED
Rank 862 [Fri Jun 30 15:20:52 2023] [c5-0c0s8n2] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x210e4e0, count=8, MPI_INTEGER, dest=1050, tag=2102863, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2102863
Rank 861 [Fri Jun 30 15:20:52 2023] [c5-0c0s8n2] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x210a300, count=5, MPI_INTEGER, dest=1050, tag=2102862, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2102862
Rank 902 [Fri Jun 30 15:20:52 2023] [c5-0c0s9n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x2103240, count=7, MPI_INTEGER, dest=1051, tag=2104903, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2104903
Rank 901 [Fri Jun 30 15:20:52 2023] [c5-0c0s9n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x210e640, count=7, MPI_INTEGER, dest=1051, tag=2104902, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2104902
Rank 899 [Fri Jun 30 15:20:52 2023] [c5-0c0s9n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x2108440, count=1, MPI_INTEGER, dest=1053, tag=2108900, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2108900
Rank 1051 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f44a0, count=8, MPI_INTEGER, dest=1052, tag=2107052, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2107052
Rank 1053 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f5580, count=8, MPI_INTEGER, dest=1049, tag=2101054, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2101054
Rank 1045 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21fba80, count=16, MPI_INTEGER, dest=1048, tag=2099046, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2099046
Rank 1048 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f8880, count=1, MPI_INTEGER, dest=1049, tag=2101049, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2101049
Rank 1052 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f33c0, count=6, MPI_INTEGER, dest=1049, tag=2101053, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2101053
Rank 1049 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f3440, count=1, MPI_INTEGER, dest=1048, tag=2099050, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2099050
Rank 1042 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f8740, count=7, MPI_INTEGER, dest=1050, tag=2103043, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2103043
Rank 1029 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f53c0, count=9, MPI_INTEGER, dest=1048, tag=2099030, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2099030
Rank 1046 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f22c0, count=3, MPI_INTEGER, dest=1048, tag=2099047, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2099047
Rank 1047 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21fca60, count=8, MPI_INTEGER, dest=1048, tag=2099048, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2099048
Rank 1050 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f6680, count=7, MPI_INTEGER, dest=1049, tag=2101051, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2101051
Rank 1041 [Fri Jun 30 15:20:52 2023] [c5-0c0s10n0] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x21f6600, count=7, MPI_INTEGER, dest=1050, tag=2103042, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2103042
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
fvcom_Physical_GC 0000000000998F7A for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAAAD4FAB20 Unknown Unknown Unknown
libc-2.22.so 00002AAAAD73B8C7 gsignal Unknown Unknown
libc-2.22.so 00002AAAAD73CC9A abort Unknown Unknown
libmpich_intel.so 00002AAAAD12E2C8 Unknown Unknown Unknown
libmpich_intel.so 00002AAAAD0B8182 MPIR_Handle_fatal Unknown Unknown
libmpich_intel.so 00002AAAAD0B82BE MPIR_Err_return_c Unknown Unknown
libmpich_intel.so 00002AAAAD049E79 MPI_Send Unknown Unknown
libmpich_intel.so 00002AAAAD082858 mpi_send Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
Unknown Unknown
fvcom_Physical_GC 00000000008013FD Unknown Unknown Unknown
fvcom_Physical_GC 000000000081F011 Unknown Unknown Unknown
fvcom_Physical_GC 00000000007FE03A Unknown Unknown Unknown
fvcom_Physical_GC 000000000040F53E Unknown Unknown Unknown
libc-2.22.so 00002AAAAD7276D5 __libc_start_main Unknown Unknown
fvcom_Physical_GC 000000000040F449 Unknown Unknown Unknown
_pmiu_daemon(SIGCHLD): [NID 01000] [c5-0c0s10n0] [Fri Jun 30 15:20:52 2023] PE RANK 1029 exit signal Aborted

Any help will be appreciated.

Thanks

Harshal

Re:FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

AishwaryaCV_Intel — Thu, 06 Jul 2023 10:13:42 GMT

Hi,

Thank you for posting in intel community.

Could you please provide the following details :

OS and output with lscpu command.
The sample reproducer along with steps to reproduce the issue at our end.
The MPI library you are using is from Intel?

Could you please try to run on the latest Intel MPI version(2021.9) and let us know if you still face the issue?

Thanks And Regards,

Aishwarya

Re: FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

AishwaryaCV_Intel — Fri, 14 Jul 2023 03:04:16 GMT

Hi,

We have not heard back from you, Could you please provide details we asked in the previous response?

Thanks And Regards,

Aishwarya

Re:FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

AishwaryaCV_Intel — Mon, 17 Jul 2023 11:30:26 GMT

Hi,

Since this is a duplicate thread of https://community.intel.com/t5/Intel-Fortran-Compiler/Fatal-error-in-MPI-Send-Invalid-tag-error-stack/td-p/1501772 ,we will no longer monitor this thread. We will continue addressing this issue in the other thread.

Thanks And Regards,

Aishwarya

topic Re:FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack in Intel® MPI Library

FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

Re:FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

Re: FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack

Re:FVCOM model : Fatal error in MPI_Send: Invalid tag, error stack