Fatal error in MPI_Send: Invalid tag, error stack

[harshal05]

Hello, 

I am running the FVCOM model which is running successfully with less than or equal to 1048 processes but when I am increasing the number of the processes to more than 1048 it is giving me an error: Fatal error in MPI_Send: Invalid tag, error stack 

I am using Intel 2017 and Cray-MPICH v7.6.3 implementation (MPI_TAG_UB=1681915906) 

Machine: Cray XC40

Rank 862 [Fri Jun 30 15:20:52 2023] [c5-0c0s8n2] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x210e4e0, count=8, MPI_INTEGER, dest=1050, tag=2102863, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2102863
Rank 861 [Fri Jun 30 15:20:52 2023] [c5-0c0s8n2] Fatal error in MPI_Send: Invalid tag, error stack:
MPI_Send(186): MPI_Send(buf=0x210a300, count=5, MPI_INTEGER, dest=1050, tag=2102862, MPI_COMM_WORLD) failed
MPI_Send(111): Invalid tag, value is 2102862

I have attched log file.  

Please help me to understand this. 

Thanks 

Harshal

[Ron_Green]

This isn't something we can help you with on this Forum.  

Invalid tags in MPI usually indicate that your code is not doing the data decomposition correctly, or you have missing cell data or fields in your simulation.  Or you input deck has and error in the parameters that describe the decomposition.   Check your input data and/or restart files.

Your best bet is to contact help from the FVCOM developers or community.  I suspect your simulation setup is wrong.  

[JohnNichols]

You could try it on a Intel NUC Core i3, my current computing machine.