https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897158#M2133 I am building a computing environment on a Rocks cluster (3.3.0) with Intel compiler (icc and ifort ver. 9.1) and mpi lib (ver. 3.0). It runs well with the test codes which come with the Intel mpi and mpich2. However, when I submit my code (a gasdynamics solver), the program run well for a while and then it breaks with the following error message:<BR /><BR /><BLOCKQUOTE>forrtl: severe (41): insufficient virtual memory<BR />Image PC Routine Line Source<BR />a.out 080F0B78 Unknown Unknown Unknown<BR />a.out 080EF311 Unknown Unknown Unknown<BR />a.out 080C8897 Unknown Unknown Unknown<BR />a.out 0809F6C4 Unknown Unknown Unknown<BR />a.out 080B8328 Unknown Unknown Unknown<BR />a.out 080B8235 Unknown Unknown Unknown<BR />a.out 08078CC2 Unknown Unknown Unknown<BR />a.out 0804B0A6 Unknown Unknown Unknown<BR />a.out 08049B9E Unknown Unknown Unknown<BR />libc.so.6 401FE79D Unknown Unknown Unknown<BR />a.out 08049AD1 Unknown Unknown Unknown<BR />[cli_4]: aborting job:<BR />Fatal error in MPI_Wait: Other MPI error, error stack:<BR />MPI_Wait(139).............................: MPI_Wait(request=0xbfffdd44, status0x81172d0) failed<BR />MPIDI_CH3_Progress(733)...................: handle_sock_op failed<BR />MPIDI_CH3I_Progress_handle_sock_event(277):<BR />MPIDU_Socki_handle_read(623)..............: connection closed by peer (set=0,sock=1)<BR />rank 0 in job 1 tanya_35204 caused collective abort of all ranks<BR /> exit status of rank 0: return code 41<BR /></BLOCKQUOTE><BR />I already set virtual memory to be unlimited. Any suggestion? <BR /><BR />Thanks.<BR /> Sun, 26 Nov 2006 12:12:58 GMT lunawing 2006-11-26T12:12:58Z https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897158#M2133 I am building a computing environment on a Rocks cluster (3.3.0) with Intel compiler (icc and ifort ver. 9.1) and mpi lib (ver. 3.0). It runs well with the test codes which come with the Intel mpi and mpich2. However, when I submit my code (a gasdynamics solver), the program run well for a while and then it breaks with the following error message:<BR /><BR /><BLOCKQUOTE>forrtl: severe (41): insufficient virtual memory<BR />Image PC Routine Line Source<BR />a.out 080F0B78 Unknown Unknown Unknown<BR />a.out 080EF311 Unknown Unknown Unknown<BR />a.out 080C8897 Unknown Unknown Unknown<BR />a.out 0809F6C4 Unknown Unknown Unknown<BR />a.out 080B8328 Unknown Unknown Unknown<BR />a.out 080B8235 Unknown Unknown Unknown<BR />a.out 08078CC2 Unknown Unknown Unknown<BR />a.out 0804B0A6 Unknown Unknown Unknown<BR />a.out 08049B9E Unknown Unknown Unknown<BR />libc.so.6 401FE79D Unknown Unknown Unknown<BR />a.out 08049AD1 Unknown Unknown Unknown<BR />[cli_4]: aborting job:<BR />Fatal error in MPI_Wait: Other MPI error, error stack:<BR />MPI_Wait(139).............................: MPI_Wait(request=0xbfffdd44, status0x81172d0) failed<BR />MPIDI_CH3_Progress(733)...................: handle_sock_op failed<BR />MPIDI_CH3I_Progress_handle_sock_event(277):<BR />MPIDU_Socki_handle_read(623)..............: connection closed by peer (set=0,sock=1)<BR />rank 0 in job 1 tanya_35204 caused collective abort of all ranks<BR /> exit status of rank 0: return code 41<BR /></BLOCKQUOTE><BR />I already set virtual memory to be unlimited. Any suggestion? <BR /><BR />Thanks.<BR /> Sun, 26 Nov 2006 12:12:58 GMT https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897158#M2133 lunawing 2006-11-26T12:12:58Z https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897159#M2134 I think I have the same problem. Have anybody a solution in that time? <BR /> <BR />Thanks Fri, 16 Feb 2007 12:25:30 GMT https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897159#M2134 babe 2007-02-16T12:25:30Z https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897160#M2135 It looks like the appliaction ate whole memory on the compute node. Did you saw the "out of memory" messages in the system logs? Tue, 27 Feb 2007 11:06:01 GMT https://community.intel.com/t5/Intel-MPI-Library/Error-insufficient-virtual-memory/m-p/897160#M2135 Andrey_D_Intel 2007-02-27T11:06:01Z