Error: insufficient virtual memory

[lunawing]

I am building a computing environment on a Rocks cluster (3.3.0) with Intel compiler (icc and ifort ver. 9.1) and mpi lib (ver. 3.0). It runs well with the test codes which come with the Intel mpi and mpich2. However, when I submit my code (a gasdynamics solver), the program run well for a while and then it breaks with the following error message:

forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
a.out 080F0B78 Unknown Unknown Unknown
a.out 080EF311 Unknown Unknown Unknown
a.out 080C8897 Unknown Unknown Unknown
a.out 0809F6C4 Unknown Unknown Unknown
a.out 080B8328 Unknown Unknown Unknown
a.out 080B8235 Unknown Unknown Unknown
a.out 08078CC2 Unknown Unknown Unknown
a.out 0804B0A6 Unknown Unknown Unknown
a.out 08049B9E Unknown Unknown Unknown
libc.so.6 401FE79D Unknown Unknown Unknown
a.out 08049AD1 Unknown Unknown Unknown
[cli_4]: aborting job:
Fatal error in MPI_Wait: Other MPI error, error stack:
MPI_Wait(139).............................: MPI_Wait(request=0xbfffdd44, status0x81172d0) failed
MPIDI_CH3_Progress(733)...................: handle_sock_op failed
MPIDI_CH3I_Progress_handle_sock_event(277):
MPIDU_Socki_handle_read(623)..............: connection closed by peer (set=0,sock=1)
rank 0 in job 1 tanya_35204 caused collective abort of all ranks
exit status of rank 0: return code 41

I already set virtual memory to be unlimited. Any suggestion?

Thanks.

[babe]

I think I have the same problem. Have anybody a solution in that time?

Thanks

[Andrey_D_Intel]

It looks like the appliaction ate whole memory on the compute node. Did you saw the "out of memory" messages in the system logs?