forrtl: severe (41): insufficient virtual memory

[Amit_P_]

Hi,

While running GSI application global test for "T574" resolution it gives me below error message:

----------------------------------------------------------QUOTE------------------------------------------------------------------

forrtl: severe (41): insufficient virtual memory
Image              PC                Routine            Line        Source
gsi.exe            0000000000434B9F  Unknown               Unknown  Unknown
gsi.exe            00000000004A88C2  Unknown               Unknown  Unknown
gsi.exe            0000000000A6E63A  Unknown               Unknown  Unknown
gsi.exe            0000000000412B99  Unknown               Unknown  Unknown
gsi.exe            00000000004078E3  Unknown               Unknown  Unknown
gsi.exe            0000000000405B09  Unknown               Unknown  Unknown
gsi.exe            0000000000405A96  Unknown               Unknown  Unknown
libc.so.6          00002BA355A67CDD  Unknown               Unknown  Unknown
gsi.exe            0000000000405989  Unknown               Unknown  Unknown
----------------------------------------------------------UNQUOTE------------------------------------------------------------------

Please suggest

[jimdempseyatthecove]

What this means is either your ulimit is set too low or your swap file size is too small (assuming you have sufficient RAM).

Jim Dempsey

[Amit_P_]

Hi Jim,

Thanks for the reply!!!

Ulimit is unlimited.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030831
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 10000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

vmstat output covers both the proper runtime and crash time:

$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
7 0 0 59276100 0 3103520 0 0 0 0 0 0 23 0 76 0 0
16 0 0 53086784 0 3104392 0 0 0 0 11121 12765 28 2 70 0 0
13 0 0 47619524 0 3114916 0 0 0 0 30555 37395 29 6 65 0 0
32 0 0 47345328 0 3114976 0 0 0 0 30266 8554 78 3 20 0 0
22 0 0 47347816 0 3114984 0 0 0 0 27103 2142 82 2 16 0 0
16 0 0 47348256 0 3114984 0 0 0 0 27125 6198 82 2 16 0 0
0 1 0 59929024 0 3092912 0 0 0 0 11263 26771 12 4 82 2 0
0 0 0 59929252 0 3092892 0 0 0 0 297 456 0 0 100 0 0
0 0 0 59932724 0 3092952 0 0 0 0 310 374 0 0 100 0 0
0 0 0 59934972 0 3092952 0 0 0 0 107 142 0 0 100 0 0 

[jimdempseyatthecove]

If your compiler options for Fortran are having the code make calls to MKL, in this MPI application, then insure that the single thread MKL library is linked into the program. Otherwise you will have an explosion of oversubscribed thread pools (one pool per MPI rank), each thread in each pool of size of number of hardware threads, each thread attempting to get unlimited stack. (perhaps number of hardware threads**2)

This is not to say you cannot use a mix of multi-threaded MKL with reduced number of threads, together with reduced number of MPI ranks. Which will require fine tuning to avoid running out of resources.

Jim Dempsey