insufficient virtual memory

[dkokron]

I am getting an error while loading a result from a general exploration experiment.  This is an MPI executable, but collection was only done for one process on each node.  See attached PNG for the error message.  Any ideas?

limit
cputime      unlimited
filesize     unlimited
datasize     4096000 kbytes
stacksize    7340032 kbytes
coredumpsize unlimited
memoryuse    1024000 kbytes
vmemoryuse   unlimited
descriptors  65536
memorylocked unlimited
maxproc      600

Intel(R) VTune(TM) Amplifier XE 2015 Update 1 (build 380310) Command Line Tool
Operating System          3.0.101-0.31.1.20140612-nasa SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

The result was collected using amplxe-cl -collect general-exploration -data-limit=100 ./3dh-r516+

I have tried running the GUI under gdb.  Here is a partial trace.  I don't have permission to install debug symbols on this machine.

Missing separate debuginfo for /nasa/intel/vtune_2015u1/intel/vtune_amplifier_xe_2015.1.1.380310/lib64/../lib64/../lib64/../lib64/libamplxe_file_finder_symbol_file_resolver_2.17.so
Try: zypper install -C "debuginfo(build-id)=4d9f614da242ad1479871d7dbfad96f2ea857b14"
Missing separate debuginfo for /usr/lib64/gio/modules/libgiogconf.so
Try: zypper install -C "debuginfo(build-id)=040d5845f253866ca3a688f112eb7695ed27d9da"
Missing separate debuginfo for /usr/lib64/libgconf-2.so.4
Try: zypper install -C "debuginfo(build-id)=4fa3827905f6332eb159e295a280bb1d3ad71c20"
Missing separate debuginfo for /usr/lib64/libORBit-2.so.0
Try: zypper install -C "debuginfo(build-id)=686f87bf164f077e1671c53cf7a9c0cc8b0be2d5"
Missing separate debuginfo for /usr/lib64/libdbus-glib-1.so.2
Try: zypper install -C "debuginfo(build-id)=c122a3188160b6baafdbab4395c30f971313859c"
Missing separate debuginfo for /lib64/libnsl.so.1
Try: zypper install -C "debuginfo(build-id)=1ca235c2af788c5f2505a4d7f3502c050e1c8462"
Missing separate debuginfo for /lib64/libdbus-1.so.3
Try: zypper install -C "debuginfo(build-id)=1482da7004b3bfeab07ac2f7d9bd4c1ec0a70098"
Missing separate debuginfo for /usr/lib64/gio/modules/libgiofam.so
Try: zypper install -C "debuginfo(build-id)=0755b389e9b7fea346a93a49dbf93c696f739673"
Missing separate debuginfo for /usr/lib64/libfam.so.0
Try: zypper install -C "debuginfo(build-id)=d3687487c4923fce53ba11c78a8103c657935dab"
Missing separate debuginfo for /usr/lib64/gio/modules/libgioremote-volume-monitor.so
Try: zypper install -C "debuginfo(build-id)=faf97450f43cca2c44b6dc98155c22bd283862ca"
Missing separate debuginfo for /usr/lib64/libgvfscommon.so.0
Try: zypper install -C "debuginfo(build-id)=5581c980efc0d98b142b17aad2f94b70f6b6ee35"
Missing separate debuginfo for /usr/lib64/gio/modules/libgvfsdbus.so
Try: zypper install -C "debuginfo(build-id)=3ad48cca78be85e98678248d144faa6345b669ef"
Missing separate debuginfo for /usr/lib64/libbeagle.so.1
Try: zypper install -C "debuginfo(build-id)=dd68a302550d54e69772fd36cae359f9a5ee139f"
[New Thread 0x7ff8dd7c0700 (LWP 89043)]
[New Thread 0x7ff71d7bf700 (LWP 89044)]
[Thread 0x7ff71d7bf700 (LWP 89044) exited]
[New Thread 0x7ff71d7bf700 (LWP 89045)]
[Thread 0x7ff71d7bf700 (LWP 89045) exited]
Missing separate debuginfo for /usr/lib64/gtk-2.0/2.10.0/loaders/libpixbufloader-xpm.so
Try: zypper install -C "debuginfo(build-id)=9b054e0277a6e97ee7198379695e13fa79facf75"
[Thread 0x7ff8dd7c0700 (LWP 89043) exited]
[Thread 0x7ffaa7fff700 (LWP 88898) exited]
Detaching after fork from child process 89131.
Detaching after fork from child process 89132.
[Thread 0x7ffe2ef08700 (LWP 88892) exited]
Detaching after fork from child process 89133.
Detaching after fork from child process 89134.
Detaching after fork from child process 89135.
Detaching after fork from child process 89136.
Detaching after fork from child process 89137.
[Thread 0x7ffc6ef07700 (LWP 88893) exited]
[Inferior 1 (process 88839) exited normally]
(gdb) quit

[Peter_W_Intel]

It seemed there was no problem (error message) during data collection, not sure if there was resource limitation in system when using VTune GUI to open a VTune result, you can try CLI to verify: "amplxe-cl -report hw-events -r vtune_result_dir" or copy vtune result to other machine (which installed VTune Amplifier) to open result.Probably you need to verify/adjust system configuration? 

[Vitaly_S_Intel]

Based on gdb output I'm assuming you are running 64bit amplxe-gui from bin64/amplxe-gui, right? Does this error happen when you open result first time right after amplxe-gui start?

Can you please reproduce this issue, run command below while amplxe-gui running and provide its output:

$ cat /proc/<amplxe-gui PID>/stat

[dkokron]

Vitaly,

Yes, I am running the 64-bit amplxe-gui

which amplxe-gui
{PATH}/vtune_2015u1/intel/vtune_amplifier_xe_2015/bin64/amplxe-gui

It seems to matter if I launch the GUI from a regular ssh session (which works, but is very slow) versus a VNC session (which fails with the virtual memory error)

When launched from a VNC session

cat /proc/55557/stat
55557 (amplxe-gui) S 54541 55557 54541 34823 55557 8192 44474 3416 1 0 153 24 0 0 20 0 4 0 341951674 23065718784 18798 1048576000 4194304 5078812 140737488345232 140737488344192 140729872533942 0 0 16781312 1073743022 18446744073709551615 0 0 17 2 0 0 84 0 0

When launched from a regular ssh session on a different host (works)

cat /proc/2018/stat
2018 (amplxe-gui) S 2756 2018 2756 34933 2179 8192 45470 3423 4 0 169 24 0 0 20 0 4 0 223193354 23075815424 18712 1048576000 4194304 5078812 140737488345680 140737488344704 140729872533942 0 0 16781312 1073743022 18446744073709551615 0 0 17 9 0 0 40 0 0

I took a suggestion from idz.admin (private message) to change my VNC session stacksize to 10240 KB and now the gui works under VNC.

limit stacksize 10240

57863 (amplxe-gui) S 54541 57863 54541 34823 57863 8192 273845 3416 117 0 1617 5047 0 0 20 0 32 0 342044638 1523097600 126354 1048576000 4194304 5078812 140737488345232 140737488344192 140737254508982 0 0 16781312 1073743022 18446744073709551615 0 0 17 2 0 0 21 0 0

Thanks to all who responded.

Dan

[dkokron]

This is still an issue under vtune_amplifier_2018.3.0.558279.  I get an error "insufficient virtual memory" when attempting to open the source/assembly window on tall poll from a general exploration experiment.  Setting the shell stacksize to 10000 doesn't help.