VTune 2018 outputs "out of memory" when memory is far away of exhaustion

[Zheng__Eric]

Hello,

I want to analyze hotspots for a program but vtune2018 outputs "out of memory" and some assertion fail(after a period time of execution). But actually, memory is far away of exhaustion. Its output is as following

Freepages: Out of memory

vcs/collectunits1/sal/src/linux/sal.c:961 sal_munmap: Assertion 'result == 0' failed.

vcs/collectunits1/sal/src/linux/sal.c:961 sal_munmap: Assertion 'result == 0' failed.

AMPLXE_TPSSCOLLECTOR: init:1785: ((((&(g_tpss_module_map->cache_list))->prev == (&(g_tpss_module_map->cache_list))) && ((&(g_tpss_module_map->cache_list))->next == (&(g_tpss_module_map->cache_list))))) : 

Assertion failed: init:1785: ((((&(g_tpss_module_map->cache_list))->prev == (&(g_tpss_module_map->cache_list))) && ((&(g_tpss_module_map->cache_list))->next == (&(g_tpss_module_map->cache_list))))) : . Please contact the technical support. amplxe: Error: Assertion failed: ?invalid?. Please contact the technical support. 

amplxe: Collection stopped.

And if I execute it with srun in another machine(with the same environment and hardware), vtune says it starts but no corresponding process is created(only amplex-cl process exists). Could anyone help me with this problem?

My environment is 

• Ubuntu 16.04.4 LTS
• Intel(R) VTune(TM) Amplifier 2018 Update 2 (build 551022) Command Line Tool
• Selftest reuslt

    Intel(R) VTune(TM) Amplifier Self Check Utility

    Copyright (C) 2009-2017 Intel Corporation. All rights reserved.

    Build Number: 551022

    Instrumentation based analysis check

    Example of analysis types: Hotspots, Concurrency, Locks and Waits

        Collection: Ok

        Finalization: Ok

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libpthread.so.0'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libc.so.6'.

    amplxe: Warning: Cannot locate debugging information for file `/opt/intel/vtune_amplifier_2018.2.0.551022/lib64/libtpsstool.so'.

        Report: Ok

    HW event-based analysis check (Intel driver)

    Example of analysis types: Advanced Hotspots, HPC Performance Characterization, etc.

        Collection: Ok

    amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

        Finalization: Ok

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/ld-2.23.so'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libc-2.23.so'.

    amplxe: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel An

    alysis topic in the product online help for instructions.

        Report: Ok

    HW event-based analysis check (Intel driver)

    Example of analysis types: General Exploration

        Collection: Ok

    amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

        Finalization: Ok

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/ld-2.23.so'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/modules/4.4.0-116-generic/kernel/drivers/net/ethernet/broadcom/tg3.ko'.

    amplxe: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel An

    alysis topic in the product online help for instructions.

        Report: Ok

    HW event-based analysis with uncore events (Intel driver)

    Example of analysis types: Memory Access

        Collection: Ok

    amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

        Finalization: Ok

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libgcc_s.so.1'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libpthread-2.23.so'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/modules/4.4.0-116-generic/kernel/drivers/net/ethernet/broadcom/tg3.ko'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/modules/4.4.0-116-generic/kernel/arch/x86/kvm/kvm.ko'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libc-2.23.so'.

    amplxe: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel An

    alysis topic in the product online help for instructions.

        Report: Ok

    HW event-based analysis with stacks (Intel driver)

    Example of analysis types: Advanced Hotspots with Stacks, etc.

        Collection: Ok

    amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

        Finalization: Ok

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libgcc_s.so.1'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libpthread-2.23.so'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/ld-2.23.so'.

    amplxe: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libm-2.23.so'.

    amplxe: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel An

    alysis topic in the product online help for instructions.

        Report: Ok

    The system is ready to be used for performance analysis with Intel VTune Amplifier.

    Review warnings in the output above to find product limitations, if any.

This program is open source, if necessary I can provide it.

Thank you in advance.

[Vladimir_R_Intel]

Hi Eric,

So as the self check has passed it seems on some specific interaction with this particular application, so please share it.

And you can use advanced hotspots with stacks analysis as a workaround.

BR,

Vladimir

[elenamesga]

Hi, I have the same problem. I am using the vtune and advisor profiler but I always get the following error:

AMPLXE_TPSSCOLLECTOR: init:1898: ((((&(g_tpss_module_map->cache_list))->prev == (&(g_tpss_module_map->cache_list))) && ((&(g_tpss_module_map->cache_list))->next == (&(g_tpss_module_map->cache_list))))) :
To stop the collection, either press CTRL-C or enter from another console window: vtune -r /mnt/scratch/users/pab_001_uma/emesga/tmp_Softwares_Compare/softwares_compare/testing_cluster/results_hifiasm/32THREADS/D.melanogaster/r002hs -command stop.
vcs/collectunits1/sal/src/linux/sal.c:955 sal_munmap: Assertion 'result == 0' failed.

Freepages: Out of memory
Freepages: Out of memory
vcs/collectunits1/sal/src/linux/sal.c:955 sal_munmap: Assertion 'result == 0' failed.

Assertion failed: init:1898: ((((&(g_tpss_module_map->cache_list))->prev == (&(g_tpss_module_map->cache_list))) && ((&(g_tpss_module_map->cache_list))->next == (&(g_tpss_module_map->cache_list))))) : . Please contact the technical support. vtune: Error: Assertion failed: ?invalid?. Please contact the technical support.
vtune: Collection stopped.
vtune: Using result path `/mnt/scratch/users/pab_001_uma/emesga/tmp_Softwares_Compare/softwares_compare/testing_cluster/results_hifiasm/32THREA
DS/D.melanogaster/r002hs'

Do you know what could be the problem and how could I solve it?

Thank you. I look forward to hearing from you,

Elena.