Re: Re:VTune exception: 'NoneType' object has no attribute '_get_as_query'

wspear · ‎01-19-2021

I'm running a pretty straightforward openmp matrix multiplication example on Ubuntu 20.04 which fails when I try to run it with VTune, as shown below. Any ideas how to proceed?

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect performance-snapshot -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/wspear/bin/tau2/examples/mm-nopie/r002ps -command stop.
vtune: Collection stopped.
vtune: Using result path `/home/wspear/bin/tau2/examples/mm-nopie/r002ps'
vtune: Executing actions 75 % Generating a report                              exception 'NoneType' object has no attribute '_get_as_query'
Traceback (most recent call last):
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/reporter.python/src/python/tmpbu237__3/wrapper.py", line 96, in run
  File "/opt/intel/oneapi/vtune/2021.1.1/config/report/summary.tmpl", line 107, in <module>
    drawer = parser.create_drawer(adv_footer=adv_footer)
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 250, in create_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 243, in _create_child_drawers
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 288, in _create_section_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 321, in _create_tree_tt
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 237, in _create_query
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/dicerhelpers.python/python/tmp3vbjshma/qlibrary.py", line 256, in append
AttributeError: 'NoneType' object has no attribute '_get_as_query'
vtune: Executing actions 100 % done                                            
vtune: Error: 0x40000028 (Reporter error)

Sanity check:

wspear@yu:~/bin/tau2/examples/mm-nopie$ file ./matmult
./matmult: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b48145f254a00cabd7529aadc5a211af1bfd47ca, for GNU/Linux 3.2.0, with debug_info, not stripped

AthiraM_Intel · ‎01-20-2021

Hi,

Thanks for reaching out to us.

We are not able to reproduce the issue.

Can you try to run the self_check.py script which will be available in <vtune-install-dir>/bin64. Please share the output with us so that we can validate it from our end.

Thanks.

wspear · ‎01-20-2021

self_check.py doesn't exist but here is the output of vtune-self-check.sh.

wspear@yu:~/bin/tau2/examples/LULESH$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune-self-checker.sh 
Intel(R) VTune(TM) Profiler Self Check Utility
Copyright (C) 2009-2020 Intel Corporation. All rights reserved.
Build Number: 613804

HW event-based analysis (counting mode)
Example of analysis types: Performance Snapshot
    Collection: Fail

Instrumentation based analysis check
Example of analysis types: Hotspots with default knob sampling-mode=sw, Threading with default knob sampling-and-waits=sw
    Collection: Fail
vtune: Error: Cannot start data collection because the scope of ptrace system call application is limited. To enable profiling, please set /proc/sys/kernel/yama/ptrace_scope to 0. See the Release Notes for instructions on enabling it permanently.
vtune: Warning: Hardware collection of CPU events is not possible on this system. Microarchitecture performance insights will not be available.

HW event-based analysis check
Example of analysis types: Hotspots with knob sampling-mode=hw, HPC Performance Characterization, etc.
    Collection: Fail
vtune: Error: This analysis type requires either an access to kernel-mode monitoring in the Linux perf subsystem or installation of the VTune Profiler drivers (see the "Sampling Drivers" help topic for further details). Please set the /proc/sys/kernel/perf_event_paranoid value to 1 or less to continue without installing the drivers.
vtune: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel module symbols.
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.

HW event-based analysis check
Example of analysis types: Microarchitecture Exploration
    Collection: Fail
vtune: Error: This analysis type is not applicable to the current machine microarchitecture.

HW event-based analysis with uncore events
Example of analysis types: Memory Access
    Collection: Fail
vtune: Error: This analysis type is not applicable to the current machine microarchitecture.

HW event-based analysis with stacks
Example of analysis types: Hotspots with knob sampling-mode=hw and knob enable-stack-collection=true, etc.
    Collection: Fail
vtune: Error: To run this analysis, either set the Stack size option to the unlimited value (0 in command line), or provide access to kernel-mode monitoring with the /proc/sys/kernel/perf_event_paranoid value set to 1 or less.
vtune: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel module symbols.
vtune: Error: Unlimited stack size (0) not allowed in driverless mode.
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.

HW event-based analysis with context switches
Example of analysis types: Threading with knob sampling-and-waits=hw
    Collection: Fail
vtune: Error: This analysis type requires either an access to kernel-mode monitoring in the Linux perf subsystem or installation of the VTune Profiler drivers (see the "Sampling Drivers" help topic for further details). Please set the /proc/sys/kernel/perf_event_paranoid value to 1 or less to continue without installing the drivers.
vtune: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel module symbols.
vtune: Warning: Context switch data cannot be collected in the current driverless mode if the kernel version is less than 4.3 or /proc/sys/kernel/perf_event_paranoid value is greater than 1. Update your system configuration for driverless collection or consider switching to the Intel sampling driver by setting an unlimited (0) value for the Stack size option.
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.

vtune: Warning: VTune Profiler driver with insufficient permission is detected on the system.
vtune: Warning: Consider setting proper driver permissions (see the "Sampling Drivers" help topic).
vtune: Warning: Otherwise, the driverless collection with limited analysis support will be enabled by default.

The check observed a product failure on your system.
Review errors in the output above to fix a problem or contact Intel technical support.

Log location: /tmp/vtune-tmp-wspear/self-checker-2021.01.20_07.51.40/log.txt

I tried setting perf_event_paranoid, kptr_restrict and ptrace_scope to 0 and the check output became:

Intel(R) VTune(TM) Profiler Self Check Utility
Copyright (C) 2009-2020 Intel Corporation. All rights reserved.
Build Number: 613804

HW event-based analysis (counting mode) (Perf)
Example of analysis types: Performance Snapshot
    Collection: Ok
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
    Finalization: Ok...
    Report: Fail
vtune: Error: 0x40000028 (Reporter error)

Instrumentation based analysis check
Example of analysis types: Hotspots with default knob sampling-mode=sw, Threading with default knob sampling-and-waits=sw
    Collection: Fail

HW event-based analysis check (Perf)
Example of analysis types: Hotspots with knob sampling-mode=hw, HPC Performance Characterization, etc.
    Collection: Ok
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
    Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
    Report: Ok

HW event-based analysis check
Example of analysis types: Microarchitecture Exploration
    Collection: Fail
vtune: Error: This analysis type is not applicable to the current machine microarchitecture.

HW event-based analysis with uncore events
Example of analysis types: Memory Access
    Collection: Fail
vtune: Error: This analysis type is not applicable to the current machine microarchitecture.

HW event-based analysis with stacks (Perf)
Example of analysis types: Hotspots with knob sampling-mode=hw and knob enable-stack-collection=true, etc.
    Collection: Ok
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
    Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
    Report: Ok

HW event-based analysis with context switches (Perf)
Example of analysis types: Threading with knob sampling-and-waits=hw
    Collection: Ok
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
    Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
    Report: Ok

vtune: Warning: VTune Profiler driver with insufficient permission is detected on the system.
vtune: Warning: Consider setting proper driver permissions (see the "Sampling Drivers" help topic).
vtune: Warning: Otherwise, the driverless collection with limited analysis support will be enabled by default.

The check observed a product failure on your system.
Review errors in the output above to fix a problem or contact Intel technical support.

Log location: /tmp/vtune-tmp-wspear/self-checker-2021.01.20_08.02.34/log.txt

However, my original test still fails in the same way:

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect performance-snapshot -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/wspear/bin/tau2/examples/mm-nopie/r000ps -command stop.
Done.
vtune: Collection stopped.
vtune: Using result path `/home/wspear/bin/tau2/examples/mm-nopie/r000ps'
vtune: Executing actions 75 % Generating a report                              exception 'NoneType' object has no attribute '_get_as_query'
Traceback (most recent call last):
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/reporter.python/src/python/tmpbu237__3/wrapper.py", line 96, in run
  File "/opt/intel/oneapi/vtune/2021.1.1/config/report/summary.tmpl", line 107, in <module>
    drawer = parser.create_drawer(adv_footer=adv_footer)
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 250, in create_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 243, in _create_child_drawers
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 288, in _create_section_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 321, in _create_tree_tt
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 237, in _create_query
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/dicerhelpers.python/python/tmp3vbjshma/qlibrary.py", line 256, in append
AttributeError: 'NoneType' object has no attribute '_get_as_query'
vtune: Executing actions 100 % done                                            
vtune: Error: 0x40000028 (Reporter error)

AthiraM_Intel · ‎01-21-2021

Hi,

Could you please share the result directory and cpu information.

Are you observed any issue during installation?

Also check the driver installation:

To verify that the sampling driver is installed correctly, please follow below steps:

cd <install-dir>/sepdk/src

./insmod-sep -q

This provides information on whether the drivers are currently loaded. Please share the observations.

Thanks.

wspear · ‎01-21-2021

There seems to be an issue with some of the drivers but I'm not sure how to address this.

wspear@yu:/opt/intel/oneapi/vtune/latest/sepdk/src$ ./insmod-sep -q
pax driver is loaded and owned by group "vtune" with file permissions "660".
socperf3 driver is not correctly loaded.
sep5 driver is not correctly loaded.

socwatch2_13 driver is loaded and owned by group "vtune" with file permissions "660".

vtsspp driver is loaded and owned by group "vtune" with file permissions "660".

I didn't notice any issues installing OneAPI, though I wonder if I somehow got a beta version by accident.

Results directory from an attempted run is attached.

CPU Info:

wspear@yu:~/bin/tau2/examples/mm-nopie$ cpuinfo 
Intel(R) processor family information utility, Version 2021.1 Build 20201112 (id: b9c9d2fc5)
Copyright (C) 2005-2020 Intel Corporation.  All rights reserved.

=====  Processor composition  =====
Processor name    : Intel(R) Xeon(R)  X5680  
Packages(sockets) : 2
Cores             : 12
Processors(CPUs)  : 24
Cores per package : 6
Threads per core  : 2

wspear · ‎01-21-2021

Based on those driver loading errors I unloaded the drivers with sudo ./rmmod-sep -s and then reinstalled them and then made sure I was in the vtune group (I hadn't been before). All of the drivers are listed as loaded now. However I get the the same final error as before when I run my test. The only change is this new error output showing up before the "collection started" message:

***ERROR: could not retrieve time stamp!
Warning: CPU_CLK_UNHALTED.REF_P includes an unrecognized modifier sample. Discarding modifier.
Warning: CPU_CLK_UNHALTED.REF_P includes an unrecognized modifier sa133000. Discarding modifier.
ERROR with Preparing core for Start (228)- Abort current run.
Event configuration failed - Aborting emon collection...

AthiraM_Intel · ‎01-25-2021

Hi,

We are forwarding your case to Subject Matter Experts.

Could you please share the result directory for the analysis after loading the SEP driver.

Thanks.

wspear · ‎01-25-2021

Here it is. Thank you.

Heinrich_B_Intel · ‎01-26-2021

Hi Wyatt,

from your description I see that you are calling vtune with full path. Did you source the /opt/intel/oneapi/setvars.sh file?

I will check your results directory later.

best regards,

Heinrich

wspear · ‎01-26-2021

That's just the string I extracted from the vtune gui, which I tried out when my first command line efforts returned this error. setvars.sh is sourced in my .bash_aliases file. I get the same result invoking the vtune command without its path.

Thanks,

Wyatt

Heinrich_B_Intel · ‎02-18-2021

Hi Wyatt,

sorry for the long delay. there might be an issue with the vtune installation on Ubuntu 20.04. Colleagues have identified that some permissions were not right. If you have sudo rights you may try:

sudo chmod o+rx /opt/intel/oneapi/vtune/2021.1.1bin64/pinbin

this issues should be fixed in the next update of oneAPI.

best regards,

Heinrich

wspear · ‎02-18-2021

I was able to chmod pinbin but I still get the same error. Output directory attached.

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect performance-snapshot -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
***ERROR: could not retrieve time stamp!
Warning: CPU_CLK_UNHALTED.REF_P includes an unrecognized modifier sample. Discarding modifier.
Warning: CPU_CLK_UNHALTED.REF_P includes an unrecognized modifier sa133000. Discarding modifier.
ERROR with Preparing core for Start (228)- Abort current run.
Event configuration failed - Aborting emon collection...
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/wspear/bin/tau2/examples/mm-nopie/r000ps -command stop.
Done.
vtune: Collection stopped.
vtune: Using result path `/home/wspear/bin/tau2/examples/mm-nopie/r000ps'
vtune: Executing actions 75 % Generating a report                              exception 'NoneType' object has no attribute '_get_as_query'
Traceback (most recent call last):
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/reporter.python/src/python/tmpbu237__3/wrapper.py", line 96, in run
  File "/opt/intel/oneapi/vtune/2021.1.1/config/report/summary.tmpl", line 107, in <module>
    drawer = parser.create_drawer(adv_footer=adv_footer)
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 250, in create_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 243, in _create_child_drawers
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 288, in _create_section_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 321, in _create_tree_tt
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 237, in _create_query
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/dicerhelpers.python/python/tmp3vbjshma/qlibrary.py", line 256, in append
AttributeError: 'NoneType' object has no attribute '_get_as_query'
vtune: Executing actions 100 % done                                            
vtune: Error: 0x40000028 (Reporter error)

Kirill_U_Intel · ‎02-19-2021

Hi,

Could you fix configuration file and try the collection again?

Replace VTUNE_INSTALL_DIR\config\analysis_type\include\common_events.xsl

<xsl:text>,CPU_CLK_UNHALTED.REF_P:sample:sa</xsl:text>

Should be

<xsl:text>,CPU_CLK_UNHALTED.REF_P:sample:sa=</xsl:text>

Thanks, Kirill

wspear · ‎02-19-2021

That change (adding the equals sign before the closing tag) didn't seem to make any difference. New output directory attached.

Kirill_U_Intel · ‎02-19-2021

ok.

could you run a few simple collections?

/opt/intel/oneapi/vtune/2021.1.1/bin64/sep -dry-run

/opt/intel/oneapi/vtune/2021.1.1/bin64/sep -start -d 1

remove vtune drivers 
cd /opt/intel/oneapi/vtune/2021.1.1/sepdk/src && ./rmmod-sep

/opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect performance-snapshot -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult

remove vtune drivers 
cd /opt/intel/oneapi/vtune/2021.1.1/sepdk/src && ./rmmod-sep

/opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect hotspots -knob sampling-mode=hw -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult

wspear · ‎02-19-2021

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/sep -dry-run
***ERROR: could not retrieve time stamp!
PMU type: Core (core)
Group 0:
INST_RETIRED.ANY: event id 2
CPU_CLK_UNHALTED.THREAD: event id 1
CPU_CLK_UNHALTED.REF: event id 0

CCCR
reg_type       = 1	event_id       = g	reg_id         = 38f	reg_value      = 700000000
reg_type       = 1	event_id       = g	reg_id         = 390	reg_value      = 700000000
reg_type       = 1	event_id       = g	reg_id         = 3f1	reg_value      = 0
reg_type       = 1	event_id       = g	reg_id         = 1d9	reg_value      = 1800
reg_type       = 1	event_id       = g	reg_id         = 38d	reg_value      = bbb

ESCR
reg_type       = 2	event_id       = g	reg_id         = 1c8	reg_value      = 0

DATA
reg_type       = 4	event_id       = 2	reg_id         = 309	reg_value      = ffffffe17b80
reg_type       = 4	event_id       = 1	reg_id         = 30a	reg_value      = ffffffe17b80
reg_type       = 4	event_id       = 0	reg_id         = 30b	reg_value      = ffffffe17b80

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/sep -start -d 1
***ERROR: could not retrieve time stamp!
CPU Event configuration failed - Aborting sampling run ...
Failed to verify PMU programming information

wspear@yu:/opt/intel/oneapi/vtune/2021.1.1/sepdk/src$ sudo ./rmmod-sep 
[sudo] password for wspear: 

Removing sep5 driver from the kernel ... done.
The sep5 driver has been successfully unloaded.

Removing vtsspp driver from the kernel ... done.
The vtsspp driver has been successfully unloaded.

Removing socwatch2_13 driver from the kernel ... done.
The socwatch2_13 driver has been successfully unloaded.

Removing socperf3 driver from the kernel ... done.
The socperf3 driver has been successfully unloaded.

No driver needs PMU arbitration anymore: the PAX service can be safely removed.
Attempting to stop PAX service ...
Removing pax driver from the kernel ... done.
The pax driver has been successfully unloaded.
PAX service has been stopped.

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect performance-snapshot -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/wspear/bin/tau2/examples/mm-nopie/r000ps -command stop.
Done.
vtune: Collection stopped.
vtune: Using result path `/home/wspear/bin/tau2/examples/mm-nopie/r000ps'
vtune: Executing actions 75 % Generating a report                              exception 'NoneType' object has no attribute '_get_as_query'
Traceback (most recent call last):
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/reporter.python/src/python/tmpbu237__3/wrapper.py", line 96, in run
  File "/opt/intel/oneapi/vtune/2021.1.1/config/report/summary.tmpl", line 107, in <module>
    drawer = parser.create_drawer(adv_footer=adv_footer)
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 250, in create_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 243, in _create_child_drawers
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 288, in _create_section_drawer
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 321, in _create_tree_tt
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/perfconfigs/templates/common/tmphu7ca65z/common_summary.py", line 237, in _create_query
  File "/localdisk/BB/INNLphep2l4r/b/tmpT2UjPi/build/build_release_posix-x86_64_icl_19.1.2_gnutools_6.3_binutils_2.28/dicerhelpers.python/python/tmp3vbjshma/qlibrary.py", line 256, in append
AttributeError: 'NoneType' object has no attribute '_get_as_query'
vtune: Executing actions 100 % done                                            
vtune: Error: 0x40000028 (Reporter error)

I assume the drivers only need to be stopped once.

wspear@yu:~/bin/tau2/examples/mm-nopie$ /opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect hotspots -knob sampling-mode=hw -app-working-dir /home/wspear/bin/tau2/examples/mm-nopie -- /home/wspear/bin/tau2/examples/mm-nopie/matmult
vtune: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Profiler on such systems.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/wspear/bin/tau2/examples/mm-nopie/r001hs -command stop.
Done.
vtune: Collection stopped.
vtune: Using result path `/home/wspear/bin/tau2/examples/mm-nopie/r001hs'
vtune: Executing actions 19 % Resolving information for `kvm.ko'               
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-65-generic/kernel/arch/x86/kvm/kvm.ko'.
vtune: Executing actions 19 % Resolving information for `matmult'              
vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28'.
vtune: Executing actions 19 % Resolving information for `libittnotify_collector
vtune: Warning: Cannot locate debugging information for file `/opt/intel/oneapi/vtune/2021.1.1/lib64/runtime/libittnotify_collector.so'.
vtune: Warning: Cannot locate file `vmlinux'.
vtune: Executing actions 19 % Resolving information for `libiomp5.so'          
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 20 % Resolving information for `nf_conntrack.ko'      
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-65-generic/kernel/net/netfilter/nf_conntrack.ko'.
vtune: Executing actions 20 % Resolving information for `igb.ko'               
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-65-generic/kernel/drivers/net/ethernet/intel/igb/igb.ko'.
vtune: Executing actions 75 % Generating a report                              Elapsed Time: 15.474s
    CPU Time: 355.299s
        Effective Time: 350.556s
            Idle: 0.006s
            Poor: 3.184s
            Ok: 64.294s
            Ideal: 283.072s
            Over: 0s
        Spin Time: 4.741s
            Imbalance or Serial Spinning: 4.677s
            Lock Contention: 0s
            Other: 0.064s
        Overhead Time: 0.002s
            Creation: 0s
            Scheduling: 0s
            Reduction: 0s
            Atomics: 0s
            Other: 0.002s
    Instructions Retired: 793,852,020,000
    Microarchitecture Usage: 32.5%
        CPI Rate: 1.540
         | The CPI may be too high. This could be caused by issues such as
         | memory stalls, instruction starvation, branch misprediction or long
         | latency instructions. Explore the other hardware-related metrics to
         | identify what is causing high CPI.
         |
    Total Thread Count: 25
    Paused Time: 0s

Top Hotspots
Function                                                                                          Module       CPU Time
------------------------------------------------------------------------------------------------  -----------  --------
compute$omp$parallel@102                                                                          matmult      229.687s
compute_interchange$omp$parallel@123                                                              matmult      114.707s
_INTERNALd2589eef::__kmp_wait_template<kmp_flag_64<(bool)0, (bool)1>, (bool)1, (bool)0, (bool)1>  libiomp5.so    4.344s
native_queued_spin_lock_slowpath                                                                  vmlinux        2.388s
kmp_flag_native<unsigned long long, (flag_type)1, (bool)1>::notdone_check                         libiomp5.so    0.457s
[Others]                                                                                          N/A            3.716s
    Effective Logical Core Utilization: 94.4% (22.655 out of 24)
Collection and Platform Info
    Application Command Line: /home/wspear/bin/tau2/examples/mm-nopie/matmult 
    Operating System: 5.4.0-65-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
    Computer Name: yu
    Result Size: 240.0 MB 
    Collection start time: 16:11:16 19/02/2021 UTC
    Collection stop time: 16:11:31 19/02/2021 UTC
    Collector Type: Driverless Perf per-process sampling
    CPU
        Name: Intel(R) Xeon(R) Processor 980X series code named Westmere
        Frequency: 3.333 GHz 
        Logical CPU Count: 24
        Cache Allocation Technology
            Level 2 capability: not detected
            Level 3 capability: not detected

If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done

That's the first 100% I've seen. r000ps.tgz is the 3rd run, r001hs-log.tgz is the log directory from the last. Let me know if I should extract anything else. The whole r001hs directory is too big to include.

Kirill_U_Intel · ‎02-19-2021

Looks like Westmere is too old for VTune. I'm not sure that it is still supported.

VTune drivers are not working here but driverless hotspots collection is ok.

I suggest using it and skip performance-snapshot here.

Probably you could try another server machine?

Is it hotspots analysis enough for you? What kind of problem you could catch here?

VTune exception: 'NoneType' object has no attribute '_get_as_query'

Intel VTune™ Profiler