topic No Microarchitectural Profiling Data Available in Analyzers

No Microarchitectural Profiling Data Available

Huzaifa — Tue, 20 Oct 2020 02:45:35 GMT

Hello,

I am only able to get user mode hotspot profiling to work. I am unable to get microarchitectural information of any sort from any kind of application. I've tried runs from several seconds to a full minute, but in each run no data is collected and the tool reports 100% of cycles are spent retiring instructions and none on stalls. I know that is wrong because the same workloads are profiled correctly by VTune on other machines (with different hardware and OSes).

The closest issue I could find related to mine is this one, but it hasn't been addressed. When I run vtune-self-checker, it reports this:

HW event-based analysis check (Intel driver)
Example of analysis types: Hotspots with knob sampling-mode=hw, HPC Performance Characterization, etc.
Collection: Ok
Finalization: Ok...
Report: Fail
vtune: Error: 0x40000024 (No data) -- No data is collected. Possible reasons:

The interesting thing is that VTune used to run fine when this machine had Ubuntu 16.04. With 18.04, it doesn't work.

Setup details:

Xeon E-2236

32 GB DDR4

RTX 2080 with 455.23.04 drivers

Ubuntu 18.04

VTune 2020 Update 2

Re:No Microarchitectural Profiling Data Available

AthiraM_Intel — Tue, 20 Oct 2020 13:15:23 GMT

Hi,

Thanks for reaching out to us.

Could you please provide the complete error log ?

Also please make sure that the driver is installed correctly.

To verify that the sampling driver is installed correctly on the host Linux system, please follow the below commands:

cd <install-dir>/sepdk/src

./insmod-sep -q

Thanks

Re: Re:No Microarchitectural Profiling Data Available

Huzaifa — Tue, 20 Oct 2020 16:01:34 GMT

Thank you for getting back to me.

Self checker result:

$ ./vtune-self-checker.sh
Intel(R) VTune(TM) Profiler Self Check Utility
Copyright (C) 2009-2019 Intel Corporation. All rights reserved.
Build Number: 610396

Instrumentation based analysis check
Example of analysis types: Hotspots with default knob sampling-mode=sw, Threading with default knob sampling-and-waits=sw
Collection: Ok
Finalization: Ok...
Report: Ok

HW event-based analysis check (Intel driver)
Example of analysis types: Hotspots with knob sampling-mode=hw, HPC Performance Characterization, etc.
Collection: Ok
Finalization: Ok...
Report: Fail
vtune: Error: 0x40000024 (No data) -- No data is collected. Possible reasons:

HW event-based analysis check (Intel driver)
Example of analysis types: Microarchitecture Exploration
Collection: Ok
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Fail

HW event-based analysis with uncore events (Intel driver)
Example of analysis types: Memory Access
Collection: Ok
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Fail

HW event-based analysis with stacks (Perf)
Example of analysis types: Hotspots with knob sampling-mode=hw and knob enable-stack-collection=true, etc.
Collection: Ok
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

HW event-based analysis with context switches (Intel driver)
Example of analysis types: Threading with knob sampling-and-waits=hw
Collection: Ok
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

The check observed a product failure on your system.
Review errors in the output above to fix a problem or contact Intel technical support.

Log location: /tmp/vtune-tmp-huzaifa2/self-checker-2020.10.20_10.38.46/log.txt

Sampling driver status:

$ ./insmod-sep -q
pax driver is loaded and owned by group "vtune" with file permissions "660".
socperf3 driver is loaded and owned by group "vtune" with file permissions "660".
sep5 driver is loaded and owned by group "vtune" with file permissions "660".
socwatch driver is loaded.
vtsspp driver is loaded and owned by group "vtune" with file permissions "660".

I have made sure that I am part of the vtune group.

One other thing that I would like to add is the performance counters on the machine in general are working fine. I am able to query them using perf.

Finally, this is what the output inside VTune looks like.

Re: Re:No Microarchitectural Profiling Data Available

Kirill_U_Intel — Wed, 21 Oct 2020 07:33:23 GMT

Hi.

Could you share some logs and files?

'VTUNE_INSTALL_DIR/bin64/sep -version' output
The simplest driver collection 'VTUNE_INSTALL_DIR/bin64/sep -start -d 3 -out test.tb7'. Pls, share test.tb7
Could you share *.tb7 file from the r004ue/data.0 result dir?

You mentioned that perf is ok. Could you try driverless collection based on perf to collect data? just remove intel drivers './rmmod-sep'

Kirill

Re: Re:No Microarchitectural Profiling Data Available

Huzaifa — Wed, 21 Oct 2020 13:39:50 GMT

Hi Kirill,

SEP version:

$ ./sep -version
Sampling Enabling Product version: 5.19 Beta built on Jun 24 2020 16:46:20
SEP User Mode Version: 5.19 Beta
SEP Driver Version: 5.19 Beta (public)
PAX Driver Version: 1.0
Platform type: 122
CPU name: Intel(R) microarchitecture code named Coffeelake
PMU: skylake
Driver configs: Non-Maskable Interrupt, REGISTER CHECK ON
Copyright(C) 2007-2019 Intel Corporation. All rights reserved.

Driver collection:

$ ./sep -start -d 3 -out test.tb7
A tbX file was not created because 0 samples were collected

The r004ue tb7 file is 325 MB in size and the attachment size limit on the forum is 71 MB. Is there another experiment that I could run to help you debug this?

Perf without SEP driver:

$ ./rmmod-sep
Removing sep5 driver from the kernel ... done.
The sep5 driver has been successfully unloaded.
Removing vtsspp driver from the kernel ... done.
The vtsspp driver has been successfully unloaded.
Removing socwatch2_12 driver from the kernel ... done.
The socwatch2_12 driver has been successfully unloaded.
Removing socperf3 driver from the kernel ... done.
The socperf3 driver has been successfully unloaded.
No driver needs PMU arbitration anymore: the PAX service can be safely removed.
Attempting to stop PAX service ...
Removing pax driver from the kernel ... done.
The pax driver has been successfully unloaded.
PAX service has been stopped.
$ perf stat -d -- solo.opt 1000 decode
Getting second/third order dodecahedron

Performance counter stats for 'solo.opt 1000 decode':

1,008.49 msec task-clock # 0.999 CPUs utilized
5 context-switches # 0.005 K/sec
0 cpu-migrations # 0.000 K/sec
867 page-faults # 0.860 K/sec
4,732,717,922 cycles # 4.693 GHz (61.93%)
16,226,535,324 instructions # 3.43 insn per cycle (74.62%)
716,489,660 branches # 710.460 M/sec (74.95%)
2,906,156 branch-misses # 0.41% of all branches (75.34%)
3,710,857,176 L1-dcache-loads # 3679.626 M/sec (75.41%)
94,742,412 L1-dcache-load-misses # 2.55% of all L1-dcache hits (75.41%)
1,383,042 LLC-loads # 1.371 M/sec (49.65%)
4,739 LLC-load-misses # 0.34% of all LL-cache hits (49.25%)

1.009175110 seconds time elapsed

1.004736000 seconds user
0.004002000 seconds sys

Thank you, and please let me know if you need more information.

Re: Re:No Microarchitectural Profiling Data Available

Kirill_U_Intel — Wed, 21 Oct 2020 13:43:12 GMT

Could you try VTune collection without drivers? In general, it will use perf in this case. Is result ok or empty counters?

Kirill

Re: Re:No Microarchitectural Profiling Data Available

Huzaifa — Thu, 22 Oct 2020 00:09:16 GMT

When I tried running microarchitectural profiling without the drivers, VTune crashed my entire machine. After the reboot, microarchitectural profiling actually worked! However, the driver crashed again in the middle of a new profiling run, and I got the same results as before (no data collected). After that, I did rmmod-sep and insmod-sep, and so far things have been working fine. It is very flaky (I don't know what causes the crashes), but at least there is a workaround now. vtune-self-checker.sh reports no errors either.

I happened to look at dmesg when the driver was having issues, and noticed that there were several errors reported by sep. Sharing them below in case they are helpful.

Thank you for all your help!

dmesg output:

[ 130.716042] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716044] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716045] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716045] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716046] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716047] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716047] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716048] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716048] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 130.716049] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 141.656348] NOHZ: local_softirq_pending 40
[ 142.267110] NOHZ: local_softirq_pending 40
[ 142.801054] NOHZ: local_softirq_pending 40
[ 142.801098] NOHZ: local_softirq_pending 40
[ 142.801152] NOHZ: local_softirq_pending 40
[ 142.994534] NOHZ: local_softirq_pending 40
[ 143.132770] NOHZ: local_softirq_pending 40
[ 143.132815] NOHZ: local_softirq_pending 40
[ 143.132861] NOHZ: local_softirq_pending 40
[ 143.132906] NOHZ: local_softirq_pending 40
[ 1619.625038] usb 1-11: reset high-speed USB device number 3 using xhci_hcd
[ 1838.904775] perf: interrupt took too long (2597 > 2500), lowering kernel.perf_event_max_sample_rate to 77000
[ 1838.915264] perf: interrupt took too long (3472 > 3246), lowering kernel.perf_event_max_sample_rate to 57500
[ 1838.932847] perf: interrupt took too long (4384 > 4340), lowering kernel.perf_event_max_sample_rate to 45500
[ 1838.954553] perf: interrupt took too long (5628 > 5480), lowering kernel.perf_event_max_sample_rate to 35500
[ 1839.103847] perf: interrupt took too long (7042 > 7035), lowering kernel.perf_event_max_sample_rate to 28250
[ 1839.426246] perf: interrupt took too long (8923 > 8802), lowering kernel.perf_event_max_sample_rate to 22250
[ 1839.442595] perf: interrupt took too long (11215 > 11153), lowering kernel.perf_event_max_sample_rate to 17750
[ 2022.550436] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550437] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550438] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550439] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550440] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550440] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550441] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550441] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550442] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2022.550443] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255951] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255952] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255953] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255954] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255954] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255955] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255955] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255956] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255957] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.
[ 2979.255957] sep5_19: [warning] [CONTROL_Allocate_Memory@691]: Cannot allocate a number of bytes <= 0.

Re:No Microarchitectural Profiling Data Available

AthiraM_Intel — Wed, 28 Oct 2020 07:44:34 GMT

Hi,

Hope your issue is resolved. Could you please confirm whether we can close this thread?

Thanks.

Re: Re:No Microarchitectural Profiling Data Available

Huzaifa — Wed, 28 Oct 2020 19:28:42 GMT

I had marked the solution hoping that that would close the thread. But yeah, please go ahead. Thank you!

Re:No Microarchitectural Profiling Data Available

AthiraM_Intel — Thu, 29 Oct 2020 04:53:57 GMT

Hi,

Thanks for the confirmation. We would discontinue monitoring this issue, please raise a new thread if you have further issues.

Thanks