[VTUNE] self-check collection-fail on GPU Computer/Media Hotspots

[Bram_S_]

When running vtune-self-checker.sh:

GPU software event-based analysis with runtime tracing
Example of analysis types: GPU Compute/Media Hotspots (source analysis mode)
Collection: Fail
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

The check observed a product failure on your system.
Review errors in the output above to fix a problem or contact Intel technical support.

The system is ready for the following analyses:
* Performance Snapshot
* Hotspots and Threading with user-mode sampling
* Hotspots with HW event-based sampling, HPC Performance Characterization, etc.
* Microarchitecture Exploration
* Memory Access
* Hotspots with HW event-based sampling and call stacks
* Threading with HW event-based sampling
* GPU HW event-based analysis with runtime tracing

The following analyses have failed on the system:
* GPU Compute/Media Hotspots (source analysis mode)

Log location: /tmp/vtune-tmp-bram/self-checker-2023.04.29_11.18.24/log.txt

Full log is attached.

It just says: "Collection: Fail" with no additional error messages, other than an NMI warning.

This is on:

OS: Ubuntu 23.04

vtune: 2023.1.0 pre-release (build 625246)

CPU: i12600k

GPU: Intel AlderLake-S GT1

[JaideepK_Intel]

Hi,

Thank you for posting in Intel communities.

We assume the matrix.dpcpp(binary, which is used in the self_check.py) sample is unable to recognize the GPU device.

Please follow the below steps.

We assume you installed Basetoolkit. Please set environment variables with the below command.

source /opt/intel/oneapi/setvars.sh

To check available devices, please run the below command

sycl-ls

The output looks like the below screenshot. (you can see GPU details if GPU is available)

As you raised three threads about this issue, can we go ahead and close this thread so we can discuss it there (https://community.intel.com/t5/Analyzers/VTUNE-self-checker-needs-updating-from-SYCL-to-ONEAPI/m-p/1481576#M23364)?

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues. Thank you!

Regards,

Jaideep

[Bram_S_]

I have no sycl-ls, even though intel-basekit-getting-started is installed, as is intel-basekit-runtime-2023.1.0

bram@deca:~$ source /opt/intel/oneapi/setvars.sh 
 
:: initializing oneAPI environment ...
   bash: BASH_VERSION = 5.2.15(1)-release
   args: Using "$@" for setvars.sh arguments: 
:: compiler -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::
 
bram@deca:~$ dpkg --list | grep intel-
ii  intel-basekit-getting-started                    2023.1.0-46401                           all          Intel® oneAPI Base Toolkit
ii  intel-basekit-runtime-2023.1.0                   2023.1.0-46401                           amd64        Intel® oneAPI Base Toolkit
ii  intel-gpu-tools                                  1.27.1-1                                 amd64        tools for debugging the Intel graphics driver
ii  intel-media-va-driver:amd64                      23.1.2+dfsg1-1                           amd64        VAAPI driver for the Intel GEN8+ Graphics family
ii  intel-microcode                                  3.20230214.0ubuntu1                      amd64        Processor microcode firmware for Intel CPUs
ii  intel-oneapi-common-licensing-2023.1.0           2023.1.0-43473                           all          oneAPI Common License
ii  intel-oneapi-common-vars                         2023.1.0-43473                           all          oneAPI Common Environment Scripts
ii  intel-oneapi-compiler-cpp-eclipse-cfg            2023.1.0-46305                           all          Intel® oneAPI DPC++/C++ Compiler 2023.1.0 for Linux* eclipse integration configuration file (C++)
ii  intel-oneapi-compiler-dpcpp-cpp-common-2023.1.0  2023.1.0-46305                           all          Intel® oneAPI DPC++/C++ Compiler 2023.1.0 for Linux*
ii  intel-oneapi-compiler-dpcpp-cpp-runtime          2023.1.0-46305                           amd64        Intel® oneAPI DPC++/C++ Compiler 2023.1.0 for Linux* runtime package for Intel(R) 64
ii  intel-oneapi-compiler-dpcpp-cpp-runtime-2023.1.0 2023.1.0-46305                           amd64        Intel® oneAPI DPC++/C++ Compiler 2023.1.0 for Linux* runtime package for Intel(R) 64
ii  intel-oneapi-compiler-dpcpp-eclipse-cfg          2023.1.0-46305                           all          Intel® oneAPI DPC++/C++ Compiler 2023.1.0 for Linux* eclipse integration configuration file (DPC++)
ii  intel-oneapi-compiler-shared-runtime-2023.1.0    2023.1.0-46305                           amd64        Intel(R) Compiler Shared Files runtime contents
ii  intel-oneapi-condaindex                          2023.1.0-43291                           amd64        oneAPI Common Install components
ii  intel-oneapi-dev-utilities-2021.9.0              2021.9.0-44447                           amd64        Dev Utilities
ii  intel-oneapi-dev-utilities-eclipse-cfg           2021.9.0-44447                           all          intel-oneapi-dev-utilities-eclipse-cfg
ii  intel-oneapi-dnnl                                2023.1.0-46343                           amd64        Intel® oneAPI Deep Neural Network Library
ii  intel-oneapi-icc-eclipse-plugin-cpp-2023.1.0     2023.1.0-46305                           all          Standards driven high performance cross architecture compiler and high performance C++ CPU focused compiler
ii  intel-oneapi-openmp-2023.1.0                     2023.1.0-46305                           amd64        Intel® OpenMP* Runtime Library 2023.1.0 for Linux* for Intel(R) 64
ii  intel-oneapi-openmp-common-2023.1.0              2023.1.0-46305                           all          Intel® OpenMP* Runtime Library 2023.1.0 for Linux*
ii  intel-oneapi-runtime-opencl                      2023.1.0-46305                           amd64        Intel® CPU Runtime for OpenCL(TM) Applications runtime
ii  intel-oneapi-runtime-tbb                         2021.9.0-43484                           amd64        Intel® oneAPI Threading Building Blocks runtime
ii  intel-oneapi-runtime-tbb-common                  2021.9.0-43484                           all          Intel® oneAPI Threading Building Blocks runtime common
ii  intel-oneapi-tbb-2021.9.0                        2021.9.0-43484                           amd64        Intel® oneAPI Threading Building Blocks
ii  intel-oneapi-tbb-common-2021.9.0                 2021.9.0-43484                           all          Intel® oneAPI Threading Building Blocks common
ii  intel-oneapi-vtune                               2023.1.0-44286                           amd64        Intel® VTune(TM) Profiler
ii  intel-oneapi-vtune-eclipse-plugin-vtune          2023.1.0-44286                           all          Locate and optimize performance bottlenecks fast across CPU, GPU, and FPGA systems.
ii  intel-opencl-icd                                 22.43.24595.41-1                         amd64        Intel graphics compute runtime for OpenCL
ii  libdrm-intel1:amd64                              2.4.115+git2302101558.332809f3~j~mesarc0 amd64        Userspace interface to intel-specific kernel DRM services -- runtime
bram@deca:~$ sycl-ls
sycl-ls: command not found

[JaideepK_Intel]

Hi,

Can you install Basetoolkit with the below command and let us know the output of sycl-ls.

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/7deeaac4-f605-4bcf-a81b-ea7531577c61/l_BaseKit_p_2023.1.0.46401_offline.sh

sycl-ls

Thanks,

Jaideep

[Bram_S_]

Thank you JaideepK,

Can you please tell me in which .deb package sycl-ls is located?

I am usin the intel package repository, and I prefer to install via packages, not via an .sh script.

$ cat /etc/apt/sources.list.d/oneAPI.list
deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main

[Bram_S_ · Accepted solution]

Sorry for the duplicate post, the forum somehow lost my first one after adding a label, so I redid it.

The solution to this selfcheck fail is to preload libpthread, as described here.

[JaideepK_Intel]

Hi,

I hope you are doing well.

Sorry to say this, but the OS that you are using is not supported for Vtune. Could you please try on a supported operating system?

If you want to try Ubuntu 22, please follow the documentation below.

https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-arc.html

You can directly profile any GPU binary with the below command:

 example: vtune -collect gpu-hotspots <Path to the GPU Binary>

Thanks,

Jaideep

[Bram_S_]

Thank you Jaideep,

So, downgrading the OS is quite involved, so I will skip that. However, using 

/opt/intel/oneapi/diagnostics/diagnostics.py --force --filter vtune_sys_check -vv

I was able to learn that:

CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS---------------------Disable or not set

I am currently following your guide on kernel recompilation and will try doing that. If that fails too, I will give up.

UPDATE: After building kernel with correct configuration, the diagnostics error went away, but still no gpu-hotspot samples recorded. 

was disabled because it doesn't support the given collection type.

GPU device is seen, though:

# sycl-ls 
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.15.3.0.20_160000]
[opencl:cpu:1] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i5-12600K 3.0 [2023.15.3.0.20_160000]
[opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) UHD Graphics 770 [0x4680] 3.0 [22.43.24595]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) UHD Graphics 770 [0x4680] 1.3 [1.3.24595]

  Bram

[JaideepK_Intel]

Hi,

It's good to see that sycl-ls is showing GPU devices. Now you can run Vtune with any OneAPI GPU samples.

Please ignore the self-checker log. I think your machine is ready to go.

If you need any API samples, you can download them with the below command (you need to source environment variables).

oneapi-cli

If your issue is resolved, Can we go ahead and close this case?

Thanks,

Jaideep

[Bram_S_]

Nope... the matrix sample is still not working.

Finalization completed with warnings

May 05 2023 08:22:28 Result finalization has completed with warnings that may affect the representation of the analysis data. Please see details below.

Cannot load data file `/home/bram/intel/vtune/projects/sample (matrix)/r002gh/data.0/5449-5455.0.trace' (Data file is corrupted).
Cannot locate debugging information for file `/lib/modules/6.2.6/kernel/drivers/nvme/host/nvme.ko'.
Cannot read load addresses of sections from `/sys/module/nvme/sections'. This may affect the correctness of symbol resolution for `nvme'. Make sure this directory exists and all files in this directory have read permissions.
Cannot locate debugging information for file `/home/intel/oneapi/vtune/2023.1.0/lib64/libtpsstool.so'.
Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.

[JaideepK_Intel]

Hi,

I hope you are doing well.

We were able to run the Vtune self-checker without any issues after following the below workaround.

Could you please run the below commands as sudo/root user?

Setting the environment variables.

source /opt/intel/oneapi/setvars.sh

Installing dependencies (libdrm)

sudo apt install libdrm-dev
sudo apt-get update

Cloning the repo and building packages.

git clone https://github.com/intel/metrics-discovery.git
cd metrics-discovery
mkdir build && cd build
cmake ..

make -j$(nproc)
sudo make install
make package

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

Have a great day ahead

Thanks,

Jaideep

[Bram_S_]

You can close this,

i had already installed that metrics from git, which did not solve it.

I am on unsupported 23.04 so you can close this issue.

[JaideepK_Intel]

Hi,

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks,

Jaideep