VTune 2023.1 (build 625246) fails for most analysis types with Intel Sampling Driver

[paulalesius]

VTune: 23.1.0 pre-release (build 625246)

Kernel: 6.3.9

Host: Fedora 38

Container with oneAPI: Ubuntu 22.04.2

CPU: 12th gen

I've built a custom container based on Ubuntu 22.04.2 with Intel BaseKit 2023.1.0. The Intel Sampling Driver builds and loads, seemingly fine, in the host kernel. Most analysis types fail in the container and there are BUG stack dumps in dmesg in the host after the execution of vtune-self-checker in the container.

I've provided logs for:

host: sep5 service

host: dmesg.log

container: vtune-self-checker.sh

container: sycl-ls -verbose

I'm out of ideas. The analysis types work using driverless Perf mode. (I've not attached any logs for that)

With regards,

-----

The system is ready for the following analyses:
* Performance Snapshot
* Hotspots and Threading with user-mode sampling
* GPU HW event-based analysis with runtime tracing
* GPU software event-based analysis with runtime tracing

The following analyses have failed on the system:
* Hotspots with HW event-based sampling, HPC Performance Characterization, etc.
* Microarchitecture Exploration
* Memory Access
* Hotspots with HW event-based sampling and call stacks
* Threading with HW event-based sampling

[RajashekarK_Intel]

Hi, 

Thanks for posting in Intel communities.

We are working on this internally to replicate your issue and will get back to you soon with an update.

Meanwhile can you work on below suggested steps to make us understand your issue even better.

1. Launch your docker instance as root user and run VTune with your analysis, and check if you are still facing issue.
2. as Root user, Run the ./insmod-sep script under "/opt/intel/oneapi/vtune/latest/sepdk/src"
   a) ./insmod-sep -q # to check if the drivers are loaded properly
   b) If it fails, run ./build-driver
   c) ./insmod-sep -r #reload all the drivers
   d) ./insmod-sep -q #again to check if they are properly loaded.
   e) If at any of the stage you find errors. please send us the log as you sent previously, else let us know if you are still facing issue while analysis.
   f) It should look something like this.
   

   $>/opt/intel/oneapi/vtune/latest/sepdk/src# ./insmod-sep -q
   pax driver is loaded and owned by group "vtune" with file permissions "660".
   socperf3 driver is loaded and owned by group "vtune" with file permissions "660".
   sep5 driver is loaded and owned by group "vtune" with file permissions "660".
   
   socwatch2_15 driver is loaded and owned by group "vtune" with file permissions "660".
   
   vtsspp driver is loaded and owned by group "vtune" with file permissions "660".
   ​

Regards,

Rajashekar

[paulalesius]

Thank you for the reply.

The drivers all build and load fine using Kernel version 6.3.9, the drivers fail to build using later kernels.

I am using podman for containers, which is Docker-compatible. The podman container is run like this:

sudo podman run \
  --cap-add=all \
  --privileged \
  --tmpfs /tmp \
  --network=host \
  -it --rm \
  --ipc=host \
  --group-add keep-groups \
  -e DISPLAY=unix:0.0 \
  -e XDG_RUNTIME_DIR=/tmp \
  -e WAYLAND_DISPLAY \
  -e XDG_SESSION_TYPE=wayland \
  --device /dev/dri:/dev/dri:rwm \

  --device /dev/sep5:/dev/sep5:rwm \
  --device /dev/socperf3:/dev/socperf3:rwm \
  --device /dev/pax:/dev/pax:rwm \
  --mount type=bind,src=/sys/kernel/debug,dst=/sys/kernel/debug/ \
  --volume /tmp/.X11-unix/:/tmp/.X11-unix/:rw \
  --volume $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY \
  --volume $PROJECTDIR:$PROJECTDIR \
  --volume $SELFDIR/store/intel:/root/intel \
  --volume $SELFDIR/store/dotintel:/root/.intel \
  --volume /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
  my-container-name \
  /bin/bash -l /vtune.sh gui

bash -l is necessary to run the command in a login shell which runs $HOME/.bash_profile which contains "source /opt/intel/oneapi/setvars.sh" inside the container.

The host only loads the Intel Sampling Drivers and they should really be provided separately in source form.

[RajashekarK_Intel]

Hi,

Thanks for the information, we're trying to replicate your issue and will get back to you once we have an update.

As you mentioned

>> "the drivers fail to build using later kernels."

Can you provide more info on this?

FYI, The systems requirements for VTune have support to below mentioned kernels.

VTune system requirements: https://www.intel.com/content/www/us/en/developer/articles/system-requirements/vtune-profiler-system-requirements.html 

References: https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/profiling-in-docker-container.html 

The launch requires you to definitely add below flags, but it seems you've already added them.

--cap-add CAP_SYS_ADMIN
--privileged

Regards,

Rajashekar

[RajashekarK_Intel · Accepted solution]

Hi, Good day to you.

We have not heard back from you. Is your issue resolved with the above information? If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

Regards,

Rajashekar

[paulalesius]

The driver fails to build with the following error using the updated vtune from l_BaseKit_p_2023.2.0.49397.sh and kernel version 6.4.4:

---

[noname@devbox src]$ sudo bash build-driver

C compiler to use: [ /bin/gcc ]
C compiler version: 13.1.1

Make command to use: [ /bin/make ]
Make version: 4.4

Kernel source directory: [ /lib/modules/6.4.4-devbox/build ]
Kernel version: 6.4.4-devbox

Cleaning workspaces ...
Done

Building socperf driver ...
In file included from ./include/linux/linkage.h:7,
from ./arch/x86/include/asm/cache.h:5,
from ./include/linux/cache.h:6,
from ./include/linux/time.h:5,
from ./include/linux/stat.h:19,
from ./include/linux/module.h:13,
from /opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.c:61:
/opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.c: In function ‘socperf_Load’:
./include/linux/export.h:27:22: error: passing argument 1 of ‘class_create’ from incompatible pointer type [-Werror=incompatible-pointer-types]
27 | #define THIS_MODULE (&__this_module)
| ~^~~~~~~~~~~~~~~
| |
| struct module *
/opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.c:1738:34: note: in expansion of macro ‘THIS_MODULE’
1738 | pmu_class = class_create(THIS_MODULE, SOCPERF_DRIVER_NAME);
| ^~~~~~~~~~~
In file included from ./include/linux/device.h:31,
from ./include/linux/cdev.h:8,
from /opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.c:69:
./include/linux/device/class.h:230:54: note: expected ‘const char *’ but argument is of type ‘struct module *’
230 | struct class * __must_check class_create(const char *name);
| ~~~~~~~~~~~~^~~~
/opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.c:1738:21: error: too many arguments to function ‘class_create’
1738 | pmu_class = class_create(THIS_MODULE, SOCPERF_DRIVER_NAME);
| ^~~~~~~~~~~~
./include/linux/device/class.h:230:29: note: declared here
230 | struct class * __must_check class_create(const char *name);
| ^~~~~~~~~~~~
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:252: /opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src/socperfdrv.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:2026: /opt/intel/oneapi/vtune/2023.2.0/sepdk/src/socperf/src] Error 2
make[1]: *** [Makefile:145: default] Error 2
make: *** [Makefile:215: default] Error 2

Failed to build the drivers

---

I understand that I am using an unsupported kernel version, I will have to remove the oneAPI libraries and drivers, so I am unable to test this issue further on supported kernel versions.

Thank you for the help

[RajashekarK_Intel]

Hi,

Thanks for accepting it as solution and letting us know the cause of it.

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Rajashekar