Assertion failed in -collect io: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok)

[FantasticMrFox]

I have been running several profilers. All seem to work, threading, hotspot gpu-offload etc. But the `-collect io` option fails with:

AMPLXE_TPSSCOLLECTOR: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) :
Assertion failed: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) : . Please contact the technical support. vtune: Error: Assertion failed: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) : . Please contact the technical support.

I have confirmed that my system is set up to be able to do this by profiling on it, the matrix example:

vtune -collect io -knob kernel-stack=true /opt/intel/oneapi/vtune/latest/samples/en/C++/matrix/matrix -result-dir /data/profiling/matrix_example_io_01

With this command i get the correct data out for the io profile. But when i profile my own application (`rosbag record` or another GPU dependant one), then it fails with the above error. How can i begin debugging why?

[AlekhyaV_Intel]

Hi,

Thank you for posting in Intel Forums. We would like to have some additional information from your side to resolve this issue further.

1) System Information(OS, Version)

2) Intel VTune Profiler version

3) Reproducer and Please attach the application you're trying to profile.

You could also try running below commands to collect hotspots by enabling User-based Sampling. In the user-mode sampling, the collector does not gather system-wide performance data but focuses on your application only.

vtune -collect hotspots -- application.exe

Regards,

Alekhya

[FantasticMrFox]

Hi there,

1) System Information(OS, Version)

Ubuntu 20.04 docker running on a host with ubuntu16.04 with RT kernel patch. 

CPU is Intel(R) Core(TM) i7-8665UE CPU @ 1.70GHz

IO devices include an emmc drive and sd card. 

2) Intel VTune Profiler version

2021.5.0

3) Reproducer and Please attach the application you're trying to profile.

Here is how to reproduce:

1. You will need ROS, specifically rosbag: http://wiki.ros.org/noetic/Installation

2. Install VTUNE, vtune profiler drivers and run the setup for `debugfs` with:

/opt/intel/oneapi/vtune/latest/bin64/prepare-debugfs.sh -g root

3. Run a roscore in a seperate terminal on the system:

roscore

4. Run rosbag with vtune io collect

vtune -collect io rosbag record /hello1

The system will start and run for around 30 seconds with this output:

vtune -collect io rosbag record /hello1
vtune: Peak bandwidth measurement started.
vtune: Peak bandwidth measurement finished.
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /r001io -command stop.
[ INFO] [1626745580.201556938]: Subscribing to /hello1
[ INFO] [1626745580.210198116]: Recording to '2021-07-20-01-46-20.bag'.
AMPLXE_TPSSCOLLECTOR: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) : 
Assertion failed: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) : . Please contact the technical support. vtune: Error: Assertion failed: thread_manager_impl:485: (blocked == tpss_tls_op_err_ok) : . Please contact the technical support. 

I can run with `hotspots`:

vtune -collect hotspots rosbag record /hello1

but this does not gather the io data that i am interested in. I am trying to diagnose what i suspect are hardware contention in writing to the SD card. 

[AlekhyaV_Intel]

Hi,

We are trying to reproduce your issue. We could initialize the ROS environment but we couldn't run the "roscore" command. We're trying to resolve that issue. Meanwhile, could you please provide us your VTune result directory so that we could check the logs and debug your issue further.

Regards,

Alekhya

[AlekhyaV_Intel]

Hi Ben,

Could you please give us an update regarding this issue? We would like you to provide us your VTune result directory so that we can check your logs and debug your issue.

Regards,

Alekhya

[FantasticMrFox]

Hi there,

The issue i have is that this is run on target hardware and has a system wide profile going. I cannot provide this information to an open forum because i think it has debug symbols in our proprietary software. 

I am working on producing a pure reproducable example on the target hardware without our system running. But it has dropped in my priority list. 

[AlekhyaV_Intel]

Hi Ben,

We have requested the reproducer through mail but we didn't get any response from your side. Could you please give us an update regarding this issue?

Regards,

Alekhya

[AlekhyaV_Intel]

Hi,

As we have not heard back from you, we'll close this inquiry now. If you need any further assistance, please post a new question as this thread will no longer be monitored.

Regards,

Alekhya