Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Andrea_F_1
Beginner
99 Views

Linux / Xenomai, VTUNE crashes process / freezes system

[currently posting here since the Intel Premier Support portal does not seem to work for me - I stop at the first step, where I am unable to select my product. Forced to use Internet Explorer 9]

Hi,

I am currently evaluating VTUNE Amplifier XE on a peculiar Linux setup (Gentoo-based) with a Linux 3.10.17 kernel patched with the Xenomai RT framework, release 2.6.4. I am currently looking for a Linux equivalent to the 'SystemViewer' profiler available on VxWorks systems.

I first installed the application on the above kernel; then, upon trying to use the "Advanced Hotspots" analysis type I discovered that I needed to build my own SEP driver. I recompiled another version of the above kernel + RT patch with the flags required by sepdk, and then I built sepdk on it.

When I installed the associated kernel modules for group 'eats', this is what I obtained (everything appeared to work fine):

eats@finx-EATS ~ $ cd /opt/intel/vtune_amplifier_xe/sepdk/src/
eats@finx-EATS /opt/intel/vtune_amplifier_xe/sepdk/src $ sudo ./insmod-sep3 -r -g eats
Warning:  no sep3_15 driver was found loaded in the kernel.
Warning:  no vtsspp driver was found loaded in the kernel.
Attempting to stop PAX service ...
Warning:  no pax driver was found loaded in the kernel.
PAX service has been stopped.
Checking for PMU arbitration service (PAX) ... not detected.
Attempting to start PAX service ...
Executing: insmod ./pax/pax-x32_64-3.10.17-xenomaiVTUNEsmp.ko
Creating /dev/pax device with major number 244 ... done.
Setting group ownership of devices to group "eats" ... done.
Setting file permissions on devices to "666" ... done.
The pax driver has been successfully loaded.
PAX service has been started.
PAX service is accessible to all users.
Executing: insmod ./sep3_15-x32_64-3.10.17-xenomaiVTUNEsmp.ko
Creating /dev/sep3_15 base devices with major number 243 ... done.
Creating /dev/sep3_15 percpu devices with major number 242 ... done.
Setting group ownership of devices to group "eats" ... done.
Setting file permissions on devices to "666" ... done.
The sep3_15 driver has been successfully loaded.
Checking for vtsspp driver ... not detected.
Executing: insmod ./vtsspp/vtsspp-x32_64-3.10.17-xenomaiVTUNEsmp.ko gid=1000 mode=0666
The vtsspp driver has been successfully loaded.

eats@finx-EATS /opt/intel/vtune_amplifier_xe/sepdk/src $ sudo ./boot-script --install -g eats
Configuring sep3_15 boot script with the following options:
    driver files = /opt/intel/vtune_amplifier_xe_2015.4.0.410668/sepdk/src/.
    driver group = eats
Creating boot script /etc/rc.d/init.d/sep3_15 ... done.
Configuring autoload of sep3_15 driver for runlevels 2 through 5 ... done.
eats@finx-EATS /opt/intel/vtune_amplifier_xe/sepdk/src $

 

However, when I configured the tool via GUI to launch / attach to my application process and analyze it by using 'Advanced Hotspots', this is what I got:

IntelVTUNE_Errors_CROPPED.png

In fact, the process under test exits immediately (total execution time is a fraction of second, while it should stay there until an explicit break command) and the above warnings are displayed - although the build and installation phases previously appeared to complete flawlessly. And a few seconds after this message, the system freezes completely and the only option left for me is a cold reboot. I attempted to re-install the .ko modules under the 'root' group, but the outcome was exactly the same.

Ultimately, I would like to ask the following questions:

  1. Do the above warnings reveal that I have done something wrong during the installation?
  2. Is it mandatory to configure the tool for group 'root', and run it as 'root' in any case (i.e. no 'sudo')?
  3. Is 1ms the minimum time granularity that can be configured for sampling? Supposing that we would like to go down to the microsecond, could the tool allow for it?

Please let me know - or if possible, please forward my request to the above-quoted support board which I was unable to use in the first place.

-- Andrea Ferrato

 

0 Kudos
4 Replies
David_A_Intel1
Employee
99 Views

 

Hi Andrea:

In general, real-time kernel support is limited.  This is documented in the release notes:

Linux systems with real time kernel versions are not fully supported (200524374)
Event Based Sampling analysis with stacks and user mode sampling and tracing based analysis types are not supported on systems with real time kernel. Event Based Sampling analysis without stacks are supported if real time kernel is run on a supported Linux distribution.

In regards to your questions:

  1. no, but the first order of business is to get VTune Amplifier to launch and profile the app.  Please use one of the sample apps included in the product, for example, tachyon or matrix.  Untar one of the files into a writable directory and build the app with the included makefile.  Then, create a new project that launches this app and attempt to collect Advance Hotspots.  What happens?
  2. No, the group is irrelevant unless you change the access rights (default is 666, read-writable by everyone).
  3. You can change the sampling interval to 0.1 ms, but by creating a "custom analysis" of Advanced Hotspots, you can change the sample after value of each of the events and get even more samples.  Just be aware, the more samples, the more overhead.  More overhead changes your apps performance and may skew the results.  Also, we limit how low you can go so that you don't hang the system, since sampling too frequently would mean very little real code would get to execute.
99 Views

 

Hi Andrea,

VTune Amplifier can profile real time applications on a supported Linux distributions with the PREEMPT_RT kernel patch applied. Other RTOS variants are not supported. 

Will it be possible for you to apply PREEMPT_RT patch to your kernel instead of Xenomai?

Regards, Katya

 

Andrea_F_1
Beginner
99 Views

Hi MrAnderson, Katya,

 

thanks for your replies.

I tried to build and profile one of the samples pointed out by MrAnderson ('tachyon', specifically), and while the sample compiled and executed flawlessly as standalone, when I tried to run it through VTUNE to obtain an 'Advanced Hotspots' analysis of it, the tool behaved exactly like it did with our application:

IntelVTUNE_SampleCrashing_CROPPED.png

When launched from within the profiling environment, the 'tachyon' sample program failed to execute properly (exited immediately) and no data was collected. And when I tried to execute the test once more, the system became unresponsive again and I had to terminate the VTUNE environment via xkill.

In regards to the suggestion to use PREEMPT_RT instead of Xenomai, it might be a possibility that we can evaluate in the future, but at the moment we are tied to Xenomai because of its 'VxWorks skin' (i.e. an API layer that maps the VxWorks-specific calls in our code to Xenomai's native ones) and we will need a certain amount of time and effort before we are able to fully port our code to, say, the POSIX API.

Moreover, according to this last test it seems that the problem is related more to the setup of VTUNE (and the sep driver build in particular) rather than the patched kernel (also - if I am not mistaken - Red Hat Enterprise Linux 7 runs the same Linux 3.10 kernel series as one of the supported distributions). It might be of relevance to say that before recompiling the sep driver to obtain hardware-based profiling support, VTUNE worked properly - albeit it lacked some of the analysis types we are looking for, which require hardware-based support.

By looking again at the error messages, I tried to verify the actual presence and path of the files cited in the warnings:

  • vtsssp.ko is present in the sepdk/src directory of my VTUNE installation (under /opt/intel/...);
  • vmlinux is present in the /usr/src/linux-3.10.17-xenomaiVTUNE directory (standard location for a kernel compiled on Gentoo Linux).

Would it be possible to instruct VTUNE to look for such files in the correct locations? If so, do you know how that could be done?

 

Thanks again for your time and support!

Andrea Ferrato

 

 

99 Views

Hi Andrea,

We test on regular Red Hat Enterprise Linux 7 and advanced hotspots analysis work there.

Unfortunately your experiment indicates that PMU event based analysis doesn't work on your system (collection stops immidiately and cannot re-run), i.e. hardware event based sampling drivers cannot operate in the presense of Xenomai patch. Pointing the vmlinux location will not help - the warnings are from finalization time - your result may have few samples, but nothing valuable for analysis.  

We did not explore Xenomai case and cannot predict such systems behaviour under PMU based analysis. As far as I know Xenomai  provides a hardware virtualization layer for the Linux kernel via hypervisor, so it doesn't allow VTune drivers to access PMU resources.

You can still use software-based analysis like Basic hotspots, but it maybe too intrusive for real-time applications.

 

 

Just for education purpose: binary location can be specified via 

Dialog Box: Binary/Symbol Search

To access this dialog:

  1. On the Intel® VTune™ Amplifier toolbar, click the  Configure Project button.

    The result tab opens the Choose Target and Analysis Type window with the Analysis Target tab active.

  2. Click the Binary/Symbol Search button on the command toolbar on the right.

 

Regards, Katya

 

Reply