VTune fails to collect usual data on MIC

[Hans-Christian_S_]

Hello,

I've got troubles with VTune for collecting data on a native MIC application, after we upgraded MPSS to 3.5 and Intel Tools to 2015 update 3.

Here is a log of what happens and the debug logs that VTune created as an attachment.

Probably the key line is:

tb5gen_Add_Sampling_Data returned error "Unknown reason"

There is some data, but no info based on the call stack like "top/down", and the result is therefore not really useful.

Sincerely, HC

-----------------------------------------------------------------------

[root@merlinx01 k1om]# bash micboot_install.sh
SEP configuration files have been successfully installed in the configuration directory.
Please run  "service mpss restart" to start the SEP service.
itt successfully installed.
Please restart mpss service.
_amplxe_vtune_amplifier_xe_2015.3.0.403110 successfully installed.

Enabling prebuild driver installation.
vtsspp driver is ready to install.
vtsspp driver successfully installed for group "500".

Restart Intel Manycore Platform Software Stack (MPSS) to complete installation.
  sudo service mpss restart
[root@merlinx01 k1om]# systemctl restart mpss

[l_stadler_h@merlinx01 ~]$ /nfs/opt/intel/intel-15/vtune_amplifier_xe/bin64/sep -version -mic
Sampling Enabling Product version: 3.15 (private) built by patbbinn on Mar 25 2015 02:44:32
SEP User Mode Version: 3.15.5
mic 0 (merlinx01-mic0.psi.ch): SEP driver version 3.15.5
mic 1 (merlinx01-mic1.psi.ch): SEP driver version 3.15.5
[l_stadler_h@merlinx01 src]$ amplxe-cl -version
Intel(R) VTune(TM) Amplifier XE 2015 Update 3 (build 403110) Command Line Tool
Copyright (C) 2009-2015 Intel Corporation. All rights reserved.
[l_stadler_h@merlinx01 src]$ cat /nfs/home/l_stadler_h/micrun.sh
#!/bin/bash

export LD_LIBRARY_PATH=/nfs/opt/intel/intel-15/lib/mic:/nfs/opt/intel/intel-15/composerxe/lib/mic:/nfs/opt/intel/intel-15/composerxe/mkl/lib/mic

exec $@
[l_stadler_h@merlinx01 src]$ export MSNGR_DEBUG=1
[l_stadler_h@merlinx01 src]$ export EXCHANGE_DEBUG=1
[l_stadler_h@merlinx01 src]$ export AMPLXE_LOG_DIR=~/logs
[l_stadler_h@merlinx01 src]$ amplxe-cl -target-system=mic-native:1 -c advanced-hotspots -duration 30 -follow-child -search-dir=/nfs/home/l_stadler_h/cSAXS/TestCode/src -- /nfs/home/l_stadler_h/micrun.sh /nfs/home/l_stadler_h/cSAXS/TestCode/src/dm-mic /nfs/home/l_stadler_h/cSAXS/TestCode/tst/S01308
amplxe: Using target: mic-native:1
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /nfs/home/l_stadler_h/cSAXS/TestCode/src/r003ah -command stop.
<begin program output>
iterations 30
data measurement
object object_0
object_cols 421
object_rows 420
probe probe_0
probe_cols 192
probe_rows 192
npoints 323
Reading 323 positions...Reading probe...
Reading object...
Initializing thread data...
...
 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
wall_time = 5.27789
Write result probe...
Write result object...
<end program output>
tb5gen_Add_Sampling_Data returned error "Unknown reason"
amplxe: Error: Unknown reason
amplxe: Collection stopped.
amplxe: Using result path `/nfs/home/l_stadler_h/cSAXS/TestCode/src/r003ah'
amplxe: Executing actions  8 % Loading 'userapicollector-5500-a5a64c6d.trace' f
amplxe: Error: Cannot load data file `/nfs/home/l_stadler_h/cSAXS/TestCode/src/r003ah/data.0/tbs125261854_1.tb6_' ().
amplxe: Executing actions 50 % Generating a report                             
Collection and Platform Info
----------------------------
Parameter                 r003ah                                                                                                                              
------------------------  ------------------------------------------------------------------------------------------------------------------------------------
Application Command Line  /nfs/home/l_stadler_h/micrun.sh "/nfs/home/l_stadler_h/cSAXS/TestCode/src/dm-mic" "/nfs/home/l_stadler_h/cSAXS/TestCode/tst/S01308"
Operating System          Linux                                                                                                                               
Computer Name             merlinx01-mic1.psi.ch                                                                                                               
Result Size               1894354                                                                                                                             
Collection start time     09:23:44 07/05/2015 UTC                                                                                                             
Collection stop time      09:23:54 07/05/2015 UTC                                                                                                             

CPU
---
Parameter          r003ah                    
-----------------  --------------------------
Name               Intel Xeon Phi coprocessor
Logical CPU Count  1                         

Summary
-------
Elapsed Time:  9.467
amplxe: Executing actions 100 % done

[l_stadler_h@merlinx01 src]$ micinfo -deviceInfo 1 -group Versions
MicInfo Utility Log
Created Thu May  7 12:15:49 2015

        System Info
                HOST OS                 : Linux
                OS Version              : 3.10.0-229.1.2.el7.x86_64
                Driver Version          : 3.5-1
                MPSS Version            : 3.5

                Host Physical Memory    : 131752 MB

Device No: 1, Device Name: mic1

        Version
                Flash Version            : 2.1.02.0391
                SMC Firmware Version     : 1.17.6900
                SMC Boot Loader Version  : 1.8.4326
                uOS Version              : 2.6.38.8+mpss3.5
                Device Serial Number     : ADKC43600046

[TimP]

That message said you must e.g. sudo service mpss restart subsequent to installing the new Sep driver

[Peter_W_Intel · Accepted solution]

>Elapsed Time:  9.467

I saw that your collection had completed after 9.467s

>amplxe-cl -target-system=mic-native:1 -c advanced-hotspots -duration 30 -follow-child -search-dir=/nfs/home/l_stadler_h/cSAXS/TestCode/src -- /nfs/home/l_stadler_h/micrun.sh /nfs/home/l_stadler_h/cSAXS/TestCode/src/dm-mic /nfs/home/l_stadler_h/cSAXS/TestCode/tst/S01308

Note: ensure that "/nfs/home/l_stadler_h/cSAXS/TestCode/src" is in the host, and "/nfs/home/l_stadler_h/micrun.sh" is no the target.

>There is some data, but no info based on the call stack like "top/down", and the result is therefore not really useful.

You may need to add option "-knob collection-detail=stack-sampling" to profile.

[Hans-Christian_S_]

Thanks for having a look at this.

@Tim: I'm running Centos7, and the command to restart mpss was thus given in the records as 'systemctl restart mpss'

[l_stadler_h@merlinx01 ~]$ lsb_release -d
Description:    CentOS Linux release 7.1.1503 (Core)
 

@Peter: Thanks a lot for solving this issue. This is indeed a very short running process (if the number of iterations is set to be low, that is). Setting the -knob collection-detail=stack-sampling makes the error

tb5gen_Add_Sampling_Data returned error "Unknown reason"

disappear and the stack based sampling data is present.

Thanks a lot!