Issue while collecting hotspot profile with vtune 2016 beta

[Pramod_K_]

Hello All,

I have searched forum but didn't find any relevant solution for below error:

We have installed parallel studio 2016 beta version. All components (compilers, advisor etc) works but I get following error while using Vtune:

$ source /opt/intel/vtune_amplifier_xe_2016/amplxe-vars.sh
Copyright (C) 2009-2015 Intel Corporation. All rights reserved.
Intel(R) VTune(TM) Amplifier XE 2016 (build 402940)

kumbhar@knc1:~$ amplxe-cl -collect advanced-hotspots --target-system=mic-native:2 -- `pwd`/matrix/linux/matrix.mic
amplxe: Using target: mic-native:2
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/kumbhar/workarena/systems/jknc/repos/bbp/coreneuron/paper/results/vtune_manual/imp_host_mic/r000ah -command stop.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Addr of buf1 = 0x7f99d37b8010
..........
Using multiply kernel: multiply1
Execution time = 24.017 seconds
tb5gen_Add_Sampling_Data returned error "Unknown reason"
amplxe: Error: Unknown reason
amplxe: Collection stopped.
amplxe: Using result path `/home/kumbhar/workarena/systems/jknc/repos/bbp/coreneuron/paper/results/vtune_manual/imp_host_mic/r000ah'
amplxe: Executing actions  8 % Clearing the database
amplxe: Error: Cannot load data file `/home/kumbhar/workarena/systems/jknc/repos/bbp/coreneuron/paper/results/vtune_manual/imp_host_mic/r000ah/data.0/tbs2103234249_2.tb6_' ().
amplxe: Executing actions 50 % done
amplxe: Error: 0x4000001e (Cannot load raw collector data)

We also have Vtune 2015 version installed which works fine with above steps. Any hints to address above error with 2016 beta version?

Thanks!

[Peter_W_Intel]

I cannot reproduce this problem after installing the 2016 beta.

# amplxe-cl -version
Intel(R) VTune(TM) Amplifier XE 2016 Beta (build 402940) Command Line Tool
Copyright (C) 2009-2015 Intel Corporation. All rights reserved.

# ssh mic0 lsmod | grep sep
sep3_15                45337  2 

# amplxe-cl -target-system=mic-native:0 -c advanced-hotspots -d 5
amplxe: Using target: mic-native:0
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/peter/r000ah -command stop.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
amplxe: Collection detached.
amplxe: Collection stopped.
amplxe: Using result path `/home/peter/r000ah'

....
 

1. Was it possible that you had no enough space to restore results?

2. Was it possible that your application should be executed by other users? Can you run my CLI to verify issue to know if it is an application specific issue?

3. Can you please check if some temp contents of VTune under /tmp, such as amplxe-log-$user and amplxe-tmp-$user, which may be generated on last time, but it impacts on your result this time?

[Pramod_K_]

Thanks Peter! This is what I tried now:

1. Does application works natively? (simple hello_world with some math ops)

$ ssh mic0 "~/hello_mic"
0.000000        1.000000        1.414214        1.732051        2.000000        2.236068        2.449490        2.645751        2.828427   3.000000        3.162278        3.316625        3.464102        3.605551        3.741657        3.872983        …………………… 899495   9.949874

2. Which version? Driver loaded?

kumbhar@knc1:~$ amplxe-cl -version
Intel(R) VTune(TM) Amplifier XE 2016 Beta (build 402940) Command Line Tool
Copyright (C) 2009-2015 Intel Corporation. All rights reserved.

kumbhar@knc1:~$ ssh mic0 lsmod | grep sep
sep3_15                45337  0

3. Remove any temporary data files:

$ rm -rf /tmp/amplxe-tmp-${USER}
$ rm -rf /tmp/amplxe-log-${USER}

4. You command (without application) works fine :

$ amplxe-cl -target-system=mic-native:0 -c advanced-hotspots -d 5
amplxe: Using target: mic-native:0
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/kumbhar/r005ah -command stop.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
amplxe: Collection detached.
amplxe: Collection stopped.
amplxe: Using result path `/home/kumbhar/r005ah'
amplxe: Executing actions 16 % Resolving module symbols
amplxe: Warning: Cannot locate file `intel_micveth.ko'.
amplxe: Executing actions 16 % Resolving information for `intel_micveth'
amplxe: Locating file `/usr/lib/debug/lib/modules/2.6.38.8+mpss3.2.1/vmlinux' on the remote system
amplxe: Executing actions 17 % Resolving information for `intel_micveth'
amplxe: Locating file `/boot/vmlinuz-2.6.38.8+mpss3.2.1' on the remote system
amplxe: Locating file `/boot/vmlinuz' on the remote system
amplxe: Warning: Cannot locate file `/usr/lib/debug/lib/modules/2.6.38.8+mpss3.2.1/vmlinux'.
amplxe: Executing actions 50 % Generating a report
General Exploration Metrics
---------------------------
Parameter             r005ah
--------------------  --------------
Clockticks            1660284054.000
Instructions Retired  115142742
……………………

5. But if I try application then I get an error:

$ amplxe-cl -collect advanced-hotspots --target-system=mic-native:2 -- ~/hello_mic
amplxe: Using target: mic-native:2
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/kumbhar/r007ah -command stop.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
         0.000000        1.000000        1.414214        1.732051        2.000000        2.236068        2.449490        2.645751        2.828427   3.000000   ...........   9.949874

tb5gen_Add_Sampling_Data returned error "Unknown reason"
amplxe: Error: Unknown reason
amplxe: Collection stopped.
amplxe: Using result path `/home/kumbhar/r007ah'
amplxe: Executing actions  8 % Clearing the database
amplxe: Error: Cannot load data file `/home/kumbhar/r007ah/data.0/tbs324865693_2.tb6_' ().
amplxe: Executing actions 50 % done
amplxe: Error: 0x4000001e (Cannot load raw collector data)

6. If I change Vtune version to 2015, then it works again:

$ source /opt/intel/vtune_amplifier_xe_2015/amplxe-vars.sh
Copyright (C) 2009-2014 Intel Corporation. All rights reserved.
Intel(R) VTune(TM) Amplifier XE 2015 (build 380310)

kumbhar@knc1:~$ amplxe-cl -collect advanced-hotspots --target-system=mic-native:2 -- ~/hello_mic
amplxe: Using target: mic-native:2
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/kumbhar/r008ah -command stop.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
         0.000000        1.000000        1.414214        1.732051        2.000000        2.236068        2.449490        2.645751    ...............   9.949874

amplxe: Collection stopped.
amplxe: Using result path `/home/kumbhar/r008ah'
amplxe: Executing actions 16 % Resolving module symbols
amplxe: Locating file `/lib64/ld-2.14.90.so' on the remote system
amplxe: Locating file `/usr/sbin/sshd' on the remote system
amplxe: Locating file `intel_micveth.ko' on the remote system
amplxe: Locating file `/lib64/libc-2.14.90.so' on the remote system
amplxe: Locating file `/lib64/libpthread-2.14.90.so' on the remote system

Any suggestion?

[Peter_W_Intel]

Was it due to your application running shortly? 

I built a simple hello case and VTune had no problem to run -

# amplxe-cl -target-system=mic-native:0 -c advanced-hotspots -search-dir=. -- /root/hello.MIC

Would you please run VTune with long run app? If you still have this problem, is it possible to attach vtune result dir?