Stopping Analysis From the Command Line

[Marcel_Kornacker]

Hello,

I am trying to use vtune to analyze database perfromance using amplxe-cl. I want to attach vtune to the running databse process. This much seems to work for me using the -target-process=<process name> command. I am trying to get a report for each query, however, and I would like to start vtune before each query and stop it after each query. My problem is that I cannot figure out how to stop amplxe-cl. Whenever I attach it to a process, it only stops running and generates results when that process ends. I can pause in the gui, but I am trying to set up an automated system. Is there a way to stop amplxe from the command line?

Thanks

[David_A_Intel1]

Hi Marcel:

Yes!  See the "-command" option.  From another terminal, execute the command 'amplx-cl -command stop -r <results-dir>', where <results-dir> is the same one used in the command where you attached to the process.

[Marcel_Kornacker]

Thank you for your speedy reply. the stop command works, but I am still running into issues. I am getting this error:

amplxe: Warning: Error 0x40000026 (Database interface error) -- Cannot run data transformation `Compute CPU Usage'.

This error seems to be affecting the ability to generate a report, becuase the report is basically empty. 

Summary
-------
Elapsed Time: 0.000
amplxe: Executing actions 100 % done

I tried researching this error, but couldn't find any post with this exact error. Do you have any insight? Thanks.

[David_A_Intel1]

Can you share with me your exact command line?  You can rename the app, if you would rather not share that info.  I just want to see what options you provided to amplxe-cl.  Thanks.

[Marcel_Kornacker]

In the first terminal I ran the following commands:

[root@a2319 leblang]# source /opt/intel/vtune_amplifier_xe_2013/amplxe-vars.sh
Copyright (C) 2009-2013 Intel Corporation. All rights reserved.
Intel(R) VTune(TM) Amplifier XE 2013 (build 290588)
[root@a2319 leblang]# amplxe-cl -collect hotspots -r /home/leblang/databaseResults -target-process=impalad

I then opened an impala shell (my database) and ran some queries. I then opened a third terminal and ran the following command:

[leblang@a2319 ~]$ amplxe-cl -command stop -r /home/leblang/databaseResults
Collection has stopped.

After I ran that command, this was the output in the first terminal:

amplxe: Using result path `/home/leblang/databaseResults'
amplxe: Executing actions 24 % Processing profile metrics and debug information
amplxe: Warning: Error 0x40000026 (Database interface error) -- Cannot run data transformation `Compute CPU Usage'.
amplxe: Executing actions 50 % Generating a report

Collection and Platform Info
----------------------------
Parameter databaseResults
------------------------ -----------------------------------------------------
Application Command Line
Computer Name a2319.halxg.cloudera.com
Environment Variables
MPI Process Rank
Operating System 2.6.32-220.23.1.el6.x86_64 CentOS release 6.2 (Final)
Result Size 1014093
User Name

CPU
---
Parameter databaseResults
----------------- -----------------------------------------
Frequency
Logical CPU Count 16
Name Intel(R) Xeon(R) / Core i7 980X Processor

Summary
-------
Elapsed Time: 0.000
amplxe: Executing actions 100 % done

My machine is running CentOS 6.2. 

[Peter_W_Intel]

My comments are:

1. You have to use user-account which has permission to access your database process.

2. Some processes work on system mode, you may need to use lightweight-hotspots instead of hotspots 

3. I give an example to work commands on my side, it may help you to dig out root-cause on your side.

[root@snb01 problem_report]# amplxe-cl -collect lightweight-hotspots -target-pid 30403 -r peter001 &
[1] 2502
[root@snb01 problem_report]# amplxe-cl -command pause -r peter001
Collection has paused.
[root@snb01 problem_report]# amplxe-cl -command resume -r peter001
Collection has resumed.
[root@snb01 problem_report]# amplxe-cl -command detach -r peter001
Collection has detached.
amplxe: Using result path `/home/peter/problem_report/peter001'
amplxe: Executing actions 35 % Resolving information for `libc-2.3.4.so'
amplxe: Warning: Cannot locate symbols for file `/opt/intel/vtune_amplifier_xe_2013/lib64/pinruntime/glibc/libc-2.3.4.so'.
amplxe: Executing actions 50 % Generating a report

Result Info
-----------
Parameter peter001
------------------------ ---------------------------------------------------------------------------
Application Command Line
CPU Name Intel(R) Core(TM) Processor 2xxx Series
Computer Name snb01.sh.intel.com
Environment Variables
Frequency 3400000000
Logical CPU Count 8
MPI Process Rank
Operating System 2.6.32-71.el6.x86_64 Red Hat Enterprise Linux Server release 6.0 (Santiago)
Result Size 4739301
User Name root

Summary
-------
Elapsed Time: 37.187
CPU Usage: 0
amplxe: Executing actions 100 % done

[1]+ Done amplxe-cl -collect lightweight-hotspots -target-pid 30403 -r peter001

[root@snb01 problem_report]# amplxe-cl -report hotspots -r peter001
amplxe: Using result path `/home/peter/problem_report/peter001'
amplxe: Executing actions 50 % Generating a report
Function Module CPU Time:Self
----------------------- ------------- -------------
finish_task_switch vmlinux 0.001
__libc_poll libc-2.3.4.so 0.001
_spin_unlock_irqrestore vmlinux 0.001
copy_from_user vmlinux 0.001
dput vmlinux 0.001
estimate_accuracy vmlinux 0.001
hrtimer_init vmlinux 0.001
hrtimer_try_to_cancel vmlinux 0.001
set_normalized_timespec vmlinux 0.001
do_sys_poll vmlinux 0
free_poll_entry vmlinux 0
amplxe: Executing actions 100 % done

[Marcel_Kornacker]

I am running amplxe-cl as root which I know has access to the database process. I tried running lightweight hotspots and got the following errror:

[root@a2319 leblang]# amplxe-cl -collect lightweight-hotspots -r /home/leblang/peterTest -target-process=impalad
amplxe: Error: Cannot start data collection. nmi_watchdog interrupt capability is enabled on your system, which prevents collecting accurate event-based sampling data. Please disable nmi_watchdog interrupt or see the Troubleshooting section of the product documentation for details.
amplxe: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Amplifier on such systems.

Do you think that my original error could be from this BIOS issue? How should I proceed from here? Thanks.

[Marcel_Kornacker]

I changed the settings on nmi_watchdog and was able to get the lightweight hotspots to work. 

[root@a2319 leblang]# echo 0 > /proc/sys/kernel/nmi_watchdog
[root@a2319 leblang]# source /opt/intel/vtune_amplifier_xe_2013/amplxe-vars.sh
Copyright (C) 2009-2013 Intel Corporation. All rights reserved.
Intel(R) VTune(TM) Amplifier XE 2013 (build 290588)
[root@a2319 leblang]# amplxe-cl -collect lightweight-hotspots -r /home/leblang/peterTest -target-process=impalad
amplxe: Warning: On some systems based on the Intel microarchitecture code name Nehalem / Westmere with C-states enabled, this analysis type may cause system hanging due to a known hardware issue (see errata AAJ134 in http://download.intel.com/design/processor/specupdt/320836.pdf). To avoid this situation, disable all "Cn(ACPI Cn) report to OS" BIOS options before sampling with VTune Amplifier on such systems.
amplxe: Using result path `/home/leblang/peterTest'
amplxe: Executing actions 16 % Resolving information for `libstdc++.so.6.0.13'
amplxe: Warning: Cannot locate symbols for file `/usr/lib64/libstdc++.so.6.0.13'.
amplxe: Executing actions 16 % Resolving information for `libc-2.12.so'
amplxe: Warning: Cannot locate symbols for file `/lib64/libc-2.12.so'.
amplxe: Executing actions 17 % Resolving information for `nf_conntrack'
amplxe: Warning: Cannot locate symbols for file `/lib/modules/2.6.32-220.23.1.el6.x86_64/kernel/net/netfilter/nf_conntrack.ko'.
amplxe: Executing actions 17 % Resolving information for `libc-2.3.4.so'
amplxe: Warning: Cannot locate symbols for file `/opt/intel/vtune_amplifier_xe_2013/lib64/pinruntime/glibc/libc-2.3.4.so'.
amplxe: Executing actions 17 % Resolving information for `libm-2.12.so'
amplxe: Warning: Cannot locate symbols for file `/lib64/libm-2.12.so'.
amplxe: Executing actions 18 % Resolving information for `ip_tables'
amplxe: Warning: Cannot locate symbols for file `/lib/modules/2.6.32-220.23.1.el6.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko'.
amplxe: Executing actions 19 % Resolving information for `librt-2.12.so'
amplxe: Warning: Cannot locate symbols for file `/lib64/librt-2.12.so'.
amplxe: Executing actions 19 % Resolving information for dangling locations
amplxe: Warning: Cannot locate file `apwr3_1.ko'.
amplxe: Executing actions 20 % Resolving information for `libpthread-2.12.so'
amplxe: Warning: Cannot locate symbols for file `/lib64/libpthread-2.12.so'.
amplxe: Executing actions 50 % Generating a report

Collection and Platform Info
----------------------------
Parameter peterTest
------------------------ -----------------------------------------------------
Application Command Line
Computer Name a2319.halxg.cloudera.com
Environment Variables
MPI Process Rank
Operating System 2.6.32-220.23.1.el6.x86_64 CentOS release 6.2 (Final)
Result Size 9745428
User Name root

CPU
---
Parameter peterTest
----------------- -----------------------------------------
Frequency 2130000000
Logical CPU Count 16
Name Intel(R) Xeon(R) / Core i7 980X Processor

Summary
-------
Elapsed Time: 50.295
CPU Usage: 0

Event summary
-------------
Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample
----------------------- ------------------------- -------------------------------- -----------------
CPU_CLK_UNHALTED.THREAD 84000000 42 2000000
INST_RETIRED.ANY 18000000 9 2000000
CPU_CLK_UNHALTED.REF_P 8200000 82 100000
amplxe: Executing actions 100 % done

I am still fairly new to vtune and there are a lot of warnings that were generated in this report. Do you expect this to affect my ability to use vtune? 

[Peter_W_Intel]

I'm glad that you can attach to running process basically (after disabling NMI Watchdog Timer). Furthermore you can use "pause", "resume" and "detach" to control data collection, such as I did -

# amplxe-cl -command pause -r peter001
# amplxe-cl -command resume -r peter001
# amplxe-cl -command detach -r peter001

Don't worry about warning message -

1.  C-State issue and asked to disable all "Cn(ACPI Cn) report to OS" in BIOS. This is seldom case, I never met this on NHM machine. Just ignore it.

2.  Can not resolve in some libraries, because these used libraries has no symbol info - all hot addresses will not be located in hot functions, use [libname] instead of function names. However It will not impact on results on your program, it you used "-g" option to build.