cannot find ssh

[jimdempseyatthecove]

Setting up first VTune Amplifier XE 2013 project for remote collection on Xeon Phi (MIC)

In the Target, the application is supposed to be "ssh" or "sudo ssh", and the command line arguments (to ssh) is to be the remote identity and file name (on MIC) such as: "mic0 /tmp/foobar"

The first problem is when using "ssh" alone, amplifier pre-pends the project directory?!!! (home/this/that/other/ssh)
"/opt/ssh" didn't work either
I can browse for file, browse to the /opt folder, but then ssh is hidden. (as if ssh not valid "program")

Any suggestions?

Jim Dempsey

[jimdempseyatthecove]

Apparently the way is to set the application to "sudo" and move "ssh" to the command line.

Moving on,

Next issue:

VTune is complaining about nmi_watchdog being set...

However, this is the nmi_watchdog on the host system, not the target system (mic0).

Do I ignore this:
and will VTune use event based sampling on the target
.OR.
VTune will  not use event based sampling on the target eventhough nmi_watchdog is 0 on the target?

Jim Dempsey

[robert-reed]

NMI watchdog complaints only regard the host-side use of VTune Amplifier, and not an issue on the coprocessor.  There were versions of VTune Amplifier that would not accept /bin/ssh as the command to run, but I haven't seen that problem in the latest releases.  The fact that you're using root to initiate test runs on the coprocessor suggests that you're not using "user accounts" for secure ssh passwordless access to the coprocessor?  Enabling the ability for you to be you on the coprocessor should make it possible to use ssh on the command line and the rest of the ssh line in command line arguments.

[jimdempseyatthecove]

Robert,

My concern was (and your answer does not address):

When NMI watchdog enabled on host (no Host event based sampling supported)
.AND. NMI watchdog disabled on MIC
.WILL. will VTune permit me to configure an event based run?
.OR. will VTune "hand-hold" me and say to the effect "cannot permit event based handling because of nmi_watchdog enabled"?

After disabling nmi on host (though probably not required), next problem is on attempt of sample run "sampling driver not installed"

When I installed VTune, it complained about the NMI watchdog being enabled, which I thought I could ignore at least until I run an analysis. Would this also have not installed the sampling driver?

When following the instructions to install the sampling driver, I get "cannot find sep driver".

This is quite frustrating, especially when one is use to everything working nicely when installing Parallel Studio XE on Windows (without MIC).

I did find a different thread on how to build the sep driver, but your comment indicated this is not necessary should you install the "correct" version of VTune.

How does one coordinate the versions of: MPSS, Parallel Studio XE, and VTune such than all parts are happy with each other?

Jim Dempsey

[robert-reed · Accepted solution]

Always go with the latest?  No, even that doesn't work sometimes, but it's usually the best bet.  Latest MPSS is 3.1.2 and the latest VTune Amplifier is 2013 U15.  The installer should detect the presence of a coprocessor:

Step 1 of 7 | Welcome > Missing Optional Prerequisite(s)
--------------------------------------------------------------------------------
There are one or more optional unresolved issues. It is highly recommended to
resolve them all before you continue the installation. You can fix them without
exiting from the installation and re-check. Or you can quit from the
installation, fix them and run the installation again.
--------------------------------------------------------------------------------
Missing optional prerequisites
-- Power analysis is not enabled.
-- Intel(R) Manycore Platform Software Stack (Intel(R) MPSS) is detected
--------------------------------------------------------------------------------
1. Skip missing optional prerequisites [default]
2. Show the detailed info about issue(s)
3. Re-check the prerequisites

With this installer on the machine I selected for this test, picking the install defaults got me to the installation summary declaring "Restart Intel MPSS: yes" and accepting that, I get another prompt giving me a chance to avoid automatically restarting MPSS as part of the install.  I let it go with the default (restart).  The installer automatically uninstalls the old VTune Amplifier, installs the new components,  That component section concludes with:

Installing drivers. It may take several minutes... done
--------------------------------------------------------------------------------
Sampling driver built successfully
Sampling driver loaded with warning: NMI watchdog timer is enabled. Turn off the nmi_watchdog timer before running sampling.
Sampling driver boot script installed successfully
--------------------------------------------------------------------------------
Installing SEP for Intel(R) Xeon Phi coprocessors... done
--------------------------------------------------------------------------------
Restarting Intel(R) Xeon Phi coprocessors. It may take several minutes...

The upper section is for the host side, where NMI watchdog matters and only applies to sampling on the host. The lower section is for SEP on the coprocessor, where NMI watchdog setting is not relevant.  At this point VTune Amplifier should be ready to collect.  NMI watchdog time has NOT been disabled on the host:

[rreed@orspt-le64-81:/opt/intel/composerxe/lib/mic]
$ cat /proc/sys/kernel/nmi_watchdog
1

NMI watchdog timer doesn't exist on the coprocessor (at least with MPSS 3.1.2):

[rreed@orspt-le64-81-mic0:~]
$ find /proc -name \*nmi\* -print 2>/dev/null
/proc/sys/kernel/unknown_nmi_panic
/proc/sys/kernel/panic_on_unrecovered_nmi
/proc/sys/kernel/panic_on_io_nmi

You should get a similar experience from a Windows host, using the latest versions.  If you're seeing problems connecting to the driver and have not made any errors in the installation or invocation, you may have some weird configuration bug.  Installation is pretty cut and dried with current releases.  Some of the horror stories of earlier days are thankfully behind us.

From this point you should be able to customize a run with the events you're interested in, either from amplxe-gui (which is kind of hard to show here without attaching a bunch of screen shots), or from the command line tool, using an incantation like this:

[rreed@orspt-le64-81:~/local/knc-575]
$ amplxe-cl -collect-with runsa-knc -knob event-config=DATA_PAGE_WALK,DATA_READ_MISS_OR_WRITE_MISS,DATA_READ_OR_WRITE -- ssh `hostname`-mic0 projects/knc-575/amc-wmup
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/michome/rreed/projects/knc-575/r001runsa_knc -command stop.
numPaths = 524288, numSteps = 256
Init time: 1504.69 ms
rng Time     = 40.143
init asset time = 57.777
init cf time = 8.721
gemm Time    = 91.894
symm Time    = 238.35
reginit Time = 494.918
apply Time   = 65.798
Computation time: 1014.96 ms
European Put Option Price: 20.2504307
American Put Option Price: 20.0802493
amplxe: Collection stopped.
amplxe: Using result path `/home/michome/rreed/projects/knc-575/r001runsa_knc'
amplxe: Executing actions 16 % Resolving information for `coi_daemon'
amplxe: Warning: Cannot load symbols from `/usr/bin/coi_daemon'. Make sure the file format is valid.
amplxe: Warning: Cannot locate file `/lib64/libiomp5.so'.
amplxe: Warning: Cannot locate file `/lib64/libmkl_core.so'.
amplxe: Warning: Cannot locate file `/usr/lib64/libcoi_device.so.0'.
amplxe: Warning: Cannot locate file `sep3_10.ko'.
amplxe: Executing actions 16 % Resolving information for `libcoi_device.so.0'
amplxe: Warning: Cannot locate file `/usr/lib64/libstdc++.so.6.0.16'.
amplxe: Executing actions 16 % Resolving information for `libiomp5.so'
amplxe: Warning: Cannot locate file `/lib64/libmkl_intel_lp64.so'.
amplxe: Executing actions 16 % Resolving information for `libmkl_intel_lp64.so'
amplxe: Warning: Cannot locate file `/home/rreed/projects/knc-575/amc-wmup'.
amplxe: Executing actions 16 % Resolving information for `amc-wmup'
amplxe: Warning: Cannot locate file `/boot/vmlinuz-2.6.38.8+mpss3.1.2'.
amplxe: Executing actions 18 % Resolving information for `vmlinux'
amplxe: Warning: Cannot locate file `/lib64/libc-2.14.90.so'.
amplxe: Executing actions 18 % Resolving information for `libc-2.14.90.so'
amplxe: Warning: Cannot locate file `/lib64/libpthread-2.14.90.so'.
amplxe: Executing actions 18 % Resolving information for `libpthread-2.14.90.so
amplxe: Warning: Cannot locate file `/lib64/libcrypto.so.1.0.0'.
amplxe: Executing actions 18 % Resolving information for `libcrypto.so.1.0.0'
amplxe: Warning: Cannot locate file `/lib64/ld-2.14.90.so'.
amplxe: Executing actions 50 % Generating a report

General Exploration Metrics
---------------------------
Parameter            r001runsa_knc
-------------------  ----------------
CPU Time             287.095
Clockticks           315804473706.000
 CPU_CLK_UNHALTED    315804473706.000
Cache Usage          0.0
Vectorization Usage  0.0
TLB Usage            0.0
 L1 TLB Miss Ratio   0.008

Collection and Platform Info
----------------------------
Parameter                 r001runsa_knc                                         
------------------------  -----------------------------------------------------------------
Application Command Line  ssh "orspt-le64-81-mic0" "projects/knc-575/amc-wmup"  
User Name                 rreed                                                 
Operating System          Intel MIC Platform Software Stack (Built by Poky 7.0) 3.1.2 \n \l
Computer Name             orspt-le64-81-mic0                                    
Result Size               19262753                                              

CPU
---
Parameter          r001runsa_knc
-----------------  -----------------------------
Name               Intel(R) Xeon(R) E5 processor
Frequency          1100000000
Logical CPU Count  228

Summary
-------
Elapsed Time:  4.772

Event summary
-------------
Hardware Event Type           Hardware Event Count:Self  Hardware Event Sample Count:Self  Events Per Sample
----------------------------  -------------------------  --------------------------------  -----------------
DATA_PAGE_WALK                                174000522                                87  1000003
DATA_READ_MISS_OR_WRITE_MISS                  324000972                               162  1000003
DATA_READ_OR_WRITE                          21584064752                             10792  1000003
CPU_CLK_UNHALTED                           315804473706                             78951  2000003
amplxe: Executing actions 100 % done

Here I picked one of the coprocessor native apps I happen to be playing with these days, and I could manage a lot fewer warnings if I had added more information to that invocation for symbol resolution, but you can see above that I did get samples for the events I requested, despite the setting of NMI watchdog on the host.  Or I could take the result directory from this run into a project with the GUI tool, add the missing binary paths and refinalize the collection to resolve those symbol references.  

So you expressed an interest in doing event based SAMPLING? on the coprocessor.  Hopefully with this pass I got to the answer you were seeking.  If not, let me know what's lacking and I'll try again.

[robert-reed]

p.s., and the GUI tool in U15 accepts a raw "ssh" on the command line (see attached image)