Xeon Phi: Failed to create a tb6 file because 0 samples were collected

[Prasanna_B_]

Dear Vtune experts,

I am using vtune in command line mode to profile the time taken by each function in my code. 

amplxe-cl -collect knc-general-exploration --target-duration-type=veryshort --search-dir all:rp=./ -- ssh mic0 -t "ulimit -s unlimited; /tmp/myname/nekbone"

I can see the output of my application (name: nekbone) but vtune ends with the following message:

amplxe: Error: Failed to create a tb6 file because 0 samples were collected.
amplxe: Collection stopped.
amplxe: Using result path `/home/pbalapra/Projects/nekbone-2.3.4/test/example1/r002ge'
amplxe: Executing actions 50 % done                                            
amplxe: Error: Error 0x4000001e (Cannot load raw collector data)

Soft and hardware info:

amplxe-cl --version

Intel(R) VTune(TM) Amplifier XE 2013 Update 13 (build 313935) Command Line Tool
Copyright (C) 2009-2013 Intel Corporation. All rights reserved.

 /opt/intel/mic/bin/micinfo
MicInfo Utility Log

Created Mon Jan 20 17:22:53 2014

    System Info
        HOST OS            : Linux
        OS Version        : 2.6.32-358.6.2.el6.x86_64
        Driver Version        : 6720-21
        MPSS Version        : 2.1.6720-21
        Host Physical Memory    : 24535 MB

Device No: 0, Device Name: mic0

    Version
        Flash Version          : 2.1.02.0383
        SMC Firmware Version     : 1.11.4404
        SMC Boot Loader Version     : 1.8.4326
        uOS Version          : 2.6.38.8-gefd324e
        Device Serial Number      : ADKC22900202

    Board
        Vendor ID          : 0x8086
        Device ID          : 0x225c
        Subsystem ID          : 0x2500
        Coprocessor Stepping ID     : 1
        PCIe Width          : Insufficient Privileges
        PCIe Speed          : Insufficient Privileges
        PCIe Max payload size     : Insufficient Privileges
        PCIe Max read req size     : Insufficient Privileges
        Coprocessor Model     : 0x01
        Coprocessor Model Ext     : 0x00
        Coprocessor Type     : 0x00
        Coprocessor Family     : 0x0b
        Coprocessor Family Ext     : 0x00
        Coprocessor Stepping      : B0
        Board SKU          : ES2-P/A/X 1750
        ECC Mode          : Enabled
        SMC HW Revision      : Product 300W Active CS

    Cores
        Total No of Active Cores : 61
        Voltage          : 981000 uV
        Frequency         : 1090909 kHz

    Thermal
        Fan Speed Control      : On
        Fan RPM          : 2400
        Fan PWM          : 50
        Die Temp         : 61 C

    GDDR
        GDDR Vendor         : Elpida
        GDDR Version         : 0x1
        GDDR Density         : 2048 Mb
        GDDR Size         : 7936 MB
        GDDR Technology         : GDDR5
        GDDR Speed         : 5.500000 GT/s
        GDDR Frequency         : 2750000 kHz
        GDDR Voltage         : 1501000 uV

[robert-reed]

So, how long does nekbone run?  I noticed the selection of target-duration-type as veryshort, but also using the knc-general-exploration collector, which is a multiplexed event collector, needing more run time in order to rotate through the event sets to give a sufficient picture of each set of events.  It's possible that whatever parallelism is available in your code, spread across 240+ threads, results in too short a duration to run the collection.

Do you get the same result when you run the Hotspots KNC collector?

[Prasanna_B_]

Robert, Thanks a buch for your help! Previously the runtime was 5s. Now I increased to 60s but still I have the same issue for both hotspots and general exploration. Is there something obvious that I am missing with the compile flags. The code is mix of Fortran and C and I compile with the following flas. F77:= ifort -openmp -mmic -g -D -O2 CC:= icc -openmp -mmic -g -D -O2

[robert-reed]

You're doing cross-compilation (-mmic) which is consistent with the native run via ssh.  I tried duplicating your amplxe-cl invocation with a native program I happen to be playing with, and it collected and worked fine, so I don't see anything wrong with your invocation of amplxe-cl.

Are you sure your code is running on the coprocessor?  Or might there be a missing library that's causing the native run to abort?  You might not see any notice outside of the amplxe-cl invocation, and it's the only thing that occurs to me right now that is consistent with the facts you've presented.  What you should see is something similar to my test results, parts of which I show below:

[rreed@orspt-le64-82:~/local/smallmatrix/knc-575]
$ amplxe-cl -collect knc-hotspots --target-duration-type=veryshort  --search-dir all:rp=./ -- ssh `hostname`-mic0 -t "ulimit -s unlimited; projects/smallmatrix/knc-575/knc-matrix-test"

amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/michome/rreed/projects/smallmatrix/knc-575/r002hs -command stop.
numPaths = 524288, numSteps = 256
Computation time: 506.064 ms
Connection to orspt-le64-82-mic0 closed.
amplxe: Collection stopped.
amplxe: Using result path `/home/michome/rreed/projects/smallmatrix/knc-575/r002hs'
amplxe: Executing actions 16 % Resolving module symbols

...

[TimP]

As Robert hinted at, there's a tradeoff among how many options you select in the general category and how long a minimum suitable run might be without ticking on "allow multiple runs."  A 60s run should be plenty to get useful hotspots as that doesn't involve multiplexing.  At 5s you would surely need to "allow multiple" which would repeat your run however many times are required to collect all events for the full 5 sec. as well as setting the Advanced options to "less than 1 minute" if relying on the GUI.

I would also agree with Robert's hint that you should be running a realistic combination of numbers of threads and ranks.  If I remember nekbone correctly, that might be 16-20 MPI ranks, with perhaps 3 threads per rank, spread out across cores e.g. by KMP_AFFINITY=balanced, provided that you have added threading which isn't in the normal versions.

You must make certain that your vtune sampling daemons are running, e.g. by using the install script in the k1om directory of the vtune installation on the host.

Compile options don't impact your ability to make a .tb6 file although they will affect the usefulness of the results.

People have invested hours specifically in modifying nekbone to take advantage of MIC.  Take advantage of their experience.

You need 'sudo micinfo' to gather a full set of information there.

[Prasanna_B_]

Robert and Tim,

Thanks a lot for your inputs.

First, I did the same experiment with a simple matrix multiplication with repititions that run for 60 sec to remove any effect due to nekbone. I get the same issue. I made sure that I am using 60 cores (by looking at the resulting runtime and power consumption).

Concerning the nekbone, currently there is a new openmp only version (no MPI). I am doing some experiments with that code.

Before posting in the forum, I asked the system admin to perform the steps in http://software.intel.com/sites/products/documentation/doclib/iss/2013/amplifier/lin/ug_docs/GUID-92B13766-1617-44D0-BA70-AC61F5833CA4.htm. It doesn’t appear to have changed anything from the already existing config.

How to make sure that vtune sampling daemons are running (using ps aux for example)?

[robert-reed]

Tim asks about whether the VTune Amplifier sampling daemon is running on the coprocessor because there have been some issues in the past about ensuring the drivers get installed properly.  I have not seen such problems myself in recent releases and the errors you have reported have not been the ones usually associated with a failure to launch the collection driver, so I didn't bring up that topic.  It's easy enough to see whether the sample collection driver is running on a particular coprocessor:

$ ssh `hostname`-mic0 ps aux |grep sep
root       4841  0.0  0.0  78056  2256 ?        Sl   Jan12   0:08 ./sep_mic_server3.10

The string above launches a ps run on the local coprocessor called "mic0" and then scans the output back on the host for instances of SEP, the sampling collector VTune Amplifier uses for gathering data from the coprocessors.  I'm running the latest release of VTune Amplifier, which currently uses SEP 3.10 for collection driver.  If your sys admin has successfully installed VTune Amplifier on your cluster nodes, you should get a similar message.

The message that you have been getting--no data for analysis--is usually associated with some failure in the execution of the target program on the coprocessor, which is why I asked what happens to your nekbone run if you invoke it without the amplxe-cl preamble seen on your previous examples.  If you use OpenMP and the libiomp5.so shared object is not found on the coprocessor, for example, aborting the coprocessor run, I would expect to see possibly some error messages in the window where you launched and a report from VTune Amplifier that it didn't get any data.  Before trying to go to extraordinary means to understand what is failing in the link between VTune Amplifier and the data supposedly being generated by your test run, I just want to be sure we've passed the sanity check that your test run is actually executing.

[Prasanna_B_]

Robert,

Thanks again!

The driver is running and I get the following output with your command.

3582 root   0:04 ./sep_mic_server3.10

I have attached the output of the executable with and without vtune commandline.

Note that the sum of Solve time is approx 12 sec in both cases and the residuals are also similar. Therfore, I am sure that the excutable is fine.

I also figured out that I was not the part of vtune user group. I changed it but still I get the same issue.

Moving further, NMI_Watchdog was enabled before and I disabled now. But still the same issue.

When I use the Vtune GUI, I get the same issue. 

Finally, I compiled the code for westmere host (without mmic) and in the GUI, I can't run General Exploration. I get "Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*)".

The MPSS version is 2.1.6720-21 and not new 3.x! Do you think that might be an issue?

[robert-reed]

Wow.  It sure LOOKS like your program is running, even when run with amplxe-cl.  The latest releases of VTune Amplfier contain drivers for both MPSS 2.1 and MPSS 3, so having the old MPSS should not be a problem.  The VTune  user group is deprecated and should not be an issue on the coprocessor.  The rest of the ideas that occur to me at the moment are increasingly unsatisfying.  It really does look like the events are not being issued, and I can't spend any more time on it right now.  Let me think about this some more and get back to you soon.  It looks to me like it SHOULD be working.  You're not seeing any errors of the nature of "couldn't communicate with driver" so it looks, bizarrely, like the run is not generating any events.  

[robert-reed · Accepted solution]

I'm coming around to the belief that something may be wrong in your VTune Amplifier installation.  So the next test I'll suggest is just a basic sanity check on the running of amplxe-cl.  Please try the following line on your system:

amplxe-cl -collect knc-hotspots ssh `hostname`-mic0 ls -CF /usr/bin

Perform this from a directory on your host that has sufficient storage to capture the files generated by amplxe-cl.  This line invokes VTune Amplifier to collect against the coprocessor copy of 'ls' and so should eliminate any confusion about what's installed and what's not.  When I performed this simple line on one of our local test machines, I got the following:

$ amplxe-cl -collect knc-hotspots ssh `hostname`-mic0 ls -CF /usr/bin
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/michome/rreed/projects/smallmatrix/knc-575/r004hs -command stop.
[@                       groups@                  rpms2solv*
[[@                      groups.shadow*           run-parts@
ar@                      gzexe*                   scdaemon*
augparse*                head@                    scp@
augtool*                 helix2solv*              scp.openssh*
awk@                     hexdump@                 seq@
basename@                ibv_asyncwatch*          sg@
bunzip2@                 ibv_devices*             skill*
bzcat@                   ibv_devinfo*             slabtop*
chage*                   ibv_rc_pingpong*         snice*
chattr*                  ibv_srq_pingpong*        solv*
chfn@                    ibv_uc_pingpong*         sort@
chfn.shadow*             ibv_ud_pingpong*         ssh@
chsh@                    id@                      ssh-keygen*
chsh.shadow*             installation_sources*    ssh.openssh*
chvt@                    installcheck*            strings@
clear@                   kbxutil*                 su*
cmp@                     killall@                 susetags2solv*
coi_daemon*              ksba-config*             systool*
compile_et*              last@                    tail@
cr_checkpoint*           last.sysvinit*           tee@
cr_restart*              lastb@                   telnet@
cr_run*                  lastlog*                 test@
cut@                     ldd*                     tftp@
dapltest*                less@                    time@
dc@                      libassuan-config*        tload*
deallocvt@               libibverbs.so@           top@
deltainfoxml2solv*       libusb-config*           top.procps*
deptestomatic*           logger@                  tr@
diff@                    logname@                 traceroute@
dirname@                 lsattr*                  tty@
dlist_test*              mckey*                   ucmatose*
dtest*                   md5sum@                  udaddy*
dtestcm*                 mergesolv*               uncompress*
dtestx*                  mesg@                    uniq@
du@                      mesg.sysvinit*           unzip@
dumpleases@              microcom@                update-alternatives*
dumpsolv*                mk_cmds*                 updateinfoxml2solv*
env@                     mkfifo@                  uptime@
eu-ar*                   nc@                      uptime.procps*
eu-elfcmp*               newgrp@                  users@
eu-elflint*              newgrp.shadow*           utmpdump*
eu-findtextrel*          nohup@                   uuidgen*
eu-make-debug-archive*   nslookup@                vlock@
eu-objdump*              od@                      vmstat*
eu-ranlib*               openvt@                  w*
eu-strings*              passwd@                  wall@
eu-unstrip*              passwd.shadow*           wall.sysvinit*
expiry*                  patch@                   watch@
expr@                    pgrep@                   watch.procps*
fadot*                   pgrep.procps*            watchgnupg*
faillog*                 pkill@                   wc@
file*                    pkill.procps*            wget@
find@                    pmap@                    which@
flock@                   pmap.procps*             who@
free@                    printf@                  whoami@
free.procps*             pwdx@                    xargs@
fuser@                   pwdx.procps*             xml2-config*
gdbserver*               rdma_client*             xmlcatalog*
get_device*              rdma_server*             xmllint*
get_driver*              rdma_xclient*            yes@
get_module*              rdma_xserver*            zcmp*
gpasswd*                 readlink@                zdiff*
gpg@                     realpath@                zegrep*
gpg-agent*               renice@                  zfgrep*
gpg-connect-agent*       repo2solv.sh*            zforce*
gpg2*                    repomdxml2solv*          zgrep*
gpgconf*                 reset@                   zless*
gpgkey2ssh*              rping*                   zmore*
gpgparsemail*            rpm*                     znew*
gpgsm*                   rpmconstant*             zypp-CheckAccessDeleted*
gpgsm-gencert.sh*        rpmdb2solv*              zypper*
gpgv2*                   rpmmd2solv*
amplxe: Collection stopped.
amplxe: Using result path `/home/michome/rreed/projects/smallmatrix/knc-575/r004hs'
amplxe: Executing actions 17 % Resolving information for dangling locations
amplxe: Warning: Cannot locate file `/lib64/libcrypto.so.1.0.0'.
amplxe: Executing actions 17 % Resolving information for `libcrypto.so.1.0.0'
amplxe: Warning: Cannot locate file `intel_micveth.ko'.
amplxe: Executing actions 19 % Resolving information for `intel_micveth'
amplxe: Warning: Cannot locate file `/boot/vmlinuz-2.6.38.8+mpss3.1.2'.
amplxe: Executing actions 50 % Generating a report

General Exploration Metrics
---------------------------
Parameter             r004hs
--------------------  --------------
CPU Time              2.164
Clockticks            2380000000.000
 CPU_CLK_UNHALTED     2380000000.000
Instructions Retired  130000000
CPI Rate              18.308
Cache Usage           0.0
Vectorization Usage   0.0
TLB Usage             0.0

Collection and Platform Info
----------------------------
Parameter                 r004hs                                                
------------------------  -----------------------------------------------------------------
Application Command Line  ssh "orspt-le64-82-mic0" "ls" "-CF" "/usr/bin"        
User Name                 rreed                                                 
Operating System          Intel MIC Platform Software Stack (Built by Poky 7.0) 3.1.2 \n \l
Computer Name             orspt-le64-82-mic0                                    
Result Size               1800333                                               

CPU
---
Parameter          r004hs
-----------------  -----------------------------
Name               Intel(R) Xeon(R) E5 processor
Frequency          1100000000
Logical CPU Count  228

Summary
-------
Elapsed Time:  1.342
CPU Usage:     0.786

Event summary
-------------
Hardware Event Type    Hardware Event Count:Self  Hardware Event Sample Count:Self  Events Per Sample
---------------------  -------------------------  --------------------------------  -----------------
CPU_CLK_UNHALTED                      2380000000                               238  10000000
INSTRUCTIONS_EXECUTED                  130000000                                13  10000000
amplxe: Executing actions 100 % done

My invocation line was a bare minimum, and there are warnings and errors in the output indicating the failure to resolve various libraries, but even running just 'ls' you can see just above that this run generated 238 CPU_CLK_UNHALTED samples and 13 INSTRUCTIONS_EXECUTED samples (these are the two events collected for simple hot spot analysis).  If you are not able to get a similar result on your system, either there's a problem writing the outputs, which may be a file space problem, or possibly there is a problem in the installation of VTune Amplifier on your system.  My hunch is that it is the latter, and if you're not able to reproduce this simple collection test, I'd recommend that your administrator reinstall VTune Amplifier, and if reinstalling, you might as well install the latest version, Update 15, if you haven't done so already.  You should be able to get results similar to what I listed above.  Once you can complete a collection on such a basic test, then you might have better luck with your nekbone runs.

[Prasanna_B_]

Robert,

You are right! There is something wrong with VTune Amplifier installation. I do not know what it is! When I run the command, I got the same issue.

The admin told me about the previous version of vtune and the test works and I got the output similar to yours. I also run nekbone and vtune can collect the hardware counters.

Thanks a lot!!

[robert-reed]

I'm glad we figured it out.  My recommendation for your admin is to uninstall, ream and reinstall VTune Amplifier on that node (delete any files and folders that might remain--excepting maybe license files--that may be artifacts of a bad install). 

Or better yet, have your admin upgrade to Update 15 if you're still running Update 13.  13 is so last summer! :-)

And I hope VTune Amplifier gives you great insights about your nekbone runs.