Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Giuseppe_C_
Beginner
705 Views

amplxe-gui - the corresponding command in command line

Which is the text-based command-line analysis and report utilities that match at amplxe-gui?
 
Thanks
0 Kudos
21 Replies
PAVEL_G_Intel
Employee
666 Views

Hi, 

VTune Amplifier provides CLI interface.

To get the command line to start analysis that match at GUI please find button "Get Command line" on analysis configuration tab. As for report CLI read this topic.

Could you describe your usage model? For what reasons do you need command line reports? Do you have system limitations which do not allow start GUI on the system?

Giuseppe_C_
Beginner
666 Views

Thanks Pavel,

I have tried this:

mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
..
..
..
Event summary
-------------
Hardware Event Type                 Hardware Event Count:Self  Hardware Event Sample Count:Self  Events Per Sample
----------------------------------  -------------------------  --------------------------------  -----------------
CPU_CLK_UNHALTED.THREAD                            1716000000                               660  2600000          
CPU_CLK_UNHALTED.REF_TSC                           1716000000                               660  2600000          
INST_RETIRED.ANY                                   1892800000                               728  2600000          
CPU_CLK_UNHALTED.REF_XCLK                            66001980                               660  100003           
CPU_CLK_UNHALTED.ONE_THREAD_ACTIVE                   66001980                               660  100003   
 
amplxe-cl -report summary -r my_result.login.cluster.net/
amplxe: Using result path `/users/home/sysm03/my_result.login.cluster.net'
amplxe: Executing actions 50 % done                                            
amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `summary': no valid license can be found (This platform is not supported: your license contains platform restrictions.).
 
I do not have license problems if I use amplxe-gui or if I use "amplxe-cl -report summary " on other compute node:
 
other compute node:
 
amplxe-cl -report summary -r my_result.login2
amplxe: Using result path `/users/home/sysm03/my_result.login2'
amplxe: Executing actions 75 % Generating a report 
..
..
 
I'm trying to reload sep driver on login node (insmod-sep -r), but now "amplxe" remains blocked and does not start (on login node)
 
I don't have system limitations to start GUI, but X session is very very slowly.
 
Thanks
Vladimir_R_Intel
Employee
666 Views

Hi,

you don't need to reload sep. The issue you observe fully relates to an ability to find a proper license file. Could you please check that it is available on the node where you try to generate a report? As I remember it should locate at /opt/intel/licenses if VTune Amplifier was installed from root user. 

BR,

Vladimir

Giuseppe_C_
Beginner
666 Views

Thanks Vladimir,

I don't use license file but a license manager.

I had to reload the sep, because when I run "amplxe-cl collect" on login node, the tool returns:

PMU resources currently being used by another profiling tool or process.

Now, after sep reload, "amplxe-cl collect" hanging with out log

 

Regards

Vladimir_R_Intel
Employee
666 Views

Please give a little more details what you mean under "hanging with out log"

Giuseppe_C_
Beginner
666 Views

Sorry, I mean "without logs"

 

 

 

 

[login1]> module load IMPI/intel_mpi_2018 INTEL/intel_xe_2018
[login1]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
 
The task not start.
 
If I check the process in another console, I get:
 
user   17674 31477  0 14:11 pts/74   00:00:00 /bin/sh /users/home/opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi/intel64/bin/mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
user   17681 17674  0 14:11 pts/74   00:00:00 mpiexec.hydra -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
user   17682 17681  0 14:11 pts/74   00:00:00 /users/home/opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi/intel64/bin/pmi_proxy --control-port login1:36980 --pmi-connect alltoall --pmi-aggregate -s 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1732786525 --usize -2 --proxy-id 0
user   17689 17682  2 14:11 pts/74   00:00:00 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
user   17690 17682  2 14:11 pts/74   00:00:00 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
user   17773 17690  1 14:11 pts/74   00:00:00 /users/home/opt/intel/vtune_amplifier_2018.3.0.558279/bin64/amplxe-runss --ui-output-format xml --ui-output-fd 17 --option-file /users/home/user/my_result.login1.cluster.net/config/runsa.options.11
user   17775 17689  1 14:11 pts/74   00:00:00 /users/home/opt/intel/vtune_amplifier_2018.3.0.558279/bin64/amplxe-runss --ui-output-format xml --ui-output-fd 18 --option-file /users/home/user/my_result.login1.cluster.net/config/runsa.options.10
 
I'm forced to use ctrl-c to exit:
 
mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong
 
[mpiexec@login1.cluster.net] Sending Ctrl-C to processes as requested
[mpiexec@login1.cluster.net] Press Ctrl-C again to force abort
^CCtrl-C caught... cleaning up processes
APPLICATION TERMINATED WITH THE EXIT STRING: job ending due to timeout = -1
Thanks
Vladimir_R_Intel
Employee
666 Views

Could you try to use a different result directory? If it doesn't help , please set a PERFRUN_DEBUG=1 environment variable, repeat your experiment with a new result directory and upload this result and collected logs? (logs will be located in <result>/log or in working directory).

BR,

Vladimir

Giuseppe_C_
Beginner
666 Views

it was enough to change the folder result and it worked (worked with PERFRUN_DEBUG and without PERFRUN_DEBUG):
 
export  PERFRUN_DEBUG=1
[login1][/users/home/user/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test-dir-ampl -- IMB-MPI1 pingpong
amplxe: Analyzing data in the node-wide mode. The hostname (login1.cluster.net) will be added to the result path/name.
amplxe: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel modules symbols.
amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
amplxe: Collection started.
#------------------------------------------------------------
#    Intel (R) MPI Benchmarks 2018, MPI-1 part    
#------------------------------------------------------------
# Date                  : Fri Jun 29 14:29:27 2018
# Machine               : x86_64
# System                : Linux
# Release               : 2.6.32-642.6.2.el6.x86_64
# Version               : #1 SMP Wed Oct 26 06:52:09 UTC 2016
# MPI Version           : 3.1
# MPI Thread Environment: 
 
 
# Calling sequence was: 
 
# IMB-MPI1 pingpong
 
# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT
# MPI_Op                         :   MPI_SUM  
#
#
 
# List of Benchmarks to run:
 
# PingPong
 
#---------------------------------------------------
# Benchmarking PingPong 
# #processes = 2 
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         0.68         0.00
            1         1000         0.73         1.37
            2         1000         0.72         2.76
            4         1000         0.74         5.42
            8         1000         0.74        10.86
           16         1000         0.76        21.04
           32         1000         0.78        40.95
           64         1000         0.80        79.80
          128         1000         0.88       144.81
          256         1000         0.97       264.89
          512         1000         0.97       530.24
         1024         1000         1.24       829.14
         2048         1000         1.55      1324.69
         4096         1000         2.15      1906.86
         8192         1000         3.43      2388.33
        16384         1000         6.18      2652.03
        32768         1000        11.53      2841.12
        65536          640         9.09      7209.76
       131072          320        15.97      8208.18
       262144          160        31.60      8296.44
       524288           80        61.82      8480.21
      1048576           40       125.11      8381.02
      2097152           20       243.30      8619.61
      4194304           10       503.66      8327.66
 
 
# All processes entering MPI_Finalize
 
amplxe: Collection stopped.
amplxe: Using result path `/users/home/user/test-dir-ampl.login1.cluster.net'
amplxe: Executing actions 19 % Resolving information for `libittnotify_collecto
amplxe: Warning: Cannot locate debugging information for file `/lib64/ld-2.12.so'.
amplxe: Warning: Cannot locate debugging information for file `/users/home/opt/intel/vtune_amplifier_2018.3.0.558279/lib64/runtime/libittnotify_collector.so'.
amplxe: Executing actions 19 % Resolving information for `IMB-MPI1'            
amplxe: Warning: Cannot locate debugging information for file `/users/home/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin/IMB-MPI1'.
amplxe: Executing actions 20 % Resolving information for `libmpi.so.12.0'      
amplxe: Warning: Cannot locate debugging information for file `/lib64/libc-2.12.so'.
amplxe: Executing actions 100 % done                                           
 
Collection and Platform Info
----------------------------
Parameter                 test-dir-ampl.login1.cluster.net                    
------------------------  ----------------------------------------------------
Application Command Line  IMB-MPI1 "pingpong"                                 
Operating System          2.6.32-642.6.2.el6.x86_64 CentOS release 6.2 (Final)
MPI Process Rank          1                                                   
Computer Name             login1.cluster.net                                  
Result Size               4418087                                             
Collection start time     12:29:26 29/06/2018 UTC                             
Collection stop time      12:29:27 29/06/2018 UTC                             
Collector Type            Driverless Perf per-process sampling                
 
CPU
---
Parameter          test-dir-ampl.login1.cluster.net                 
-----------------  -------------------------------------------------
Name               Intel(R) Xeon(R) E5 Processor code named Jaketown
Frequency          2600002833                                       
Logical CPU Count  16                                               
 
Summary
-------
Elapsed Time:             0.512
Paused Time:              0.0  
CPU Time:                 0.664
Average CPU Utilization:  0.720
CPI Rate:                 0.907
 
Event summary
-------------
Hardware Event Type                 Hardware Event Count:Self  Hardware Event Sample Count:Self  Events Per Sample
----------------------------------  -------------------------  --------------------------------  -----------------
CPU_CLK_UNHALTED.THREAD                            1726400000                               664  2600000          
CPU_CLK_UNHALTED.REF_TSC                           1726400000                               664  2600000          
INST_RETIRED.ANY                                   1903200000                               732  2600000          
CPU_CLK_UNHALTED.REF_XCLK                            66401992                               664  100003           
CPU_CLK_UNHALTED.ONE_THREAD_ACTIVE                   66401992                               664  100003 
 
but now seems that the sep is unloaded (for ampllxe):
 
amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location
.
[/root]> /users/home/opt/intel/vtune_amplifier_2018/sepdk/src/insmod-sep -q
pax driver is loaded and owned by group "vtune" with file permissions "775".
socperf2_0 driver is loaded and owned by group "vtune" with file permissions "775".
sep4_1 driver is loaded and owned by group "vtune" with file permissions "775".
vtsspp driver is loaded and owned by group "vtune" with file permissions "664".
[athena-login1][/root]> lsmod | grep sep
sep4_1                830967  0 
socperf2_0             24366  1 sep4_1
 
Thanks
 
 
Vladimir_R_Intel
Employee
666 Views

Great! This means that some garbage left from previous launch (I think it was unsuccessful) in the result directory. Please use an unique result directory to prevent such behavior in future. PERFRUN_DEBUG isn't needed anymore.

About the warnings you see during collection:

These warnings don't mean that sep driver is unloaded it just suggests what you have to do to see kernel related info in your result. Otherwise the time spent in kernel will be attributed in [Outside any known module] So if you don't interested in this info just ignore the warning. 

BR,

Vladimir

Giuseppe_C_
Beginner
666 Views

However, before I reloaded the sep kernel module in the login node, the warning "To profile kernel modules during the session" did not show up.
 
Another observation:
 
the computational nodes do not have sep modules loaded (i have installed parallel studio on login node in distributed directory), Do I have to manually load the sep module on all compute nodes? (amplxe also starts without kernel module loaded but amplxe return the warning "To profile kernel modules during the session")
 
Apart from that, on the login node, I still have the problem of the license for "amplxe-cl report"
 
amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `summary': no valid license can be found (This platform is not supported: your license contains platform restrictions.).
 
 
Thanks
PAVEL_G_Intel
Employee
666 Views

Hi,

Please check that license file is located in /opt/intel/licenses. If it's set env variable and run report:

export INTEL_LICENSE_FILE=/opt/intel/licenses
amplxe-cl report

- Pavel

Giuseppe_C_
Beginner
666 Views

Sorry,

but I use Intel license manager:

echo $INTEL_LICENSE_FILE
28518@mng2:28518@login2:28518@mng1
 
I realized that in the folder /opt/intel/licenses of login node, there was an old license, I removed it and now it works.
 
Apart from that, the compute nodes do not have sep drivers loaded (i have installed parallel studio on login node in distributed directory).
Do I have to manually load sep driver on all the nodes?
Until now, Amplxe also starts without driver sep loaded (apart warning:

amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location)

 

Thanks

PAVEL_G_Intel
Employee
666 Views

As I know... Yes, you need to install sep manually.

But I recommend to wait answer form Vladimir next week. I'm not very proficient in mpi profiling.

- Pavel

Giuseppe_C_
Beginner
666 Views

Ok, I'll wait the answer from Vladimir

I add that:
 
if I load manually 2018 sep driver for amplxe-2018, are they good for amplxe-2015 too?
 
Regards
PAVEL_G_Intel
Employee
666 Views

We have sep version 4 and 3 in the products. They may be loaded to system at the same time. Every instance of VTune Amplifier will use its version.

Remember! The drivers shall not be in use at the same time. So two collections from different versions of VTune will fail.

Vladimir_R_Intel
Employee
666 Views

Hi Osvaldo,

answering on your question:

>>Do I have to manually load sep driver on all the nodes?

This depends on your needs. I mean that VTune will try to use driverless Perf based approach if it detects that no driver is loaded. And if it cannot use Perf it will generate an appropriate error message describing what you need to do to get an interested collection result. (SEP driver installation should be an option there). 

But if you are interested in SEP driver as a collector by some reason then you will need to load it on all the nodes. Please look at the following article for more details https://software.intel.com/en-us/vtune-amplifier-help-driverless-event-based-sampling-collection

BR,

Vladimir

Giuseppe_C_
Beginner
666 Views

Thanks Vladimir, all clear.

I have tried to load sep4 and sep3 on the same compute nodes, but amplxe-cl don't start:

LOAD SEP4:

cd /users/home/opt/intel/vtune_amplifier_2018/sepdk/src

[root@n300 src]# ./insmod-sep -r
 
Warning:  the following driver(s) were not found loaded in the kernel:  sep4_1.
 
Warning:  no vtsspp driver was found loaded in the kernel.
 
Warning:  the following driver(s) were not found loaded in the kernel:  socperf2_0.
 
The PAX service is not loaded anymore.
Checking for PMU arbitration service (PAX) ... not detected.
Attempting to start PAX service ...
Executing: insmod ./pax/pax-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko
Creating /dev/pax device with major number 242 ... done.
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "666" ... done.
The pax driver has been successfully loaded.
PAX service has been started.
Checking for socperf driver ... not detected.
Executing: insmod ./socperf/src/socperf2_0-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko
Creating /dev/socperf2_0 base devices with major number 241 ... done.
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "666" ... done.
The socperf2_0 driver has been successfully loaded.
Executing: insmod ./sep4_1-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko
Creating /dev/sep4_1 base devices with major number 240 ... done.
Creating /dev/sep4_1 percpu devices with major number 239 ... done.
Creating /dev/sep4_1 percpu devices with major number 238 ... done.
Creating /dev/sep4_1 per package devices with major number 237 ... done.
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "666" ... done.
The sep4_1 driver has been successfully loaded.
Checking for vtsspp driver ... not detected.
Executing: insmod ./vtsspp/vtsspp-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko gid=502 mode=0666
The vtsspp driver has been successfully loaded.
 
LOAD SEP 3:
 
cd  /users/home/opt/intel/vtune_amplifier_2015/sepdk/src 
 
[root@n300 src]# ./insmod-sep3 -r
Warning:  no sep3_15 driver was found loaded in the kernel.
Removing vtsspp driver from the kernel ... done.
The vtsspp driver has been successfully unloaded.
Attempting to stop PAX service ...
Removing pax driver from the kernel ... done.
Deleting previously created /dev/pax device ... done.
The pax driver has been successfully unloaded.
PAX service has been stopped.
Checking for PMU arbitration service (PAX) ... not detected.
Attempting to start PAX service ...
Executing: insmod ./pax/pax-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko
Creating /dev/pax device with major number 242 ... done.
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "666" ... done.
The pax driver has been successfully loaded.
PAX service has been started.
PAX service is accessible to all users.
Executing: insmod ./sep3_15-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko
Creating /dev/sep3_15 base devices with major number 236 ... done.
Creating /dev/sep3_15 percpu devices with major number 235 ... done.
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "666" ... done.
The sep3_15 driver has been successfully loaded.
Checking for vtsspp driver ... not detected.
Executing: insmod ./vtsspp/vtsspp-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko gid=502 mode=0666
The vtsspp driver has been successfully loaded.
 
KERNEL MODULE LOADED:
 
[root@n300 src]# lsmod | grep sep
sep3_15               527559  0 
sep4_1                830967  0 
socperf2_0             24366  1 sep4_1
[root@n300 src]# 
 
 
 
EXEC AMPLXE-CL 2018:
 
[n300][/users/home/USER/]> module load IMPI/intel_mpi_2018 INTEL/intel_xe_2018
[n300][/users/home/USER/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test5 -- IMB-MPI1 pingpong
amplxe: Analyzing data in the node-wide mode. The hostname (n300.cluster.net) will be added to the result path/name.
amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
NMI watchdog timer is enabled. Turn off the nmi_watchdog timer and restart.
amplxe: Error: NMI watchdog timer is enabled. Turn off the nmi_watchdog timer and restart.
 
( amplxe-cl remains hanging )
 
EXEC AMPLXE-CL 2015:
 
[n300][/users/home/USER/]> module load INTEL/intel_xe_2015.3.187  IMPI/intel_mpi_5.0.3.048
[n300][/users/home/USER/]> 
[n300][/users/home/USER/]> 
[n300][/users/home/USER/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test6 -- IMB-MPI1 pingpong
amplxe: The analyzed process has a rank of 0. This rank will be added to the result path/name.
amplxe: The analyzed process has a rank of 1. This rank will be added to the result path/name.
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /users/home/USER/test6.0 -command stop.
amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /users/home/USER/test6.1 -command stop.
 
( amplxe-cl remains hanging )
 
What i am doing wrong?
 
Thanks
 
 
Vladimir_R_Intel
Employee
666 Views

What user did you launch from? This message seems that VTune failed to disable NMI watchdog. And the only reason I can suppose it happened because of no permissions. 

Please try to run a simple no mpi launch. And in case of the same error please try to disable NMI watchdog manually.

Also need to mention that in your second launch with VTune 2015 it will no work so as this version weren't adapted for per node mpi collection, it tried to collect per rank results and corresponding collections conflicted for PMU resources.

Giuseppe_C_
Beginner
666 Views

I have done other tests, recapitulating:
 
SEP3 LOADED (only sep3)
- amplxe-cl-2015 works if i don't use mpirun
- amplxe-cl-2015 hangs if I use mpirun
 
SEP4 LOADED (only sep4)
- amplxe-cl-2018 works and the warning is: 
"amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes." (seems correct)
- amplxe-cl-2015 don't works 
 
It seems impossible to use sep3 + sep4 for PE-2015 and PE-2018.
What do you suggest me? Use only sep4 for PE-2018?
Can I analyze code compiled with PE-2015 with amplex-cl-2018?
 
Best regards
Vladimir_R_Intel
Employee
230 Views

Osvaldo, thank you for the report and for your time. This is really confused. I'll try to connect experts in SEP driver to this thread to investigate this issue.

And about suggestions, I suggest to use sep4. And yes, you can use VTune Amplifier 2018 to analyze code compiled with PE-2015, there should not be any problem with that.

Reply