Thanks Vladimir, all clear.

Giuseppe_C_ · ‎06-29-2018

Which is the text-based command-line analysis and report utilities that match at amplxe-gui?

Thanks

PAVEL_G_Intel · ‎06-29-2018

Hi,

VTune Amplifier provides CLI interface.

To get the command line to start analysis that match at GUI please find button "Get Command line" on analysis configuration tab. As for report CLI read this topic.

Could you describe your usage model? For what reasons do you need command line reports? Do you have system limitations which do not allow start GUI on the system?

Giuseppe_C_ · ‎06-29-2018

Thanks Pavel,

I have tried this:

mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

..

Event summary

-------------

Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample

---------------------------------- ------------------------- -------------------------------- -----------------

CPU_CLK_UNHALTED.THREAD 1716000000 660 2600000

CPU_CLK_UNHALTED.REF_TSC 1716000000 660 2600000

INST_RETIRED.ANY 1892800000 728 2600000

CPU_CLK_UNHALTED.REF_XCLK 66001980 660 100003

CPU_CLK_UNHALTED.ONE_THREAD_ACTIVE 66001980 660 100003

amplxe-cl -report summary -r my_result.login.cluster.net/

amplxe: Using result path `/users/home/sysm03/my_result.login.cluster.net'

amplxe: Executing actions 50 % done

amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `summary': no valid license can be found (This platform is not supported: your license contains platform restrictions.).

I do not have license problems if I use amplxe-gui or if I use "amplxe-cl -report summary " on other compute node:

other compute node:

amplxe-cl -report summary -r my_result.login2

amplxe: Using result path `/users/home/sysm03/my_result.login2'

amplxe: Executing actions 75 % Generating a report

..

I'm trying to reload sep driver on login node (insmod-sep -r), but now "amplxe" remains blocked and does not start (on login node)

I don't have system limitations to start GUI, but X session is very very slowly.

Thanks

Vladimir_R_Intel · ‎06-29-2018

Hi,

you don't need to reload sep. The issue you observe fully relates to an ability to find a proper license file. Could you please check that it is available on the node where you try to generate a report? As I remember it should locate at /opt/intel/licenses if VTune Amplifier was installed from root user.

BR,

Vladimir

Giuseppe_C_ · ‎06-29-2018

Thanks Vladimir,

I don't use license file but a license manager.

I had to reload the sep, because when I run "amplxe-cl collect" on login node, the tool returns:

PMU resources currently being used by another profiling tool or process.

Now, after sep reload, "amplxe-cl collect" hanging with out log

Regards

Vladimir_R_Intel · ‎06-29-2018

Please give a little more details what you mean under "hanging with out log"

Giuseppe_C_ · ‎06-29-2018

Sorry, I mean "without logs"

[login1]> module load IMPI/intel_mpi_2018 INTEL/intel_xe_2018

[login1]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

The task not start.

If I check the process in another console, I get:

user 17674 31477 0 14:11 pts/74 00:00:00 /bin/sh /users/home/opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi/intel64/bin/mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

user 17681 17674 0 14:11 pts/74 00:00:00 mpiexec.hydra -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

user 17682 17681 0 14:11 pts/74 00:00:00 /users/home/opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi/intel64/bin/pmi_proxy --control-port login1:36980 --pmi-connect alltoall --pmi-aggregate -s 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1732786525 --usize -2 --proxy-id 0

user 17689 17682 2 14:11 pts/74 00:00:00 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

user 17690 17682 2 14:11 pts/74 00:00:00 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

user 17773 17690 1 14:11 pts/74 00:00:00 /users/home/opt/intel/vtune_amplifier_2018.3.0.558279/bin64/amplxe-runss --ui-output-format xml --ui-output-fd 17 --option-file /users/home/user/my_result.login1.cluster.net/config/runsa.options.11

user 17775 17689 1 14:11 pts/74 00:00:00 /users/home/opt/intel/vtune_amplifier_2018.3.0.558279/bin64/amplxe-runss --ui-output-format xml --ui-output-fd 18 --option-file /users/home/user/my_result.login1.cluster.net/config/runsa.options.10

I'm forced to use ctrl-c to exit:

mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir my_result -- IMB-MPI1 pingpong

[mpiexec@login1.cluster.net] Sending Ctrl-C to processes as requested

[mpiexec@login1.cluster.net] Press Ctrl-C again to force abort

^CCtrl-C caught... cleaning up processes

APPLICATION TERMINATED WITH THE EXIT STRING: job ending due to timeout = -1

Thanks

Vladimir_R_Intel · ‎06-29-2018

Could you try to use a different result directory? If it doesn't help , please set a PERFRUN_DEBUG=1 environment variable, repeat your experiment with a new result directory and upload this result and collected logs? (logs will be located in <result>/log or in working directory).

BR,

Vladimir

Giuseppe_C_ · ‎06-29-2018

it was enough to change the folder result and it worked (worked with PERFRUN_DEBUG and without PERFRUN_DEBUG):

export PERFRUN_DEBUG=1

[login1][/users/home/user/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test-dir-ampl -- IMB-MPI1 pingpong

amplxe: Analyzing data in the node-wide mode. The hostname (login1.cluster.net) will be added to the result path/name.

amplxe: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel modules symbols.

amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.

amplxe: Collection started.

#------------------------------------------------------------

# Intel (R) MPI Benchmarks 2018, MPI-1 part

#------------------------------------------------------------

# Date : Fri Jun 29 14:29:27 2018

# Machine : x86_64

# System : Linux

# Release : 2.6.32-642.6.2.el6.x86_64

# Version : #1 SMP Wed Oct 26 06:52:09 UTC 2016

# MPI Version : 3.1

# MPI Thread Environment:

# Calling sequence was:

# IMB-MPI1 pingpong

# Minimum message length in bytes: 0

# Maximum message length in bytes: 4194304

#

# MPI_Datatype : MPI_BYTE

# MPI_Datatype for reductions : MPI_FLOAT

# MPI_Op : MPI_SUM

#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------

# Benchmarking PingPong

# #processes = 2

#---------------------------------------------------

#bytes #repetitions t[usec] Mbytes/sec

0 1000 0.68 0.00

1 1000 0.73 1.37

2 1000 0.72 2.76

4 1000 0.74 5.42

8 1000 0.74 10.86

16 1000 0.76 21.04

32 1000 0.78 40.95

64 1000 0.80 79.80

128 1000 0.88 144.81

256 1000 0.97 264.89

512 1000 0.97 530.24

1024 1000 1.24 829.14

2048 1000 1.55 1324.69

4096 1000 2.15 1906.86

8192 1000 3.43 2388.33

16384 1000 6.18 2652.03

32768 1000 11.53 2841.12

65536 640 9.09 7209.76

131072 320 15.97 8208.18

262144 160 31.60 8296.44

524288 80 61.82 8480.21

1048576 40 125.11 8381.02

2097152 20 243.30 8619.61

4194304 10 503.66 8327.66

# All processes entering MPI_Finalize

amplxe: Collection stopped.

amplxe: Using result path `/users/home/user/test-dir-ampl.login1.cluster.net'

amplxe: Executing actions 19 % Resolving information for `libittnotify_collecto

amplxe: Warning: Cannot locate debugging information for file `/lib64/ld-2.12.so'.

amplxe: Warning: Cannot locate debugging information for file `/users/home/opt/intel/vtune_amplifier_2018.3.0.558279/lib64/runtime/libittnotify_collector.so'.

amplxe: Executing actions 19 % Resolving information for `IMB-MPI1'

amplxe: Warning: Cannot locate debugging information for file `/users/home/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin/IMB-MPI1'.

amplxe: Executing actions 20 % Resolving information for `libmpi.so.12.0'

amplxe: Warning: Cannot locate debugging information for file `/lib64/libc-2.12.so'.

amplxe: Executing actions 100 % done

Collection and Platform Info

----------------------------

Parameter test-dir-ampl.login1.cluster.net

------------------------ ----------------------------------------------------

Application Command Line IMB-MPI1 "pingpong"

Operating System 2.6.32-642.6.2.el6.x86_64 CentOS release 6.2 (Final)

MPI Process Rank 1

Computer Name login1.cluster.net

Result Size 4418087

Collection start time 12:29:26 29/06/2018 UTC

Collection stop time 12:29:27 29/06/2018 UTC

Collector Type Driverless Perf per-process sampling

CPU

---

Parameter test-dir-ampl.login1.cluster.net

----------------- -------------------------------------------------

Name Intel(R) Xeon(R) E5 Processor code named Jaketown

Frequency 2600002833

Logical CPU Count 16

Summary

-------

Elapsed Time: 0.512

Paused Time: 0.0

CPU Time: 0.664

Average CPU Utilization: 0.720

CPI Rate: 0.907

Event summary

-------------

Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample

---------------------------------- ------------------------- -------------------------------- -----------------

CPU_CLK_UNHALTED.THREAD 1726400000 664 2600000

CPU_CLK_UNHALTED.REF_TSC 1726400000 664 2600000

INST_RETIRED.ANY 1903200000 732 2600000

CPU_CLK_UNHALTED.REF_XCLK 66401992 664 100003

CPU_CLK_UNHALTED.ONE_THREAD_ACTIVE 66401992 664 100003

but now seems that the sep is unloaded (for ampllxe):

amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location

.

[/root]> /users/home/opt/intel/vtune_amplifier_2018/sepdk/src/insmod-sep -q

pax driver is loaded and owned by group "vtune" with file permissions "775".

socperf2_0 driver is loaded and owned by group "vtune" with file permissions "775".

sep4_1 driver is loaded and owned by group "vtune" with file permissions "775".

vtsspp driver is loaded and owned by group "vtune" with file permissions "664".

[athena-login1][/root]> lsmod | grep sep

sep4_1 830967 0

socperf2_0 24366 1 sep4_1

Thanks

Vladimir_R_Intel · ‎06-29-2018

Great! This means that some garbage left from previous launch (I think it was unsuccessful) in the result directory. Please use an unique result directory to prevent such behavior in future. PERFRUN_DEBUG isn't needed anymore.

About the warnings you see during collection:

These warnings don't mean that sep driver is unloaded it just suggests what you have to do to see kernel related info in your result. Otherwise the time spent in kernel will be attributed in [Outside any known module] So if you don't interested in this info just ignore the warning.

BR,

Vladimir

Giuseppe_C_ · ‎06-29-2018

However, before I reloaded the sep kernel module in the login node, the warning "To profile kernel modules during the session" did not show up.

Another observation:

the computational nodes do not have sep modules loaded (i have installed parallel studio on login node in distributed directory), Do I have to manually load the sep module on all compute nodes? (amplxe also starts without kernel module loaded but amplxe return the warning "To profile kernel modules during the session")

Apart from that, on the login node, I still have the problem of the license for "amplxe-cl report"

amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `summary': no valid license can be found (This platform is not supported: your license contains platform restrictions.).

Thanks

PAVEL_G_Intel · ‎06-29-2018

Hi,

Please check that license file is located in /opt/intel/licenses. If it's set env variable and run report:

export INTEL_LICENSE_FILE=/opt/intel/licenses
amplxe-cl report

- Pavel

Giuseppe_C_ · ‎06-29-2018

Sorry,

but I use Intel license manager:

echo $INTEL_LICENSE_FILE

28518@mng2:28518@login2:28518@mng1

I realized that in the folder /opt/intel/licenses of login node, there was an old license, I removed it and now it works.

Apart from that, the compute nodes do not have sep drivers loaded (i have installed parallel studio on login node in distributed directory).

Do I have to manually load sep driver on all the nodes?

Until now, Amplxe also starts without driver sep loaded (apart warning:

amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location)

Thanks

PAVEL_G_Intel · ‎06-29-2018

As I know... Yes, you need to install sep manually.

But I recommend to wait answer form Vladimir next week. I'm not very proficient in mpi profiling.

- Pavel

Giuseppe_C_ · ‎06-29-2018

Ok, I'll wait the answer from Vladimir

I add that:

if I load manually 2018 sep driver for amplxe-2018, are they good for amplxe-2015 too?

Regards

PAVEL_G_Intel · ‎06-29-2018

We have sep version 4 and 3 in the products. They may be loaded to system at the same time. Every instance of VTune Amplifier will use its version.

Remember! The drivers shall not be in use at the same time. So two collections from different versions of VTune will fail.

Vladimir_R_Intel · ‎07-02-2018

Hi Osvaldo,

answering on your question:

>>Do I have to manually load sep driver on all the nodes?

This depends on your needs. I mean that VTune will try to use driverless Perf based approach if it detects that no driver is loaded. And if it cannot use Perf it will generate an appropriate error message describing what you need to do to get an interested collection result. (SEP driver installation should be an option there).

But if you are interested in SEP driver as a collector by some reason then you will need to load it on all the nodes. Please look at the following article for more details https://software.intel.com/en-us/vtune-amplifier-help-driverless-event-based-sampling-collection

BR,

Vladimir

Giuseppe_C_ · ‎07-02-2018

Thanks Vladimir, all clear.

I have tried to load sep4 and sep3 on the same compute nodes, but amplxe-cl don't start:

LOAD SEP4:

cd /users/home/opt/intel/vtune_amplifier_2018/sepdk/src

[root@n300 src]# ./insmod-sep -r

Warning: the following driver(s) were not found loaded in the kernel: sep4_1.

Warning: no vtsspp driver was found loaded in the kernel.

Warning: the following driver(s) were not found loaded in the kernel: socperf2_0.

The PAX service is not loaded anymore.

Checking for PMU arbitration service (PAX) ... not detected.

Attempting to start PAX service ...

Executing: insmod ./pax/pax-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko

Creating /dev/pax device with major number 242 ... done.

Setting group ownership of devices to group "vtune" ... done.

Setting file permissions on devices to "666" ... done.

The pax driver has been successfully loaded.

PAX service has been started.

Checking for socperf driver ... not detected.

Executing: insmod ./socperf/src/socperf2_0-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko

Creating /dev/socperf2_0 base devices with major number 241 ... done.

Setting group ownership of devices to group "vtune" ... done.

Setting file permissions on devices to "666" ... done.

The socperf2_0 driver has been successfully loaded.

Executing: insmod ./sep4_1-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko

Creating /dev/sep4_1 base devices with major number 240 ... done.

Creating /dev/sep4_1 percpu devices with major number 239 ... done.

Creating /dev/sep4_1 percpu devices with major number 238 ... done.

Creating /dev/sep4_1 per package devices with major number 237 ... done.

Setting group ownership of devices to group "vtune" ... done.

Setting file permissions on devices to "666" ... done.

The sep4_1 driver has been successfully loaded.

Checking for vtsspp driver ... not detected.

Executing: insmod ./vtsspp/vtsspp-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko gid=502 mode=0666

The vtsspp driver has been successfully loaded.

LOAD SEP 3:

cd /users/home/opt/intel/vtune_amplifier_2015/sepdk/src

[root@n300 src]# ./insmod-sep3 -r

Warning: no sep3_15 driver was found loaded in the kernel.

Removing vtsspp driver from the kernel ... done.

The vtsspp driver has been successfully unloaded.

Attempting to stop PAX service ...

Removing pax driver from the kernel ... done.

Deleting previously created /dev/pax device ... done.

The pax driver has been successfully unloaded.

PAX service has been stopped.

Checking for PMU arbitration service (PAX) ... not detected.

Attempting to start PAX service ...

Executing: insmod ./pax/pax-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko

Creating /dev/pax device with major number 242 ... done.

Setting group ownership of devices to group "vtune" ... done.

Setting file permissions on devices to "666" ... done.

The pax driver has been successfully loaded.

PAX service has been started.

PAX service is accessible to all users.

Executing: insmod ./sep3_15-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko

Creating /dev/sep3_15 base devices with major number 236 ... done.

Creating /dev/sep3_15 percpu devices with major number 235 ... done.

Setting group ownership of devices to group "vtune" ... done.

Setting file permissions on devices to "666" ... done.

The sep3_15 driver has been successfully loaded.

Checking for vtsspp driver ... not detected.

Executing: insmod ./vtsspp/vtsspp-x32_64-2.6.32-642.6.2.el6.x86_64smp.ko gid=502 mode=0666

The vtsspp driver has been successfully loaded.

KERNEL MODULE LOADED:

[root@n300 src]# lsmod | grep sep

sep3_15 527559 0

sep4_1 830967 0

socperf2_0 24366 1 sep4_1

[root@n300 src]#

EXEC AMPLXE-CL 2018:

[n300][/users/home/USER/]> module load IMPI/intel_mpi_2018 INTEL/intel_xe_2018

[n300][/users/home/USER/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test5 -- IMB-MPI1 pingpong

amplxe: Analyzing data in the node-wide mode. The hostname (n300.cluster.net) will be added to the result path/name.

amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

NMI watchdog timer is enabled. Turn off the nmi_watchdog timer and restart.

amplxe: Error: NMI watchdog timer is enabled. Turn off the nmi_watchdog timer and restart.

( amplxe-cl remains hanging )

EXEC AMPLXE-CL 2015:

[n300][/users/home/USER/]> module load INTEL/intel_xe_2015.3.187 IMPI/intel_mpi_5.0.3.048

[n300][/users/home/USER/]>

[n300][/users/home/USER/]> mpirun -np 2 amplxe-cl -collect advanced-hotspots -result-dir test6 -- IMB-MPI1 pingpong

amplxe: The analyzed process has a rank of 0. This rank will be added to the result path/name.

amplxe: The analyzed process has a rank of 1. This rank will be added to the result path/name.

amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /users/home/USER/test6.0 -command stop.

amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /users/home/USER/test6.1 -command stop.

( amplxe-cl remains hanging )

What i am doing wrong?

Thanks

Vladimir_R_Intel · ‎07-02-2018

What user did you launch from? This message seems that VTune failed to disable NMI watchdog. And the only reason I can suppose it happened because of no permissions.

Please try to run a simple no mpi launch. And in case of the same error please try to disable NMI watchdog manually.

Also need to mention that in your second launch with VTune 2015 it will no work so as this version weren't adapted for per node mpi collection, it tried to collect per rank results and corresponding collections conflicted for PMU resources.

Giuseppe_C_ · ‎07-02-2018

I have done other tests, recapitulating:

SEP3 LOADED (only sep3)

- amplxe-cl-2015 works if i don't use mpirun

- amplxe-cl-2015 hangs if I use mpirun

SEP4 LOADED (only sep4)

- amplxe-cl-2018 works and the warning is:

"amplxe: Warning: To enable hardware event-base sampling, VTune Amplifier has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes." (seems correct)

- amplxe-cl-2015 don't works

It seems impossible to use sep3 + sep4 for PE-2015 and PE-2018.

What do you suggest me? Use only sep4 for PE-2018?

Can I analyze code compiled with PE-2015 with amplex-cl-2018?

Best regards

Vladimir_R_Intel · ‎07-02-2018

Osvaldo, thank you for the report and for your time. This is really confused. I'll try to connect experts in SEP driver to this thread to investigate this issue.

And about suggestions, I suggest to use sep4. And yes, you can use VTune Amplifier 2018 to analyze code compiled with PE-2015, there should not be any problem with that.

amplxe-gui - the corresponding command in command line