amplxe: Error: Ftrace is already in use.

[psing51]

Hi,
i am trying to run vtune amplifier 2019u2 to collect system-overview as - 

export NPROCS=36 
export OMP_NUM_THREADS=1 
mpirun -genv OMP_NUM_THREADS $OMP_NUM_THREADS -np $NPROCS  amplxe-cl -collect system-overview  -result-dir /home/puneet/run_node02_impi2019_profiler_systemoverview/profiles/attempt1_p${NPROCS}_t${OMP_NUM_THREADS}  -quiet $INSTALL_ROOT/main/wrf.exe

I had collected hpc-performance data without any issue. Afterwards , i ran aforementioned command but had to kill it (result dir was incorrect.). when i re-ran the amplxe-cl, i am getting following error messages - 
 

amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first. 
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first. 
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.

I have tried deleting the /home/puneet/run_node02_impi2019_profiler_systemoverview/profiles/* and i have also rebooted the node.
even then those error messages are showing up.

I tried running "-collect io", here is what i got - 

amplxe: Warning: Tracing of NVMe devices is available for Linux kernel 3.19.0 and higher. The kernel version of your target is lower. I/O data for your NVMe devices may be unavailable in the analysis result unless back-porting of NVMe driver is done.
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.
amplxe: Warning: Tracing of NVMe devices is available for Linux kernel 3.19.0 and higher. The kernel version of your target is lower. I/O data for your NVMe devices may be unavailable in the analysis result unless back-porting of NVMe driver is done.
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.
amplxe: Error: Ftrace is already in use. Make sure to stop previous collection first.

i am able to understand warning messages as i have older kernel version - 3.10.0-957.el7.x86_64

Then on same node i ran general-exploration , and though there are some warning messages on stdout, the collection seems to be working fine - 

amplxe: Warning: The analysis type 'general-exploration' is deprecated. Use 'uarch-exploration' analysis type instead. See more details with 'amplxe-cl -help collect uarch-exploration'.
amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
 starting wrf task            2  of           40
....

This seems to be an issue only with "system-overview" profile
Please advice.

[Vladimir_R_Intel]

Hi Puneet,

This is a known issue. Your assumption is right it relates to system-overview and disk io analyses (all analyses that uses ftrace). As a possible workaround you can gather required collections from one rank, I mean you can make something like:

if [ $PMI_RANK="1" ]
then
    amplxe-cl -collect system-overview  -result-dir /home/puneet/run_node02_impi2019_profiler_systemoverview/profiles/attempt1_p${NPROCS}_t${OMP_NUM_THREADS}  -quiet $INSTALL_ROOT/main/wrf.exe

else

$INSTALL_ROOT/main/wrf.exe
fi

Also could you please describe your case why you're interested in system-overview in MPI flow?

BR,

Vladimir

[psing51]

thank you vladimir for reply,  I will try your suggestions for system-overview (and io analysis?)

The wrf.exe generates ~12G .nc file every minute - on lustre fs (similar timing on NFS). I was trying to optimize it further, hence, trying disk-io analysis for wrf application. The disk-io was unavailable with 2019u2 as i got -

amplxe: Fatal error: Cannot find the analysis type. Check input parameters or reinstall the product. Available analysis types:
        hotspots
.....

so assuming "io" as equivalent of "disk-io", i unsuccesfully ran the amplxe analysis.

As aforementioned analysis failed, so,  out of curiosity i tried system-overview.
i had never used these two collection methods before.

[Vladimir_R_Intel]

I'm not sure that io analysis will help you. It is mostly helpful for analyzing local physical disks and network utilization.

In case you want to find some utilization holes and fill them by some computing I think it will be better to use "Threading" analysis, it will show you IO operations as wait time with detailed info by synchronization object.

If you are interested in understanding Lustre FS utilization I can suggest you to try our tech preview feature showing work with Lustre FS. It is available with hpc-performance analysis:

mpirun -genv OMP_NUM_THREADS $OMP_NUM_THREADS -np $NPROCS  amplxe-cl -c h-p -knob parallel-fs-collection=true ...

Lustre related info will be available on Summary tab, on Timeline on Bottom-up tab and in the Bottom-up grid grouping by Parallel FS.

We will be very appreciated for any feedback :)