amplxe: Error: Permission error mapping pages

[mguy44]

Hello,

I'm a new user of Vtune. I installed as root vtune_amplifier_xe_2015 on a suse 11 linux node.

I try to collect informations about hotspots for an MPI application and I get the following error :

mpirun -np 20 amplxe-cl -r .  -collect advanced-hotspots -- ./mycodeamplxe: Error: Permission error mapping pages.
Consider increasing /proc/sys/kernel/perf_event_mlock_kb,
or try again with a smaller value of -m/--mmap_pages.
(current value: -1)
/opt/intel/vtune_amplifier_xe_2015.1.0.367959/bin64/amplxe-perf-sync: Terminated

<!--break-->

The whole output is the folowing :

mpirun -np 20 amplxe-cl -r .  -collect advanced-hotspots -- ./mycode
amplxe: The analyzed process has a rank of 8. This rank will be added to the result path/name.
(twenty times)

amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /absolute/path/run.7 -command stop.
(twenty times)

zero-sized file (/absolute/path/run.4/data.0/temp.sync.perf), nothing to do!
(twenty times).


amplxe: Error: Permission error mapping pages.
Consider increasing /proc/sys/kernel/perf_event_mlock_kb,
or try again with a smaller value of -m/--mmap_pages.
(current value: -1)
/opt/intel/vtune_amplifier_xe_2015.1.0.367959/bin64/amplxe-perf-sync: Terminated
(twenty times)

I check the perf_event_mlock_kb :

more /proc/sys/kernel/perf_event_mlock_kb
516

which is what I can expect. But I don't know, and don't find what to do with the error concerning the mapping pages.

Thank you for your help.

Regards,

   Guy.

[Jackson_M_Intel]

Hello,

Are you trying to do the collection with the Linux Perf driver or the SEP driver that is included in VTune?

[David_A_Intel1]

@Jackson, It looks like the sep driver is not installed, so perf was used.

@myguy44, What is the underlying file system?  I know of a file system that doesn't support memory mapping and causes problems for VTune Amplifier.  What system are you running it on and what file system is active?

[mguy44]

Hello,

Thank you for your answers.

@Jackson, I think I use this command, a part of VTune :

> which amplxe-cl
/opt/intel/vtune_amplifier_xe_2015.1.0.367959/bin64/amplxe-cl

@MrAnderson, the operating system is Suse Linux Enterprise 11 SP3 (x86_64). I work on a panfs filesystem over InfinyBand. Moreover, I copy the needed files in a directory created in the /tmp of the running node (in fact everything is done on a single node, VTune is directly installed on it). The execution of the copy of the program in the /tmp produces the same error.

Regards,

[David_A_Intel1]

@mguy44, thanks.  It's not where VTune Amplifier is installed that is important, but where the project and results files are written.  But, maybe that is what you meant?

So, panfs is some kind of parallel file system?  Try pointing the results directory to /tmp (i.e., -r /tmp/r1).

[Dmitry_P_Intel1 · Accepted solution]

Hello,

The point is that both SEP and perf collection under VTune has limitation that PMU events can  be collected only for one rank on a node. 

In general case if you need to collect PMU event based collection (like advanced hotspots) for several ranks of one of the nodes there is a way to collect this for the whole node like:

mpirun -np 19 ./mycode : -np 1 amplxe-cl -r . -collect advanced-hotspots -analyze-system -- ./mycode

In this case all the ranks on the node where the rank under amplxe-cl was launched will be in the result file.

Please note on one limitation with this approach - itt notify calls will be collected only for one rank was pointed as application to launch and triggered the system wide analysis.

If you do the MPI run only on one node - you can simply try the following:

amplxe-cl -r . -collect advanced-hotspots -- mpirun -np 20 ./mycode

In this case all the spawned ranks will be considered as follow children of MPI run and placed in the result (with itt notify calls also).

Hope this helps, Regards, Dmitry

[mguy44]

hello,

thank you for your answers.

@MrAnderson, I try in the /tmp filesystem and it doesn't work

@dmitry-prohorov, the following command works correctly:

amplxe-cl -r . -collect advanced-hotspots -- mpirun -np 20 ./mycode

I can look at the results through the gui. So it seems OK.

N.B.: I do not change anything in the settings of the OS or in my environment.

Thank you for your help.

Regards,

[David_A_Intel1]

That doesn't make any sense to me, why it would get rid of the "Permission error mapping pages" error messages. :\

[mguy44]

I do not understand too.

[David_A_Intel1]

Just for everyone's info and reference, we now understand that this message appears because of simultaneous collection of more than one “perf record” for a non-root user when there are subsequent simultaneous runs.  Thus, if you can't install the SEP driver, VTune Amplifier will drop back to the perf collection that is included in the Linux installation.  If more than one run is kicked off, as in the case of having VTune Amplifier launch mpirun and profile all the processes that it starts, then only the first run can collect data via perf and all the other runs fail, causing the mmap_pages message.

We are investigating and determining if we can workaround this limitation.