PMU counter in use but I cannot find vtune or aps processes.

[oleotiger]

vtune: Warning: EMON Collector Message: Warning: PMU counter(s) [GP0 GP1 GP2 GP3 GP4 GP5 GP6 GP7 FP0 ] is/are already in use by other software and the data collected on these counter(s) may be incorrect
vtune: Error: EMON Collector Message: ERROR - A collection is already in progress - Abort current run.
aps Error: Cannot run the collection.

When I analysis an application with aps and  the application fails to run, I get errors as shown above after restarting the application with aps again.

I didn't find any processes with words like 'aps' or 'vtune'.  Why are there still some processes that occupy PMU counters?

How should I reset the PMU counter or find processes that occupy PMU counters?

[JananiC_Intel]

Hi,

Thanks for posting in Intel forums.

Could you share your machine details?

Regards,

Janani Chandran

[oleotiger]

OS: centos 7.6 kernel 3.10
cpu: 2 sockets with each cascadelake 6248r 24core
512G memory
Any more information needed?

[JananiC_Intel]

Hi,

We will check on this internally and let you know.

Regards,

Janani Chandran

[oleotiger]

Any ideas?

At the present stage, we have to reboot to reset the counter to avoid occupation of unknown processes.

It's really time consuming....

[Kirill_U_Intel]

Hi, you could check sep, emon or amplxe-perf processes too.

[oleotiger]

I checked all of them. There is no aps/perf/emon/sep-like processes.

More details:

When I killed my application, there are many processes `[aps] <defunct>` left. And I killed them all with `kill -9`. Will this be the reason?

[nicejunjie]

I'm having exactly the same problem.  APS worked for many jobs, but suddenly all failed for similar error: 

>aps hostname 

vtune: Error: EMON Collector Message: ERROR - A collection is already in progress - Abort current run.

aps Error: Cannot run the collection. 

And I don't have any related process running at all. I run it on compute nodes of a big HPC, all the user processes got cleaned each time the job is finished.   

Any solution or trick to get around this?  

[oleotiger]

Any ideas?

[Mariya_P_Intel]

Sorry for a delay with response.

Please provide additional info:

• vtune version you use (I would recommend to use the latest from https://software.intel.com/content/www/us/en/develop/articles/oneapi-standalone-components.html#vtune)
• the command line you use to run vtune or aps
• the list of all running processes when the issue occurred
• aps output or aps html report

Thanks, Mariya

[Mariya_P_Intel]

Hi, we have not heard from you in a while. I assume the issue is solved or became unactual and closing this case.

Fell free to start a new thread in case of further assistance need.

Thanks, Mariya

[CornelisNetworks]

Since this is the highest ranked answer on google but doesn't actually include the answer, 

# rmmod vtsspp

on each affected machine resolved the problem for me.

[vsol_ll]

Hi, I have a pretty similar one even when using the vtune-self-checker.sh .

1.Warning: PMU counter(s) [GP0 GP3 FP0 FP1 ] is/are already in use by other software and the data collected on these counter(s) may be incorrect
2. vtune: Error: 0x40000024 (No data) -- No data is collected
3. vtune: Error: Cannot load data file `/tmp/vtune-tmp-vsolontsov/self-checker-2022.02.12_10.40.36/result_ah_with_stacks/data.0/20522.perf' (Data file is corrupted).

Note `/tmp` has enough space.

The self-check log is attached (hopefully). Could you please suggest?