topic [PCM] some KPIs not collected from specific CPUs in Software Tuning, Performance Optimization & Platform Monitoring

[PCM] some KPIs not collected from specific CPUs

panko — Mon, 13 Nov 2023 13:16:50 GMT

Hi all,

We are using PCM to collect data and visualize that in Grafana. In general this work fine. The thing is that we have more that 10 machines and they have different CPUs.

We have:
- Intel Xeon CPU X5675
- Intel Xeon CPU E5-2667 v2
- Intel Xeon CPU E5-2640 v3
- Intel Xeon CPU E5-2698 v4

Only from Intel Xeon CPU E5-2667 v2 and Intel Xeon CPU E5-2640 v3 we are not able to collect some of the values. In particular Instructions Retired Any and Clock Unhalted Thread. What can be the issue?

In the attachments please find output of the 'pcm' command both for the affected machines and some other without issues. Affected are: fmighgm006, fmighgm007, fmighgm008. OK - fmighgm001 and fmighgm010. For fmighgm007 there are also some other commands executed, like: 'pcm-core' and 'pcm-memory'.

Indeed, in the log file I can see fmighgm007: Instructions retired: 0,
while on working machines something like this: Instructions retired: 26 G

The same for FREQ (which seems to be related to unhalted clock ticks): 0 for fmighgm007 and values like 0.39 on ok nodes.

If anything more is needed please let me know.

Thanks!

Re: [PCM] some KPIs not collected from specific CPUs

Roman_D_Intel — Mon, 13 Nov 2023 13:30:55 GMT

pcm-sensor-server can't be run in parallel with other pcm and non-pcm tools using the performance monitoring units (PMU). Make sure no other PMU tools are running in parallel.

Re: [PCM] some KPIs not collected from specific CPUs

dpetrin — Tue, 12 Dec 2023 14:54:27 GMT

Hi, one of the devs that setup the environment for PCM here!
My colleagues and I made some analysis about this point and we didn't find any other process that uses PMU (aside from the PCM we setup); also, processes are pretty much aligned across every server that we have in production, so we don't think that's the issue.
As a further note, the problem is present only on servers with 'Xeon(R) CPU E5-2667 v2' and 'Xeon(R) CPU E5-2640 v3' CPUs.
Please let us know if there's any additional info you need, we are available for any further clarification about the environment.

Re: [PCM] some KPIs not collected from specific CPUs

Roman_D_Intel — Wed, 13 Dec 2023 13:49:13 GMT

are you running pcm-sensor-server inside a docker container?

Re: [PCM] some KPIs not collected from specific CPUs

dpetrin — Thu, 14 Dec 2023 13:57:36 GMT

no, pcm is running natively on a RHEL7.9 OS; we installed it via a .rpm package

Re: [PCM] some KPIs not collected from specific CPUs

Roman_D_Intel — Thu, 14 Dec 2023 14:07:48 GMT

then what might help is to run with the PCM_NO_PERF=1 env variable:

https://github.com/intel/pcm/blob/master/doc/ENVVAR_README.md

Re: [PCM] some KPIs not collected from specific CPUs

dpetrin — Tue, 12 Mar 2024 10:05:03 GMT

Hi, sorry for the late response, we normally don't have root access so we had request it to perform the suggestion you gave us.

Unfortunately, running pcm with the env variable PCM_NO_PERF=1 set did not solve the issue, in fact it was even worse as the daemon wouldn't even start.

In the attached text file you can find the whole logged pty, but I'll try to summarize:

we initially tried to launch (as a command on the terminal) pcm without the PCM_NO_PERF env variable; this was just a check to be sure that everything was as before and values were missing (as expected)
after that, we tried to launch pcm from the shell once again, but this time with the PCM_NO_PERF=1 set; this returned an error saying that the PMU was occupied by another application
we stopped the pcm daemon we had running on the server (note that with this daemon running the first step did not return an error)
repeating the 2nd step, the error returned is still the same
at the end of the file you can find BIOS info about the server

We also tried to run the pcm daemon with the PCM_NO_PERF=1 set, but it returned the same error (PMU is occupied by other application) once again. Lastly, the error suggested to run pcm with the '-r' option: we did NOT run with this as this would reset PMU configuration and we didn't want to do that in a production server.

Is there anything else we can try?

Re: [PCM] some KPIs not collected from specific CPUs

dpetrin — Tue, 23 Apr 2024 14:34:58 GMT

Hello,

is there some more info we can provide? Are there updates on this issue?

Re: [PCM] some KPIs not collected from specific CPUs

Roman_D_Intel — Wed, 24 Apr 2024 10:08:13 GMT

unfortunately there is no other way around. You need to use the -r option. BTW: Linux perf tool resets the PMU configuration all the time without asking when one starts profiling with Linux perf tool. PCM tries to follow the PMU sharing guidelines and asks the user: https://cdrdv2-public.intel.com/727001/pmu-sharing-guidelines.pdf

Re: [PCM] some KPIs not collected from specific CPUs

dpetrin — Tue, 15 Apr 2025 14:00:41 GMT

Hi,

sorry for the absence in the past year, but quite a lot went on and this got somewhat pushed to the side.

Anyway, thanks for the last suggestion; finally a couple of weeks ago we got around to try that, but unfortunately even by resetting the PMU configuration with the '-r' option, nothing changed (see screenshot below). We also stopped any other process that we think could "collide" during this test.

At this point, is there anything else that we could do?

Re: [PCM] some KPIs not collected from specific CPUs

Roman_D_Intel — Tue, 15 Apr 2025 14:43:31 GMT

Hi,

you can now try the following step by step:

get/install the latest pcm version from github.com/intel/pcm
run it like this: "PCM_NO_PERF=1 pcm -r "