Avoid that Performance Counters gets Re-Programmed

[giovanni_m_1]

Hi Everybody,

I have the following problem that I hope you can help me to address: suppose that I can program and then acquire in a trusted way the CPU Performance Counters (fixed and non-fixed). How can I be sure that the non-fixed performance counters get re-programmed by an attacker?

There isn't any timestamp left by the CPU on the last program operation? Or any capabilities/tricks that allow me to do that?

Thank you in advance,

Giovanni

[McCalpinJohn]

Performance counters can only be programmed using the WRMSR instruction, which can only operate in kernel mode.  If your attacker is able to execute code in kernel mode, then you have bigger problems than the programming of the performance counters.

If you are concerned about an attacker activating a kernel API that programs the performance counters, then you need to harden or disable that API.  If this is a particular concern, you could instrument the kernel to log (or block) any WRMSR to the performance monitor control registers.   If the attacker can run arbitrary code in kernel mode then they can execute WRMSR in their own code, but this would at least identify attempts to reprogram the counters via existing APIs (which presumably don't provide sufficient privilege for the attacker to hunt down and delete the log messages).

If you can't trust the OS, Intel VMX technology can be used to trap WRMSR instructions and turn over handling to the hypervisor.  Whether that actually provides more control than being able to recompile the kernel is a point that can probably be debated....

[giovanni_m_1]

Hi John,

First of all let me thank you. I have read many replies coming from you in the forum that have been really helpful for me to understand in the proper way the performance counters. For what about the question, suppose a situation in which the "attacker" is the Cloud Provider itself that wants to modify the performance metrics because he does not respect a SLA. Now, assuming that I have the possibility to trust an area in the cloud provider server in which securely acquire the counters and make some checks, I need to find the way of knowing when the counters are re-programmed.

When you say "but this would at least identify attempts to reprogram the counters via existing APIs" what do you mean? Of what APIs you're talking about? The problem is that the only solution should be enforced at CPU level. Because from ring0 someone could make directly an assembly instruction and program the registers without going through the APIs or I'm wrong? There isn't any possiblity to activate an interrupt in the CPU when the IA32_PERFEVTSEL registers are programmed?

I'm spending my time on the vol3 programming reference but I can't figure out my problem. 

Best,

[McCalpinJohn]

In the "tacc_stats" (https://github.com/TACC/tacc_stats) infrastructure that we use to monitor all the jobs on our systems, we attempt to deal with this issue by capturing the performance monitor control registers every time we read the counters.  This is done by a program run by the root user, which uses the /dev/cpu/*/msr device drivers to read both the performance counter counts and the performance counter control registers.  Our post-processing code automatically ignores any fields for which the corresponding performance counter control register(s) have been modified from their original state (that we set up at the beginning of each batch job).

So we do not attempt to *prevent* users from changing the performance counters (which they can do through either the "perf" subsystem or the Intel SEP subsystem), but we assume that if they do use the counters for a different purpose, they will not return the programming to its original state, so we will always be able to identify cases where we need to throw out the data.

I don't think that either the "perf" subsystem or the Intel SEP subsystem have functionality that allows them to restore the original performance counter programming once they have finished using the counters -- at least I have not seen such functionality so far. 

In Volume 3 of the Intel Architectures SW Developer's Manual, Table 35-2 shows a new MSR 0x392 IA32_PERF_GLOBAL_INUSE that can be used to help coordinate performance counter use.  This assumes cooperative use, I think, so may not help your use case.   I assume that this MSR shows up starting in Skylake processors -- I have not had a chance to check Broadwell yet, but it is not in Haswell or KNL.