Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

RAPL power capping: how does it work

Bo_W_4
Beginner
1,568 Views

Hello,

i'm looking at performance variations of my application under a range of power caps.  With a low power cap, the perfromance is very bad. With a high power cap, the performance is reasonable. Since RAPL controls the power consumption, it's not so intuitive to construct a mathematic model of power consumption and performance, as using DVFS.

My questions is how RAPL caps the power. (Someone told me, if the power cap is high, it does DVFS. If the power limitation is low, it manipulates clock duty cycles. Is that correct? Where can i find official information?)   

Best,

Bo

0 Kudos
4 Replies
McCalpinJohn
Honored Contributor III
1,568 Views

Under every case that I have seen, power capping only uses DVFS (frequency and voltage) scaling.   If the power limit is still exceeded at the "maximum efficiency" frequency (typically 1.2 GHz on server parts), then the problems are more serious.   At least some processors will support frequencies below the "maximum efficiency" frequency, but I don't know if the Power Control Unit will use these or switch to duty cycle modulation (Section 14.5) or clock cycle modulation (Section 14.7.3).  (I often find it difficult to figure out which parts of Chapter 14 of Volume 3 of the Intel Architectures SW Developer's Manual actually apply to my systems.)

It is easy enough to monitor the actual unhalted processor cycles to determine the average frequency.  Changes in retired instruction rate may indicate hardware clock cycle modulation.  For processors before Skylake, you can check the instantaneous clock modulation settings in the IA32_CLOCK_MODULATION register.  Starting with Skylake, you can also monitor the accumulated duty cycle modulation cycle count using the MSR_PKG_HDC_*_RESIDENCY counters.  It may be possible to see clock cycle modulation using the hardware performance counter event CYCLE_ACTIVITY_STALLS.CYCLES_NO_EXECUTE (Event 0xA3, Umask 0x04 -- the name used varies slightly across processor generations), but it would take some careful directed testing to know whether this is a useful measurement.
 

0 Kudos
Bo_W_4
Beginner
1,568 Views

I read through the two sections in the manual, still I'm not very clear what is the difference between duty cycle modulation and clock cycle modulation?

0 Kudos
Bo_W_4
Beginner
1,568 Views

McCalpin, John wrote:

Under every case that I have seen, power capping only uses DVFS (frequency and voltage) scaling.   If the power limit is still exceeded at the "maximum efficiency" frequency (typically 1.2 GHz on server parts), then the problems are more serious.   At least some processors will support frequencies below the "maximum efficiency" frequency, but I don't know if the Power Control Unit will use these or switch to duty cycle modulation (Section 14.5) or clock cycle modulation (Section 14.7.3).  (I often find it difficult to figure out which parts of Chapter 14 of Volume 3 of the Intel Architectures SW Developer's Manual actually apply to my systems.

Hi,

I can conform your observation with compute-bound applications. Setting DVFS to 1.2 GHz the PKG power observed for an application amounts to 45 watts. If power cap is set to 45 watts (no DVFS and turbo boost is on), the observed frequency is more or less 1.2 GHz. DONE!

However, it is quite different for memory-bound applications, such as STREAM. Setting DVFS to 1.2 GHz, the power is 75 watts. Setting power cap directly to 75 watts (no DVFS and turbo boost is on) , the frequency is 1.9GHz, namely much higher than expected.

Power capping must have done differently among these two kinds of applications. Only how? I checked duty cycling and clock modulation. There is no difference, always near 100%.

 Best, 

Bo

0 Kudos
McCalpinJohn
Honored Contributor III
1,568 Views

I seem to recall one case with severe thermal throttling that was running very slowly (as if duty cycle modulation was being employed), but I never saw any changes to the corresponding MSRs.  One possible interpretation is that the MSRs show software-controlled duty-cycle modulation, but may not show hardware-initiated duty-cycle modulation.

Depending on your processor, another place to look for DVFS is in the "uncore" clock.  In one instance, I noticed that when running a power-limited application on a Xeon Platinum 8160 system, the uncore frequency was reduced by a much larger percentage than the core frequency.  

0 Kudos
Reply