topic An update for these energy in Software Tuning, Performance Optimization & Platform Monitoring

Using RAPL to read PP0 and DRAM energy on haswell

Sameer_A_ — Tue, 05 Apr 2016 22:47:57 GMT

Hello,

According to the Intel's software developer's manual, RAPL allows up to read energy values for different domains. The PKG(0 or 1) which is the whole socket on a motherboard, power plans PP0 and PP1 and the DRAM energy.

I am trying to use that to read the Package energy and the DRAM and PP0 on a server mother board for a E5-2620 v3 (a Haswell xeon with 6 cores). I am using the PAPI component for RAPL.

While I am getting meaningful numbers for both the PACKAGE_ENERGY:PACKAGE0 and the DRAM_ENERGY:PACKAGE0 , the PP0 is always returning zeros. which doesn't seem to me correct for a computationally intensive code given that PP0 is the cores power.

Any idea on what might be causing this? is there anything I need to turn on or off in my BIOS to read these values? any pointer to the solution is appreciated.

Thank you,

Intel has changed the

McCalpinJohn — Mon, 11 Apr 2016 08:35:40 GMT

Intel has changed the implementation of RAPL without much documentation.

For the Xeon E5 v3 there are two issues:

The only supported domains are "Package" and "DRAM"
The "Energy Unit" for DRAM is not the same as the value that is read from the RAPL unit register. Instead, Xeon E5 v3 processors use a DRAM Energy Unit of 15.3 microJoules. (Ref: "Intel Xeon Processor E5-1600/2400/2600/4600 v3 Product Families Datasheet, Volume 2: Registers", document 330784 -- section 5.3.2 in Revision 003).

Thank your for your reply.

Sameer_A_ — Mon, 11 Apr 2016 19:54:33 GMT

Thank your for your reply.

About the energy unit for the DRAM. I think RAPL wraps that for you and gives you a counter that returns that in Jouls, it also gives you the actual count if you need, rapl:::DRAM_ENERGY:PACKAGE0 for jouls and rapl:::DRAM_ENERGY_CNT:PACKAGE0 for count.

An update for these energy

SHIH_K_Intel — Thu, 09 Jun 2016 20:27:03 GMT

An update for these energy status counters for Intel Xeon processors E5 v3 and E5 v4 will be in the next SDM.

Dear All,

Mostafa_M_1 — Thu, 02 Feb 2017 16:26:50 GMT

Dear All,

I am facing a similar problem with my Intel Core i7-6800k. I am getting All Zeros for DRAM and PP0. Even for the Package reading, I am getting a weird reading of 0.6 Watt and 1.9 J for an intensive program running for 3.3 seconds. I tried a longer benchmark running for 3 minutes but still getting 0.6 Watt and around 111 J.

I get the same readings when I access the package energy register:

/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj

Any help would be appreciated

Regards

Mostafa

In general, the DRAM RAPL

McCalpinJohn — Thu, 02 Feb 2017 17:36:48 GMT

In general, the DRAM RAPL energy counter is only supported on "server" processors, not on "client" processors -- e.g., Section 14.9.5 of Volume 3 of the SW Developer's Manual.

The PP0 domain seems to be deprecated on the server processors (some of the PP0 registers are still there, but return zero, while other PP0 registers no longer exist), so it is not surprising that it may no longer work on the client parts either.

I have not seen problems with the Package-level RAPL counters on my Xeon E5 and Xeon Phi x200 systems...

I agree with John about the

Sameer_A_ — Thu, 02 Feb 2017 18:39:10 GMT

I agree with John about the availability of PP0 domain availability in non-server platforms, all what you can get is a Package reading (no finer grain readings possible).

Weird number you are getting for the PKG reading can be explained by overflow probably, I am not sure how you are exactly reading the registers but RAPL doesn't protect against overflow in the numbers it provides. You might want to perform this test (provided by the PAPI wrapper) to get an idea: https://github.com/tpatki/papi-rapl/blob/master/src/ctests/overflow.c

Hi McCalpin,

Kenan_L_ — Mon, 13 Feb 2017 23:05:14 GMT

Hi McCalpin,

I know it's an old thread, but could you tell me where did you get the value as 15.3 micro-joule? I found some source code project hard coded this value in their source code as well. But as I searched in the latest Intel manual (changed in Dec 2016), The energy status unit is described like this:

Energy Status Units

Energy related information (in Joules) is based on the multiplier,
1/2^ESU; where ESU is an unsigned integer represented by bits
12:8. Default value is 0EH (or 61 micro-joules)

Thanks,

Kenan

The DRAM power unit is

McCalpinJohn — Tue, 14 Feb 2017 17:36:38 GMT

The DRAM power unit is specified in Sections 5.3.2 and 5.3.3 of "Intel Xeon Processor E5-1600/2400/2600/4600 v3 Product Families Datasheet, Volume 2: Registers", document 330784, revision 003, June 2015.

Hi McCalpin,

Camilo_Celis_G_ — Mon, 10 Apr 2017 07:43:28 GMT

Hi McCalpin,

On your first response you mention some characteristics for the case of the Xeon E5 processor, does it include Xeon E7 v3 processor as well?

As Sameer A. I am also interested in measuring DRAM energy consumption, and eventually throttle its bandwidth. According to some Intel tools (power_gov, and power_gadget), this can only be done if a specific setting is set in the BIOS but I don't seem to find this option.

Thank you,

Camilo

For the Xeon E7, one could

McCalpinJohn — Mon, 10 Apr 2017 21:21:12 GMT

For the Xeon E7, one could imagine searching for "Xeon E7 v3 datasheet" and looking in Section 5.3 of volume 2.

http://www.intel.com/content/www/us/en/processors/xeon/xeon-e7-v3-datasheet-vol-2.html

Quote:Mostafa M. wrote:

Marko_S_ — Thu, 19 Oct 2017 20:49:19 GMT

Mostafa M. wrote:

Dear All,

I am facing a similar problem with my Intel Core i7-6800k. I am getting All Zeros for DRAM and PP0. Even for the Package reading, I am getting a weird reading of 0.6 Watt and 1.9 J for an intensive program running for 3.3 seconds. I tried a longer benchmark running for 3 minutes but still getting 0.6 Watt and around 111 J.

I get the same readings when I access the package energy register:

/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj

Any help would be appreciated

Regards

Mostafa

Hi Mostafa,

Have you been able to resolve the issue with the weird readings? I have the exact same issue on my i9-7900X CPU. I am getting very low power and energy values for the package when running a heavy workload (https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/746959). I tried reading the rapl values with likwid, turbostat, perf, /sys/class/powercap, msrs, and the power numbers are way too low (up to 4W max).

As per Intel, my i9-7900X is a desktop CPU (https://www.intel.com/content/www/us/en/products/processors/core/x-series/i9-7900x.html) so I should not expect RAPL DRAM domain to work? Is that correct? Also, as per John's and Sameer's comments on PP0, only package domain should be available? Just making sure that what I see on my system is what I should expect, and I only see package-0. All the tools I use either report only package numbers or package and DRAM, where DRAM is always 0W.

-Marko

Hi,

Marko_S_ — Mon, 30 Oct 2017 18:21:46 GMT

Hi,

I have been able to resolve the low energy reading issue by resetting the BIOS settings to default.

https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/746959#comment-1914284