Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

Using RAPL to read PP0 and DRAM energy on haswell

Sameer_A_
Beginner
3,324 Views

Hello, 

According to the Intel's software developer's manual, RAPL allows up to read energy values for different domains. The PKG(0 or 1) which is the whole socket on a motherboard,  power plans PP0 and PP1 and the DRAM energy. 

I am trying to use that to read the Package energy and the DRAM and PP0 on a server mother board for a E5-2620 v3 (a Haswell xeon with 6 cores). I am using the PAPI component for RAPL.

While I am getting meaningful numbers for both the PACKAGE_ENERGY:PACKAGE0 and the DRAM_ENERGY:PACKAGE0 , the PP0 is always returning zeros. which doesn't seem to me correct for a computationally intensive code given that PP0 is the cores power. 

Any idea on  what  might be causing this? is there anything I need to turn on or off in my BIOS to read these values? any pointer to the solution is appreciated.

 

Thank you,

0 Kudos
1 Solution
McCalpinJohn
Honored Contributor III
3,324 Views

Intel has changed the implementation of RAPL without much documentation.

For the Xeon E5 v3 there are two issues:

  1. The only supported domains are "Package" and "DRAM"
  2. The "Energy Unit" for DRAM is not the same as the value that is read from the RAPL unit register.   Instead, Xeon E5 v3 processors use a DRAM Energy Unit of 15.3 microJoules.  (Ref:  "Intel Xeon Processor E5-1600/2400/2600/4600 v3 Product Families Datasheet, Volume 2: Registers", document 330784 -- section 5.3.2 in Revision 003).

View solution in original post

0 Kudos
12 Replies
McCalpinJohn
Honored Contributor III
3,325 Views

Intel has changed the implementation of RAPL without much documentation.

For the Xeon E5 v3 there are two issues:

  1. The only supported domains are "Package" and "DRAM"
  2. The "Energy Unit" for DRAM is not the same as the value that is read from the RAPL unit register.   Instead, Xeon E5 v3 processors use a DRAM Energy Unit of 15.3 microJoules.  (Ref:  "Intel Xeon Processor E5-1600/2400/2600/4600 v3 Product Families Datasheet, Volume 2: Registers", document 330784 -- section 5.3.2 in Revision 003).
0 Kudos
Sameer_A_
Beginner
3,324 Views

Thank your for your reply.

About the energy unit for the DRAM. I think RAPL wraps that for you and gives you a counter that returns that in Jouls, it also gives you the actual count if you need, rapl:::DRAM_ENERGY:PACKAGE0 for jouls  and rapl:::DRAM_ENERGY_CNT:PACKAGE0 for count.

0 Kudos
SHIH_K_Intel
Employee
3,324 Views

An update for these energy status counters for Intel Xeon processors E5 v3 and E5 v4 will be in the next SDM.

0 Kudos
Mostafa_M_1
Beginner
3,324 Views

Dear All,

I am facing a similar problem with my Intel Core i7-6800k. I am getting All Zeros for DRAM and PP0. Even for the Package reading, I am getting a weird reading of 0.6 Watt and 1.9 J for an intensive program running for 3.3 seconds. I tried a longer benchmark running for 3 minutes but still getting 0.6 Watt and around 111 J.

I get the same readings when I access the package energy register:

/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj

Any help would be appreciated 

 

Regards

Mostafa

0 Kudos
McCalpinJohn
Honored Contributor III
3,324 Views

In general, the DRAM RAPL energy counter is only supported on "server" processors, not on "client" processors -- e.g., Section 14.9.5 of Volume 3 of the SW Developer's Manual.

The PP0 domain seems to be deprecated on the server processors (some of the PP0 registers are still there, but return zero, while other PP0 registers no longer exist), so it is not surprising that it may no longer work on the client parts either.

I have not seen problems with the Package-level RAPL counters on my Xeon E5 and Xeon Phi x200 systems...

0 Kudos
Sameer_A_
Beginner
3,324 Views

I agree with John about the availability of PP0 domain availability in non-server platforms, all what you can get is a Package reading (no finer grain readings possible). 

Weird number you are getting for the PKG reading can be explained by overflow probably, I am not sure how you are exactly reading the registers but RAPL doesn't protect against overflow in the numbers it provides. You might want to perform this test (provided by the PAPI wrapper) to get an idea: https://github.com/tpatki/papi-rapl/blob/master/src/ctests/overflow.c 

 

 

0 Kudos
Kenan_L_
Beginner
3,324 Views

Hi McCalpin,

I know it's an old thread, but could you tell me where did you get the value as 15.3 micro-joule? I found some source code project hard coded this value in their source code as well. But as I searched in the latest Intel manual (changed in Dec 2016), The energy status unit is described like this:

Energy Status Units


Energy related information (in Joules) is based on the multiplier,
1/2^ESU; where ESU is an unsigned integer represented by bits
12:8. Default value is 0EH (or 61 micro-joules)

Thanks,

Kenan

0 Kudos
McCalpinJohn
Honored Contributor III
3,324 Views

The DRAM power unit is specified in Sections 5.3.2 and 5.3.3 of "Intel Xeon Processor E5-1600/2400/2600/4600 v3 Product Families Datasheet, Volume 2: Registers", document 330784, revision 003, June 2015.

0 Kudos
Camilo_Celis_G_
Beginner
3,324 Views

Hi McCalpin,

On your first response you mention some characteristics for the case of the Xeon E5 processor, does it include Xeon E7 v3 processor as well? 

As Sameer A. I am also interested in measuring DRAM energy consumption, and eventually throttle its bandwidth. According to some Intel tools (power_gov, and power_gadget), this can only be done if a specific setting is set in the BIOS but I don't seem to find this option.

Thank you,

Camilo

0 Kudos
McCalpinJohn
Honored Contributor III
3,324 Views

For the Xeon E7, one could imagine searching for "Xeon E7 v3 datasheet" and looking in Section 5.3 of volume 2.

http://www.intel.com/content/www/us/en/processors/xeon/xeon-e7-v3-datasheet-vol-2.html

 

0 Kudos
Marko_S_
Beginner
3,324 Views

Mostafa M. wrote:

Dear All,

I am facing a similar problem with my Intel Core i7-6800k. I am getting All Zeros for DRAM and PP0. Even for the Package reading, I am getting a weird reading of 0.6 Watt and 1.9 J for an intensive program running for 3.3 seconds. I tried a longer benchmark running for 3 minutes but still getting 0.6 Watt and around 111 J.

I get the same readings when I access the package energy register:

/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj

Any help would be appreciated 

 

Regards

Mostafa

Hi Mostafa,

Have you been able to resolve the issue with the weird readings? I have the exact same issue on my i9-7900X CPU. I am getting very low power and energy values for the package when running a heavy workload (https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/746959). I tried reading the rapl values with likwid, turbostat, perf, /sys/class/powercap, msrs, and the power numbers are way too low (up to 4W max). 

As per Intel, my i9-7900X is a desktop CPU (https://www.intel.com/content/www/us/en/products/processors/core/x-series/i9-7900x.html) so I should not expect RAPL DRAM domain to work? Is that correct? Also, as per John's and Sameer's comments on PP0, only package domain should be available? Just making sure that what I see on my system is what I should expect, and I only see package-0. All the tools I use either report only package numbers or package and DRAM, where DRAM is always 0W.

-Marko

0 Kudos
Marko_S_
Beginner
3,324 Views

Hi,

I have been able to resolve the low energy reading issue by resetting the BIOS settings to default.

https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/746959#comment-1914284

0 Kudos
Reply