RAPL on Skylake-X

Marko_S_ · ‎10-18-2017

Hi,

I have been trying to compare the power consumption of a Skylake-X (i9-7900X) and Skylake-S (i7-6700K) CPU while running a multithreaded workload. I decided to rely on RAPL for this purpose (whether or not it is the best way to do it is a different topic). Since I do not have root privileges I read the energy counters in uJ from the Linux power cap interface, directly reading the values from /sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj (Skylake-X).

The first thing I noticed between the two CPUs is the power planes for which the energy numbers are reported.
Skylake-S shows three power planes - core, uncore, and dram (or combined as package-0). Skylake-X shows only one package-0.

Second, I noticed a huge difference when reading the energy values for the packages (package-0) between the two CPUs. While running a compute intensive multithreaded workload (HT off on both CPUs) on Skylake-S, I see 4 500 000 uJ every 100ms (45W), while on Skylake-X I see 400 000 uJ every 100ms (4W). That is while running 8 threads compared to 4 on skylake-S. I use the exact same script to poll the numbers and I double checked my calculations. I did this comparison with multiple tests and I always get way too low readings for the Skylake-X CPU.

My questions:

Which RAPL domains are supported by Skylake-X CPUs? I couldn't find any official documentation on this topic (maybe I wasn't looking in the right place).
Did anyone verify that RAPL energy readings are reasonably correct (they are not 10x off) on Skylake-X CPUs?
Could it be that I have to update the kernel to a newer version to give me correct values for Skylake-X? Or simply use perf or read MSRs directly instead (need root for that)?

Any insights into why I see these strange numbers for Skylake-X would be greatly appreciated.
Thank you. :)

P.S. Here is some info about the two systems I am reading the measurements on.

SYS1 (Skylake-S):
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
stepping : 3
microcode : 0x9e
cpu MHz : 899.902
cache size : 8192 KB
Linux kernel: 4.10.0-27-generic
HT-off; nmi watchdog-off; turbo-boost-dissabled

SYS (Skylake-X):
cpu family : 6
model : 85
model name : Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
stepping : 4
microcode : 0x2000022
cpu MHz : 1218.164
cache size : 14080 KB
Linux kernel: 4.10.0-35-generic
HT-off; nmi watchdog-off; turbo-boost-dissabled

Alexey_B_Intel · ‎10-19-2017

Hi Marko,

I see you are using kernel v 4.10 so updating may bring some improvement in numbers and Perf provides support for reading RAPL counters from user space. Which power metrics specific to your workload are you looking for? Only per-core, memory controller or some other?

Thanks, Alexey

McCalpinJohn · ‎10-19-2017

I have had no trouble with RAPL measurements using direct MSR reads on the Xeon Platinum 8160.

As far as I can tell, Intel has not yet published Volume 2 of the Datasheet for the Xeon Scalable Processor Family, which is where the RAPL infrastructure should be described.

From my experiments, the Xeon Scalable Processors support the Package and DRAM domains, with the DRAM "energy unit" using the special value of 15.3 microJoules (documented as differing from the unit used by the Package domain in the Haswell EP and Broadwell EP processor families) instead of the energy unit described in the RAPL units register (which is 61.04 microJoules on my Xeon Platinum 8160 processors).

Marko_S_ · ‎10-19-2017

Thank you for the answers.

Alexey,
I was trying to get the measurements for core power - PP0 (preferably) or at least package power. That is, read the energy values and from that calculate the power. I will try to update the kernel to 4.13 but until then I will try to read the values with a different tool or by accessing MSRs directly.

John,
I tried to use a sample program from Vince Weaver (http://web.eece.maine.edu/~vweaver/projects/rapl/rapl-read.c) which reads the rapl values either from /sys, perf or MSR linux interface. I got the same results for all three - roughly 4 Joules per second. The RAPL units are consistent with what you see on Xeon Scalable Processors - 61.04 uJ for CPU and 15.26 uJ for DRAM (at least reported by the tool). I did the same measurement on my Skylake-S CPU and I got 60 Joules per second (which also has 61.04 uJ for CPU, but 61.04 for DRAM as well). Output from ./rapl-read -m on Skylake-X below.

EDIT: copied wrong output. Correct output below.

---------------------------------------------------------------------------------

RAPL read -- use -s for sysfs, -p for perf_event, -m for msr

Found Skylake-X Processor type
0 (0), 1 (0), 2 (0), 3 (0), 4 (0), 5 (0), 6 (0), 7 (0)
8 (0), 9 (0),
Detected 10 cores in 1 packages

Trying /dev/msr interface to gather results

Listing paramaters for package #0
DRAM: Using 0.000015 instead of 0.000061
Power units = 0.125W
CPU Energy units = 0.00006104J
DRAM Energy units = 0.00001526J
Time units = 0.00097656s

Package thermal spec: 140.000W
Package minimum power: 57.000W
Package maximum power: 308.000W
Package maximum time window: 0.000000s
Package power limits are unlocked
Package power limit #1: 4095.000W for 0.009766s (enabled, not_clamped)
Package power limit #2: 4095.000W for 0.000977s (enabled, clamped)

Sleeping 1 second

Package 0:
Package energy: 4.177673J
PowerPlane0 (cores): 0.000000J
DRAM: 0.000000J

Marko_S_ · ‎10-20-2017

Hi all,

I tried upgrading to Linux kernel 4.13.8-041308-generic and the results haven't changed. Still a very low reading in energy and power. Here is a reading from LIKWID while under heavy load.

CPU name: Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
CPU type: Intel Skylake SP processor
CPU clock: 3.45 GHz
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Runtime: 2.00067 s
Measure for socket 0 on CPU 0
Domain PKG:
Energy consumed: 5.84808 Joules
Power consumed: 2.92306 Watt
Domain PP0:
Energy consumed: 0 Joules
Power consumed: 0 Watt
Domain DRAM:
Energy consumed: 0 Joules
Power consumed: 0 Watt

jayne_f_ · ‎10-30-2017

I also tried to use the sample from Vince Weaver (http://web.eece.maine.edu/~vweaver/projects/rapl/rapl-read.c). But when I tried to get the power through MSR interface, one problem happened. It said that "rdmsr:open: No such file or directory Trying to open /dev/cpu/0/msr".

Do you kown how to resolve it ? or is there anything to do before use MSR interface?

Thank you

Marko_S_ · ‎10-30-2017

jayne f. wrote:

I also tried to use the sample from Vince Weaver (http://web.eece.maine.edu/~vweaver/projects/rapl/rapl-read.c). But when I tried to get the power through MSR interface, one problem happened. It said that "rdmsr:open: No such file or directory Trying to open /dev/cpu/0/msr".

Do you kown how to resolve it ? or is there anything to do before use MSR interface?

Thank you

You have to load the msr module if you want to be able to read the msr values with rdmsr. Execute sudo modprobe msr
If you don't have sudo permissions you will only be able to read the values from the /sys/class/powercap interface.

Marko_S_ · ‎10-30-2017

I have been able to resolve the issue with low energy readings on Skylake-X. Turns out there was some issue with the BIOS settings. I tried manually turning on an off different settings (HT, Speed Step, Speed Shift, Turbo) to see whether it has an effect on the readings. I also tried enabling/disabling SVID. But none of the combinations showed any difference.

At one point I got a failed POST, after which I had to run BIOS setup and reset to default settings. At which point RAPL started reporting normal energy numbers ~90W for a heavy workload. Afterwards I tried turning on/off HT, SS, SSh, turbo, and there were no discrepancies in the readings. Not sure which setting caused the issue though as I couldn't reproduce the bug. Maybe someone knows which setting could cause the issue? Or could it be vendor specific?

Solution: try resetting BIOS settings to default.

jayne_f_ · ‎11-06-2017

Marko S. wrote:

Quote:

jayne f. wrote:

I also tried to use the sample from Vince Weaver (http://web.eece.maine.edu/~vweaver/projects/rapl/rapl-read.c). But when I tried to get the power through MSR interface, one problem happened. It said that "rdmsr:open: No such file or directory Trying to open /dev/cpu/0/msr".

Do you kown how to resolve it ? or is there anything to do before use MSR interface?

Thank you

You have to load the msr module if you want to be able to read the msr values with rdmsr. Execute sudo modprobe msr
If you don't have sudo permissions you will only be able to read the values from the /sys/class/powercap interface.

Thanks ! and do you know the difference among these three interfaces(powercap,perf and MSR)?

Matteo_Fusi · ‎11-07-2017

jayne f. wrote:

Quote:

Marko S. wrote:

Quote:

jayne f. wrote:

I also tried to use the sample from Vince Weaver (http://web.eece.maine.edu/~vweaver/projects/rapl/rapl-read.c). But when I tried to get the power through MSR interface, one problem happened. It said that "rdmsr:open: No such file or directory Trying to open /dev/cpu/0/msr".

Do you kown how to resolve it ? or is there anything to do before use MSR interface?

Thank you

You have to load the msr module if you want to be able to read the msr values with rdmsr. Execute sudo modprobe msr
If you don't have sudo permissions you will only be able to read the values from the /sys/class/powercap interface.

Thanks ! and do you know the difference among these three interfaces(powercap,perf and MSR)?

- powercap. It is a kernel module of Linux. you can read the files by accessing the folder /sys/class/powercap/intel-rapl . Be sure the module is loaded. ( sudo modprobe intel-rapl ). The folder intel_rapl:0 contains the package power zone. If you enter in this folder there are all the files needed to power-cap the core. The file that shows the consumed energy is (obviously) energy_uj. The subfolder of intel_rapl:0 (intel_rapl:0:0, intel_rapl:0:1, intel_rapl:0:2) are the subzones. They are core, dram and uncore. See the file name to know which zone you're observing.

- perf. I'm not quite sure, but perf has its own structure to read rapl counters. It does not rely on powercap interface because it has its own. If you want to use perf to read RAPL counters use the following command:

sudo perf stat -a -e power/energy-cores/,power/energy-pkg/,power/energy-ram/ ls

to read energy values of cores, package and ram. You can only see precision to 1/100 of joule, so powercap is more precise. Remember that you must have root privileges.

- msr . Read raw Machine Specific Registers. If you have read the Intel SDM you can do it, otherwise is better to use other tools like the two above. MSRs can be read in write in kernel mode, so you must use a module to handle them. You must have msr-tool module loaded.

In my opinion, the best choice is to use powercap interface. A simple and useful tool is powerstat. It supports different ways to collect power consumption, and also RAPL:

https://www.hecticgeek.com/2012/02/powerstat-power-calculator-ubuntu-linux/

It is an ubuntu project, but you can easily compile it and use in you linux distribution (i use it on Arch). Here it is the source:

https://github.com/ColinIanKing/powerstat

If you want to instrument code the most common solution is PAPI. It has a specific module that uses powercap to read energy consumption.

Marko_S_ · ‎11-07-2017

Alternatively you can use LIKWID: https://github.com/RRZE-HPC/likwid

or even turbostat (available in the linux-tools-common package if I am not mistaken): https://www.mankier.com/8/turbostat