some uncore PMU event not available on XEON-E5-2670

[Zhu_G_]

Hi community.

I am using RAPL to measure DRAM power consumption on XEON-E5-2670. I checked the arch_soft_dev_manual provided by intel. 

I read the chapter about SNB aval events.

And I set the counter-seleter pair respectively according to the manual.  Then I found that:

UNCORE_DRAM_PAGE_OPEN        Y

UNCORE_DRAM_PAGE_CLOSE      Y

UNCORE_DRAM_PAGE_MISS          N

UNCORE_DRAM_READ_CAS           Y

UNCORE_DRAM_WRITE_CAS         N

UNCORE_DRAM_REFRESH             N

UNCORE_DRAM_PRE_ALL               N

Y means that I can see the counter working, N mean that the event seems to be not aval on my CPU.

It seems not every event is supported on my CPU.Why?

[McCalpinJohn · Accepted solution]

What you are seeing is probably an artifact of the software you are running, and not a limitation in the Xeon E5-2670 processor.   What software are you using? (The names you listed are not the standard names used by the Intel documentation or by any other software I have seen?)

The Xeon E5-2680 systems at TACC are configured to measure:

1. CAS_COUNT.RD                        <-- probably the same as UNCORE_DRAM_READ_CAS above
2. CAS_COUNT.WR                       <-- probably the same as UNCORE_DRAM_WRITE_CAS above
3. ACT_COUNT                              <-- probably the same as UNCORE_DRAM_PAGE_OPEN above
4. PRE_COUNT.PAGE_MISS        <-- probably the same as UNCORE_DRAM_PAGE_MISS above

So two of the events that your software lists as "not supported" certainly work on my systems.   I have many of the other memory controller events as well and have found no obvious problems. Some events are a lot harder to interpret, but the DRAM transactions (CAS, ACT, PRE, REFRESH, etc) all look correct, and the "Major Mode" counters give plausible numbers.

[Zhu_G_]

Hi Dr.B

I also tried the event on table 19-14 on TACC. I got the same result. It seems some event is missing on my machine.

I am considering buring some new machine adopting HASWELL-E processor with DDR4. And I am considering buying intel s2600cwt motherboard. I am also considering dell poweredge R730.

What I am trying to say is that can I measure DRAM power event on these machines I mentioned above?

Since I have read a lot of thread mentioning DRAM power measurement using RAPL, I know about the famous available motherboard intel s2600cp4. But I want to do it on a new platform using HASWELL-E cpu such as intel XEON-E5-2603. How can I make sure I can do the job on the new platform?

[McCalpinJohn]

On the TACC Sandy Bridge systems the uncore performance counters are not currently accessible to users using the "perf" interface.  They are available using VTune.

I have been looking at RAPL measurements on some early Haswell EP systems.  The RAPL counters MSR_PKG_ENERGY_STATUS and MSR_DRAM_ENERGY_STATUS are available and give reasonable results.  For example on a system with two Xeon E5-2690 v3 processors (12 core, 135 Watt TDP), I ran an Intel MKL DGEMM benchmark using all 12 cores on socket 0 (leaving socket 1 idle).   Measuring the PKG and DRAM counters before and after a 270 second run (100 iterations of DGEMM with N=8000), gave

• Socket 0 PKG:    130.0 Watts average
• Socket 0 DRAM:   39.4 Watts average
• Socket 1 PKG:      28.4 Watts average
• Socket 1 DRAM:   12.0 Watts average

The socket values seem quite reasonable.  The Socket 0 PKG power is close to the 135W TDP, while the Socket 1 PKG power is close to the 30W specification for the package C3 state (section 2.10 of the Xeon E5 v3 datasheet, volume 1, document 330783).

The DRAM values don't look reasonable -- the MSR_DRAM_POWER_INFO register shows the power limit for the DRAM domain as 18W, so 39.4W is probably not right.   I am still investigating this.

[Zhu_G_]

Hi Dr.B

I think you might want to check MSR_DRAM_POWER_LIMIT Register.

You can set the limit as 30W to MSR_DRAM_POWER_LIMIT and check the MSR_DRAM_PERF_STATUS MSR. In this way you might see the effect of throtting DRAM power.

[Roman_D_Intel]

Hi,

I have got this note from my colleague Andrey Semin:

"As described in the sections 5.3.2 (DRAM_POWER_INFO) and 5.3.3 (DRAM_ENERGY_STATUS) of Volume 2 (Registers) of Intel Xeon E5-1600 v3 and Intel Xeon E5-2600 v3 (Haswell-EP) Datasheet (Ref 330784-001, Sept.2014) the ENERGY_UNIT for DRAM domain is fixed to 15.3 uJ for Haswell server processors. For all other processors (including Haswell client/mobile SKUs) the ENERGY_UNIT for DRAM domain should be read from PACKAGE_POWER_SKU register (usually value around ~61uJ)."

Please consider it when you use values from MSR_DRAM_ENERGY_STATUS for energy computations.

Best regards,

Roman

[McCalpinJohn]

Thanks for the update, Roman!   I had heard that there was a factor of four discrepancy in the DRAM energy counters, but had not run across this specific bit of documentation.

With the updated value for DRAM_ENERGY_UNIT, the values are quite reasonable.

Updated values and comments:

With two Xeon E5-2690 v3 processors (12 core, 135 Watt TDP), I ran an Intel MKL DGEMM benchmark using all 12 cores on socket 0 (leaving socket 1 idle).   Measuring the PKG and DRAM counters before and after a 270 second run (100 iterations of DGEMM with N=8000), gave

• Socket 0 PKG:    130.0 Watts average
• Socket 0 DRAM:     9.9 Watts average
• Socket 1 PKG:      28.4 Watts average
• Socket 1 DRAM:     3.0 Watts average

The Socket 0 PKG power is close to the 135W TDP, while the Socket 1 PKG power is close to the 30W specification for the package C3 state (section 2.10 of the Xeon E5 v3 datasheet, volume 1, document 330783).

The corrected DRAM power values are in the same range as on my Xeon E5-2680 (Sandy Bridge) systems.   I did not measure the actual DRAM traffic for this case, but based on some other scaling data it appears that this case is fairly memory-bandwidth-intensive -- consistent with the relatively high DRAM power consumption on socket 0.