topic Hello Everyone, in Software Tuning, Performance Optimization & Platform Monitoring

Error reading llc_misses event in Xeon D-1540

Roberto_R_ — Wed, 24 Jun 2015 23:05:01 GMT

Hello everyone,

I am working in a tool that permits to access the different hardware events through performance counters (PMC). This tools works great I have tested in several Intel processors, SandyBridge, Haswell and Haswel-EP. Now I am working with a Broadwell processor that has some new cache monitoring features I need to work with.

Trying my tool in this processor I found that the events, described in 64-ia-32-architectures-software-developer-manual-325462.pdf Table 19.1, LLC Reference (2EH, Umask 4FH) and LLC Misses (2EH, Umask 41H) report the same number.

I though this could be an error from my tool so I tried perf and I got the same error. Also I can use only 4 programable PMCs, it is supposed to have 8 programmable PMCs, if I tried to use a 5th PMC it returns zero, same happend with perf.

My processor is:

Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Vendor : GenuineIntel
Family : 6
Model : 6
Stepping: 2
Type : OEM

The perf output is:
$ perf stat -I 1000 -e instructions:u,cycles:u,cache-references:u,cache-misses:u ./benchmarks/spec2006/mcf06
# time counts unit events
1.000170323 1.513.051.613 instructions:u
1.000170323 1.966.105.738 cycles:u
1.000170323 82.844.475 cache-references:u
1.000170323 82.845.042 cache-misses:u
2.000390792 985.271.999 instructions:u
2.000390792 2.597.388.127 cycles:u
2.000390792 77.201.120 cache-references:u
2.000390792 77.200.636 cache-misses:u
3.000546036 928.783.029 instructions:u
3.000546036 2.597.151.649 cycles:u
3.000546036 73.133.856 cache-references:u
3.000546036 73.133.954 cache-misses:u
4.000699910 906.864.990 instructions:u
4.000699910 2.597.354.693 cycles:u
4.000699910 73.593.433 cache-references:u
4.000699910 73.593.252 cache-misses:u

In my tool the LLC_misses is exactly the same that LLC_references, in perf there is a little difference because of the way perf works. I think these is a bug in the processor or an error in the manual. Does somebody know about these error? Thanks in advance for your suggestions and comments.

Hello Everyone,

Roberto_R_ — Fri, 26 Jun 2015 21:54:48 GMT

Hello Everyone,

I talked with the vendor and he thinks my processor is damaged. I have to return my server and he will replace it, actually because the processor is solded on the motherboard he has to replace the whole system.

With event code=0x2e not matters the umask used (0x4f, 0x41, 0x71, 0xf, 0x1 or 0x7f) the counter always returned the same value, not a fixed value but the same for all events.

Replace my server will take some weeks, I live in Spain and the processor has to be imported from USA. Some of you can access to this processor and could confirm that my problems are hardware bugs because a faulty processor?

My problems are:

1. Event 0x2e always returned the same value no matters the umask used

2. I have access only to 4 programmable counters (PMC).

Thanks for your feedback.

Volume 3 of the Intel SW

McCalpinJohn — Mon, 29 Jun 2015 18:48:35 GMT

Volume 3 of the Intel SW Developer's Manual (Table 18-37) says that you can only use all 8 counters per core if the system is booted with HyperThreading (sometimes called "Logical Processors") disabled. With HyperThreading enabled, each "Logical Processor" can only access 4 of the 8 programmable core performance counters.

Some estimates of the expected cache miss rate values can be obtained by looking at the extensive measurements of the SPEC 2006 benchmarks reported at http://www.jaleels.org/ajaleel/workload/

For the mcf benchmark running the reference input set, the chart above suggests a cache miss rate of about 60 misses per 1000 instructions for a 256 KiB unified L2 cache and a miss rate of about 12 misses per 1000 instructions for a 12 MiB unified L3 cache. Looking at the results above, the first trial gave just under 83M misses in 1.5B instructions, which is a rate of about one miss per 55 instructions. This is close to what you would expect without an L3 cache, so I would be concerned about the hardware. Unfortunately these processors are so new that there is not much documentation for other counters (such as memory controller counters) that might be useful for checking to see if the L3 is working correctly.

An alternative check would be to use the SPEC CPU2006 cache hit rate charts above to pick a code and input set that are expected to have a modest miss rate for the 256 KiB L2, but (very close to a) zero miss rate for a 12 MiB L3 cache. Examples include any of the input sets for 401.bzip2, any of the input sets for 464.h264ref, either of the input sets for 456.hmmer, the "checkspam" or "splitmail" inputs for 400.perlbench, and the 482.sphinx3 benchmark.

I would also run these tests with the hardware prefetchers disabled to see if that changes the counts in a useful way. This is described at https://software.intel.com/en-us/articles/disclosure-of-hw-prefetcher-control-on-some-intel-processors

John thanks for your answer.

Roberto_R_ — Mon, 29 Jun 2015 20:04:30 GMT

John thanks for your answer.

I actually have HT disabled,

$ lscpu | grep -i -E "^CPU$s$:|core|socket"
CPU(s): 8
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1

I tried 464.h264ref because is the one with the smallest working set. I took a sample every second.

nsample pid Instr Cycles LLC_requests LLC_Misses L2_requests L2_misses LLC_Usage (Bytes)
1 1042 5493409057 2111392416 12256433 12256433 39790416 12257650 10125312
2 1042 7325193416 2586067847 19203738 19203738 50093591 19204266 10846208
3 1042 7401938326 2576543380 20886234 20886234 49512747 20886660 11862016
4 1042 7473959529 2576339358 21857777 21857777 49560115 21858200 12255232

The LLC seems to work correctly, I am showing the result for the LLC_usage from Intel CMT, and the benchmark uses the whole cache, I can modify the way allocation through Intel CAT, it also seems to work well. When I modify the ways assigned to the running benchmark the LLC_usage changes. Of course could be great to have access to one off chip event to check it.

I need the LLC_misses for estimate the MPKI curves. But I don't know why that event is not working when everything seems to work great. Thanks again.

These results are certainly

McCalpinJohn — Mon, 29 Jun 2015 21:24:24 GMT

These results are certainly suggestive of a broken performance counter event.... :-(

The Broadwell processors should support the "Offcore Response Counter" events (0xB7 and 0xBB), which can be programmed to count LLC misses. It requires a fair investment in time to understand how to program the counters (and some combinations that seem reasonable don't appear to work), but it may provide an alternative source of information.

From the 464.h264ref results, the L2 access rate is definitely higher than Jaleel's results -- you are seeing about 7 L2 accesses per 1000 instructions, while Jaleel's charts show only about 2.6 L1 misses per 1000 instructions. The difference could be due to L1 prefetches (it is very hard to tell if the performance counters differentiate between L1 demand misses and L1 hardware prefetch misses), or perhaps due to significantly different code generation. (I don't know the h246ref code well enough to guess whether it is amenable to significant variability in code generation. Looking at instructions is always a bit risky, since compilers may optimize for execution time in ways that give very different instruction counts.) The L2 miss rate is also higher than Jaleel's results, but not by such a large ratio.

John thanks again for your

Roberto_R_ — Mon, 13 Jul 2015 18:15:37 GMT

John thanks again for your answer.

Our hardware vendor realized that this problem also becomes apparent in identical machines they have for testing (same CPU, same motherboard). We were wondering if you have you observed this performance-counter issue in other Broadwell processors at Intel.

Hi everyone,

Ji_X_ — Mon, 26 Oct 2015 05:44:49 GMT

Hi everyone,

I met with this problem too. I want to know can we fix the problem now?

Thanks,

Hello Xu,

Roberto_R_ — Tue, 27 Oct 2015 16:18:25 GMT

Hello Xu,

We have not solved the problem, we contacted our hardware vendor, they tested some machines they have for testing and the error is also present in that machines, It seems to be a hardware error.

Roberto