I ran across a change in the behavior of Xeon E5 v3 processors relative to Xeon E5 v1 processors and am confused about several aspects....
As an experiment I over-wrote the value in MAX_NON_TURBO_RATIO (bits 7:0 of MSR_TURBO_ACTIVATION_RATIO (0x64c)) with the nominal frequency multiplier (instead of the nominal multiplier minus 1). After I did this, I used the Linux "cpufreq" controls to set the target multiplier in IA32_PERF_CTL (MSR 0x199), and HURRAY! the processor actually ran a the nominal frequency without Turbo boosting.
So we arrive at question(s) #1:
While investigating this behavior, I noticed that the MSR_TURBO_RATIO_LIMIT registers (0x1AD, 0x1AE, 0x1AF) are writable (i.e., MSR_PLATFORM_INFO. (MSR 0xCE) is set). I wondered if I could write overwrite these values with *lower* values to limit the maximum amount of Turbo boost for a single core. The Xeon E5-2660 v3 has Turbo ratio values varying from 29 (decimal) for all cores to 33 (decimal) for 1 core, so I wrote 0x1d (29 decimal) into all of the fields. I then used Linux "cpufreq" to request the maximum available frequency (0x21 = 33 decimal), and saw that the system continued to run a single thread at 3.3 GHz instead of the 2.9 GHz that I thought would be the limit based on MSR_TURBO_RATIO_LIMIT.
So we arrive at questions #2:
Hello Dr. McCalpin,
I'll looking for the right folks to answer your questions. The 1st question will probably have a relatively simple answer. The answer to the 2nd question will probably be much more complicated. At one time the turbo behavior was controlled by 3 sets of registers: MSRs, MMIO and PECI.
Hello Dr. McCalpin,
For question 1), what is the system vendor and bios version you are using? I assume the TURBO_ACTIVATION_RATIO_Lock bit is not set, correct?
We should have question 2) shortly.
These are all Dell systems, mostly R630's with various processor models, but also including an R430 and a C4130. Some of the nodes have "alpha" BIOS revisions but most have production-level (but possibly not current) BIOS revisions. An example of a production-level BIOS is reported by "lshw":
vendor: Dell Inc.
physical id: 0
capabilities: isa pci pnp upgrade shadowing cdboot bootselect edd int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int9keyboard int14serial int10video acpi usb biosbootspecification netboot uefi
The MSR_PLATFORM_INFO (MSR 0xCE) is:
# ./rdmsr -p 0 -x -0 0xce
The MSR_TURBO_ACTIVATION_RATIO (0x64c) does not have the lock bit set (and I was able to change the value and this change did change the behavior as described in my initial note):
# ./rdmsr -p 0 -x -0 0x64c
One of the nodes includes a processor that does not support Turbo mode. This is a Dell R430 running an alpha-level (0.3.9, dated 2014-08-06) BIOS, and it shows the same pattern -- the "maximum non-turbo ratio" value in MSR_TURBO_ACTIVATION_RATIO is one less than the corresponding value in MSR_PLATFORM_INFO. On this non-Turbo-capable box, the MSR_TURBO_RATIO_LIMIT register shows 0x1010101010101010, as expected, but it is not clear that this MSR is actually doing anything here. CPUID Leaf 6, EAX bit 1 is cleared on this box (and not on any of the others), which indicates that "opportunistic processor operation" is not supported (SWDM, Vol 3, section 126.96.36.199).
$ rdmsr -p 0 -x -0 0xce
rdmsr -p 0 -x -0 0x64c
Hello Dr. McCalpin,
For question 1, I think that a new bios version from Dell would fix the issue.
For question 2, we are still working on it. It works when done correctly but the method is not yet disclosed. We are working on approval for disclosure.
Fortunately issue #1 is not relevant to our standard workloads (which are run with the cpufreq governor set to "performance") and when we need specific CPU frequency control it is easy to change MSR_TURBO_ACTIVATION_RATIO to have the desired value.
Issue #2 is also not relevant to our standard workloads, but it would help provide a greater range of controllable frequencies when we are doing performance sensitivity testing (which I do a lot).
Hello Dr. McCalpin,
For issue 2, we will have an update coming to the SDM vol 3.
You can see the attached screenshot for the change to MSR_TURBO_RATIO_LIMIT2 on Intel® Xeon® Processor E5 v3 Family.
Basically the update indicates that bit 63 has to be set for any changes to be 'seen' by the processor. So you would modify MSR_TURBO_RATIO_LIMIT, MSR_TURBO_RATIO_LIMIT1, and lastly MSR_TURBO_RATIO_LIMIT2.
On Intel(r) Xeon(r) Processor E5 v2 family, the semaphore is bit 63 in MSR_TURBO_RATIO_LIMIT1.
Hopefully this will resolve issue 2.
There are not really any "bugs" here -- just a BIOS misconfiguration and some confusion on my part.
I was running a lot of "performance vs frequency" sensitivity studies at this time. I had no trouble getting the processor to run at fixed frequencies that were below the "nominal" frequency. I also wanted to force the processors to run at a frequency between the "nominal" frequency and the "max turbo" frequency (to give me more data points in the "performance vs frequency" plot). Message #7 in this thread showed me how to do that, and it worked fine.
Unless you are doing frequency-scaling studies, the default behavior (run as fast as possible) is usually preferred.
I'm not sure if I'll get a reply because this thread is so old but I have the same problem but with an e5-2650 v2. No matter what I do the settings my 8 core clock is limited to 3GHz. Even though I have tons of power and thermal head room. I can lower the 8 core clock to 2.8 "1C" in the 0x1AD MSR. Then flick MSR_TURBO_RATIO_LIMIT1 63 and it drops to 2.8 but if I set it to 3.2 "20" and hit the switch it tops out at 3.0. :( bummer. Any ideas here?
According to the "Intel® Xeon® Processor E5 v2 Product Family: Specification Update" (document 329189, revision 014, September 2015), the Xeon E5-2650 v2 has a base frequency of 2.6 GHz, with 5-8 steps of Turbo. So I would expect the all-core active frequency to reach 3.1 GHz with appropriate power and cooling.
I don't think I have access to any Ivy Bridge Xeons any more, and did not do a lot of detailed work on that generation.... The Xeon E5-2680 (v1) processors had no trouble sustaining their max Turbo frequency (2.7 base + 0.4 = 3.1) with all cores active. With an aggressive air-cooled system it was difficult to find any workload that would cause them to ever drop below 3.1 GHz.