Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

Change in Turbo policy for Xeon E5 v3?

McCalpinJohn
Honored Contributor III
2,818 Views

I ran across a change in the behavior of Xeon E5 v3 processors relative to Xeon E5 v1 processors and am confused about several aspects....

  • On Xeon E5 v1 (Sandy Bridge EP) and Xeon E5 v3 (Haswell EP) processors, the maximum non-turbo clock multiplier ratio is contained in bits 15:8 of MSR_PLATFORM_INFO (MSR 0xCE).
    • This defines the rate at which the TSC increments, and matches the "nominal" frequency of the processor.
  • The Linux "cpufreq" controls can be used to set specific target ratios in bits MSR 0x199 (IA32_PERF_CTL)
    • For frequencies at or below the "nominal" multiplier in MSR_PLATFORM_INFO, this results in fixed-frequency operation at the specified frequency.
    • For frequencies above the nominal multiplier in MSR_PLATFORM_INFO, the OS programs the highest allowable value into these fields -- i.e., the maximum single-core Turbo ratio from bits 7:0 of MSR_TURBO_RATIO_LIMIT (MSR 0x1AD)
      • The hardware then provides the highest frequency that it is able to provide, subject to the number of cores in use (as defined in the various bit fields of MSR_TURBO_RATIO_LIMIT (MSR 0x1AD), MSR_TURBO_RATIO_LIMIT1 (MSR 0x1AE), and MSR_TURBO_RATIO_LIMIT2 (MSR 0x1AF).
  • Xeon E5 v3 (Haswell EP) has a new MSR called MSR_TURBO_ACTIVATION_RATIO (MSR 0x64c)
    • Bits 7:0 are labelled "MAX_NON_TURBO_RATIO"
      • This looks like it should be the same as bits 15:8 of MSR_PLATFORM_INFO
    • BUT! on all of my Xeon E5 v3 systems (4 different models), the new register is programmed to a value that is one unit less than the corresponding value in MSR_PLATFORM_INFO
    • As a consequence, when I program the *nominal* multiplier into IA32_PERF_CTL (MSR 0x199), the hardware sees that this value is larger than the "MAX_NON_TURBO_RATIO" in MSR_TURBO_ACTIVATION_RATIO (0x64c) and assumes that I am requesting Turbo operation, so it increases the frequency to the maximum available.
    • Perversely, this means that the nominal frequency of the processor is the one frequency that I cannot make the processor run at!

As an experiment I over-wrote the value in MAX_NON_TURBO_RATIO (bits 7:0 of MSR_TURBO_ACTIVATION_RATIO (0x64c)) with the nominal frequency multiplier (instead of the nominal multiplier minus 1).  After I did this, I used the Linux "cpufreq" controls to set the target multiplier in IA32_PERF_CTL (MSR 0x199), and HURRAY! the processor actually ran a the nominal frequency without Turbo boosting.

So we arrive at question(s) #1:

  • Is it Intel's intent that the "Maximum Non-Turbo Ratio" in MSR_TURBO_ACTIVATION_RATIO be set to a different value than the "Maximum Non-Turbo Ratio" in MSR_PLATFORM_INFO, or was this an accident?  (Or perhaps a vendor decision to force Turbo performance even when the user thought they requested non-Turbo operation?)

 

While investigating this behavior, I noticed that the MSR_TURBO_RATIO_LIMIT registers (0x1AD, 0x1AE, 0x1AF) are writable (i.e., MSR_PLATFORM_INFO.[28] (MSR 0xCE) is set).   I wondered if I could write overwrite these values with *lower* values to limit the maximum amount of Turbo boost for a single core.  The Xeon E5-2660 v3 has Turbo ratio values varying from 29 (decimal) for all cores to 33 (decimal) for 1 core, so I wrote 0x1d (29 decimal) into all of the fields.  I then used Linux "cpufreq" to request the maximum available frequency (0x21 = 33 decimal), and saw that the system continued to run a single thread at 3.3 GHz instead of the 2.9 GHz that I thought would be the limit based on MSR_TURBO_RATIO_LIMIT.

So we arrive at questions #2:

  • Does the hardware actually pay attention to changes in the values in MSR_TURBO_RATIO_LIMIT (MSR 0x1AD)?
    • If so, did I need to follow a different procedure?

 

    11 Replies
    Patrick_F_Intel1
    Employee
    2,817 Views

    Hello Dr. McCalpin,

    I'll looking for the right folks to answer your questions. The 1st question will probably have a relatively simple answer. The answer to the 2nd question will probably be much more complicated. At one time the turbo behavior was controlled by 3 sets of registers: MSRs, MMIO and PECI.

    Pat

    Patrick_F_Intel1
    Employee
    2,817 Views

    Hello Dr. McCalpin,

    For question 1), what is the system vendor and bios version you are using? I assume the TURBO_ACTIVATION_RATIO_Lock bit is not set, correct?

    We should have question 2) shortly.
    Pat

    0 Kudos
    McCalpinJohn
    Honored Contributor III
    2,819 Views

    These are all Dell systems, mostly R630's with various processor models, but also including an R430 and a C4130.   Some of the nodes have "alpha" BIOS revisions but most have production-level (but possibly not current) BIOS revisions.   An example of a production-level BIOS is reported by "lshw":

         *-firmware
              description: BIOS
              vendor: Dell Inc.
              physical id: 0
              version: 1.0.4
              date: 08/28/2014
              size: 64KiB
              capacity: 15MiB
              capabilities: isa pci pnp upgrade shadowing cdboot bootselect edd int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int9keyboard int14serial int10video acpi usb biosbootspecification netboot uefi

    The MSR_PLATFORM_INFO (MSR 0xCE) is:

    # ./rdmsr -p 0 -x -0 0xce
    00080c33f3811900


    The MSR_TURBO_ACTIVATION_RATIO (0x64c) does not have the lock bit set (and I was able to change the value and this change did change the behavior as described in my initial note):

    # ./rdmsr -p 0 -x -0 0x64c
    0000000000000018

    One of the nodes includes a processor that does not support Turbo mode.  This is a Dell R430 running an alpha-level (0.3.9, dated 2014-08-06) BIOS, and it shows the same pattern -- the "maximum non-turbo ratio" value in MSR_TURBO_ACTIVATION_RATIO is one less than the corresponding value in MSR_PLATFORM_INFO.  On this non-Turbo-capable box, the MSR_TURBO_RATIO_LIMIT register shows 0x1010101010101010, as expected, but it is not clear that this MSR is actually doing anything here.  CPUID Leaf 6, EAX bit 1 is cleared on this box (and not on any of the others), which indicates that "opportunistic processor operation" is not supported (SWDM, Vol 3, section 14.3.2.1).

    $ rdmsr -p 0 -x -0 0xce
    00080c33f3811000
    rdmsr -p 0 -x -0 0x64c
    000000000000000f

     

    0 Kudos
    Patrick_F_Intel1
    Employee
    2,819 Views

    Hello Dr. McCalpin,

    For question 1, I think that a new bios version from Dell would fix the issue.

    For question 2, we are still working on it. It works when done correctly but the method is not yet disclosed. We are working on approval for disclosure.

    Pat

    0 Kudos
    McCalpinJohn
    Honored Contributor III
    2,819 Views

    Thanks!

    Fortunately issue #1 is not relevant to our standard workloads (which are run with the cpufreq governor set to "performance") and when we need specific CPU frequency control it is easy to change MSR_TURBO_ACTIVATION_RATIO to have the desired value.

    Issue #2 is also not relevant to our standard workloads, but it would help provide a greater range of controllable frequencies when we are doing performance sensitivity testing (which I do a lot).

    0 Kudos
    Patrick_F_Intel1
    Employee
    2,819 Views

    Hello Dr. McCalpin,

    For issue 2, we will have an update coming to the SDM vol 3.

    You can see the attached screenshot for the change to MSR_TURBO_RATIO_LIMIT2 on Intel® Xeon® Processor E5 v3 Family.

    Basically the update indicates that bit 63 has to be set for any changes to be 'seen' by the processor. So you would modify MSR_TURBO_RATIO_LIMIT, MSR_TURBO_RATIO_LIMIT1, and lastly MSR_TURBO_RATIO_LIMIT2.

    On Intel(r) Xeon(r) Processor E5 v2 family, the semaphore is bit 63 in MSR_TURBO_RATIO_LIMIT1.

    Hopefully this will resolve issue 2.

    Pat

     

    0 Kudos
    McCalpinJohn
    Honored Contributor III
    2,819 Views

    Thanks!   This appears to work on my Xeon E5 v3 systems.   I will do more testing next week.

    0 Kudos
    Huang__Skycat
    Beginner
    2,819 Views

    Wait,is this the reason of Xeon E5 v3 can always running on max turbo?(thanks to this bug,we got expensive pre owned e5 v3 on taobao)

    0 Kudos
    McCalpinJohn
    Honored Contributor III
    2,819 Views

    There are not really any "bugs" here -- just a BIOS misconfiguration and some confusion on my part.

    I was running a lot of "performance vs frequency" sensitivity studies at this time.   I had no trouble getting the processor to run at fixed frequencies that were below the "nominal" frequency.   I also wanted to force the processors to run at a frequency between the "nominal" frequency and the "max turbo" frequency (to give me more data points in the "performance vs frequency" plot).  Message #7 in this thread showed me how to do that, and it worked fine. 

    Unless you are doing frequency-scaling studies, the default behavior (run as fast as possible) is usually preferred.

    0 Kudos
    Stokes__Grant
    Beginner
    2,819 Views

    I'm not sure if I'll get a reply because this thread is so old but I have the same problem but with an e5-2650 v2. No matter what I do the settings my 8 core clock is limited to 3GHz. Even though I have tons of power and thermal head room. I can lower the 8 core clock to 2.8 "1C" in the 0x1AD MSR. Then flick MSR_TURBO_RATIO_LIMIT1 63 and it drops to 2.8 but if I set it to 3.2 "20" and hit the switch it tops out at 3.0. :( bummer. Any ideas here?

    0 Kudos
    McCalpinJohn
    Honored Contributor III
    2,819 Views

    According to the "Intel® Xeon® Processor E5 v2 Product Family: Specification Update" (document 329189, revision 014, September  2015), the Xeon E5-2650 v2 has a base frequency of 2.6 GHz, with 5-8 steps of Turbo.  So I would expect the all-core active frequency to reach 3.1 GHz with appropriate power and cooling.

    I don't think I have access to any Ivy Bridge Xeons any more, and did not do a lot of detailed work on that generation....  The Xeon E5-2680 (v1) processors had no trouble sustaining their max Turbo frequency (2.7 base + 0.4 = 3.1) with all cores active.  With an aggressive air-cooled system it was difficult to find any workload that would cause them to ever drop below 3.1 GHz.

    0 Kudos
    Reply