Uncore frequency on Haswell

[BGoel]

Can someone explain how the processor controls the uncore frequency. I understand that on Haswell microarchitecture, the uncore is on a separate clock domain than the processor cores. And while the core frequency can be controlled by OS dynamically (when Speedstep is enabled), I am not sure how is the uncore frequency is controlled. Is it possible for OS to control the uncore frequency? I am guessing not. And if not, is it possible to at least get information about the current frequency at which uncore is operating?

[Bernard]

 >>>Is it possible for OS to control the uncore frequency>>>

I suppose if CPU has appropriate MSR register for controlling Uncore frequency then it can be possible write/read to that register from the kernel mode.

Some information about the Haswell uncore frequency related mostly to overclocking

http://www.overclock.net/t/1401976/the-gigabyte-z87-haswell-overclocking-oc-guide

[McCalpinJohn]

The "client" and "server" Haswell parts probably have different mechanisms for monitoring and control.

On the server parts (Xeon E5-2600 v3) I enable the fixed-function cycle counter in the UBox (set bit 22 in U_MSR_PMON_FIXED_CTL, MSR 0x703), and then read values from U_MSR_PMON_FIXED_CTR (MSR 0x704).   I read the counter, sleep for a few seconds, read the counter again and compute the average Uncore frequency.  (It is important that there be at least one process running on the chip to prevent the chip from going into a package C-state, which appears to halt the uncore clocks.)

There are several BIOS settings that influence the processor's control of the Uncore frequency on the Xeon E5 v3 parts.   The names of the settings may depend on the vendor.  On the Dell R630 servers that I have been working with the BIOS options that influence the uncore frequency are "CPU Power Management", "Turbo Boost", "Energy Efficient Turbo", "C States", and (of course) "Uncore Frequency".   Some other settings that might influence Uncore frequency include "CPU Power Management" and "Energy Efficient Policy".

When I first starting using these nodes the BIOS was controlling the CPU frequency and the Uncore frequency matched the CPU frequencies.  After some fiddling, I switched to OS control of the CPU frequency and set the Uncore to run at its maximum frequency.  The Uncore still slows down to match the core frequency when the system is in a power-throttling or thermal-throttling state.  

I have not found a way to explicitly control the uncore frequency, but these BIOS controls give me enough flexibility to do most of the tests I need to do.

[YZhan69]

I'm also trying to measure the Uncore frequency to see if that explains why we're seeing less speedup than expected  going from a 2.1 GHz  to a 2.6 GHz Sandy Bridge.

John, appreciate  you sharing how to do it. I've tested it and it works for Haswell 2680 v3 (measured 3 GHz), but not Sandy Bridge. Do you know how I can make it work for the later?

I'm looking at  http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf

and see the U_MSR_PMON_FIXED_CTL register mentioned, but no MSR index. I think in another thread, you said the uncore is a separate device from the cores and might need to be programmed via the PCI config registers?

[BGoel]

Strange! For me, the same method worked on both Haswell(i7-4770) and Ivy bridge(i7-3770). I don't have access to sandy bridge machine otherwise I would have tried it out.

[YZhan69]

I tried "rdmsr 0x703" on a Xeon E5-2658 and get this:

rdmsr: CPU 0 cannot read MSR 0x00000703

[McCalpinJohn]

The MSRs for the UBOX performance counters are at a different address on the Sandy Bridge EP.

The Xeon E5-2600 Uncore Performance Monitoring Guide (document 327043) says that the addresses are:

• U_MSR_PMON_UCLK_FIXED_CTL   MSR 0x0C08
• U_MSR_PMON_UCLK_FIXED_CTR  MSR 0x0C09

The "enable" function of the control MSR is bit 22 -- this is the same bit as used in the Haswell EP.

Ivy Bridge EP and Sandy Bridge EP use the same MSRs -- Haswell EP is the one that is different.

[francesco_b_1]

About to control the uncore frequency.

In this white paper it is mentioned that the uncore frequency is controlled by the "UNCORE_RATIO_LIMIT" MSR.

I searched a lot for this MSR in the datasheets of my CPU (Xeon E5-2699) but I did not found any reference about it.

Where can I found this info?

[McCalpinJohn]

The only way I have found to control the Uncore frequency on the Xeon E5-2600 v3 processors is indirectly via BIOS options.

On the Dell systems that I have been using the BIOS provides "Uncore Frequency" options of "dynamic" and "maximum". 

• When set to "dynamic", the uncore frequency appears to match the frequency of the fastest core (which is the policy used on Xeon E5-2600 v1 Sandy Bridge EP processors). 
• When set to "maximum", the uncore frequency typically remains fixed.  On a Xeon E5-2660 v3 processor, I measure 3.0 GHz in this mode.  This does not correspond directly to any of the limits on core frequency (2.6 GHz nominal, 3.3 GHz single-core non-AVX max Turbo, 3.1 GHz single-core AVX max Turbo, 2.9 GHz all-core max Turbo (with or without AVX)).
  • You may need to keep a thread running on the chip to prevent the chip from dropping into a package C-state (which stops the incrementing of the UBox cycle counter).
  • When the chip is running in power-constrained mode (e.g., running LINPACK on more than 1/2 of the cores), the uncore frequency is reduced to match the (throttled) core frequencies.  This can be as low as 2.2 GHz for the Xeon E5-2660 v3, e.g., when running a long (>>10 second) LINPACK test using all cores in a socket.

I also found it necessary to disable the "Energy Efficient Turbo" option in the BIOS to enable full frequency operation.  With "Energy Efficient Turbo" enabled, the processor will not boost frequency if this boost does not help performance (presumably via measurement of the running average Instructions Per Cycle by the uncore Power Control Unit).

I verified that I can reduce the Turbo boost available at various active core counts by modifying the contents of the MSR_TURBO_RATIO_LIMIT MSRs (1ADH, 1AEH, 1AFH) as described in Table 35-27 of the latest version of Volume 3 of the SWDM (document 325384, revision 055, June 2015) and in the Intel forum discussion at https://software.intel.com/en-us/forums/topic/543895 (specifically Message #7).   I have not tested the uncore frequency under these conditions to see if there is any change to the behavior in either "Dynamic" or "Maximum" uncore frequency modes.