MSRs to query processor settings

[JJoha8]

Hi all,

over the recent years, the number of BIOS settings that affect the performance properties of Intel Xeon processors has been steadily increasing. As a result, measurements are becoming more and more useless without the corresponding processor settings, which reduces the chances of reliably (re)producing measurements.

To address this problem, we'd like to include functionality to dump the processor's current settings in the LIKWID toolkit, since rebooting the system and inspecting the BIOS for the current settings (some of which are not even displayed in 3rd party BIOSs) is not an option in productive environments.

For the moment, we have identified the following (BIOS) settings to have a significant impact on performance (individual settings result in changes of 5-20% but cumulatively the impact can be much higher!): 

- Uncore frequency: MSRs have been published, 0x620 includes upper and lower bound between which the Uncore frequency can vary

- COD/SNC mode: Whether this mode is enabled or disabled can be inferred indirectly from the system's NUMA properties. Nevertheless, it would be nice to know the MSR that contain's the on/off bit for that setting, for it seems that some BIOS vendors have trouble complying with the instructions laid out in the EDS (e.g., we have a SKL in a Supermicro board where enabling SNC and disabling deal CL allocation leads to an undefined configuration in which one NUMA node is present per processor that only uses one of a processor's two memory controllers)

- QPI Snoop mode: Significant impact on latency and sustained memory bandwidth but no MSRs known/publicly available

- EPB: No MSRs known/publicly available

- New LLC HWPF on SKL: No MSRs known/publicly available, although all other MSR for the other four HWPFs is documented here: https://software.intel.com/en-us/articles/disclosure-of-hw-prefetcher-control-on-some-intel-processors

- SMT: Can be indirectly determined via node topology but no MSRs known/publicly available

- Dead LC allocation: No MSR known/publicly available

We'd greatly appreciate it if the corresponding MSRs are made public. If this is not an option, please consider providing an Intel-made tool to dump the settings.

[McCalpinJohn]

The Energy Performance Bias can be controlled by IA32_ENERGY_PERF_BIAS (MSR 0x1B0), as described in Section 14.3 of Volume 3 of the Intel Architectures SW Developer's Manual.  According to Volume 4, this is now an "architectural" register.   Of course, to keep us confused, the BIOS can disable this register, or can configure the processor to ignore it.  The latter case is irritating because you have to create and run tests to see if changing the value in this MSR changes the processor's behavior.

If "Hardware-Controlled Performance States" (HWP) is enabled, the Energy-Performance Bias (referred to as the "Energy Performance Preference") can be controlled using IA32_HWP_REQUEST (MSR 0x774) and/or IA32_HWP_REQUEST_PKG (MSR 0x772).   This is documented in section 14.4 of Volume 3 of the Intel Architectures SW Developer's Manual.   The BIOS can disable HWP as well, though I think this is less common.    I use this approach because the Linux CPU frequency control interfaces are poorly designed and are seriously broken on the CentOS 7 releases that I tested (probably 7.2 and 7.3).

I suspect that CoD/SNC and QPI Snoop Mode have to be configured before handing off control to the OS, so it would not be possible to change these settings.  I agree that it would be nice to have read-only access to that configuration information.  

If I recall correctly, on one of our Haswell systems, the configuration information related to DRAM interleaving was in a region "Extended PCI Configuration Space" that was blocked by the BIOS.  Fortunately, I was able to open a UEFI shell from the BIOS prompt, perform a hex dump of the relevant memory-mapped PCI configuration space (before the BIOS locked it), photograph the screen with my cell phone, and then manually decode the settings.  Not the most convenient approach, but at least I was able to find the data....