Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

Disabling turbo boost for only some cores on a socket

Travis_D_
New Contributor II
1,432 Views

I would like to test a single-threaded workload and I've already done what I can at the OS level to reduce noise on my system (isocpus, nohz_full, etc, etc). I have a 4-core i7-6700HQ, with HT disabled. 

It works great when I disable turbo completely (via the BIOS, or dynamically via MSR 0x1a0 (aka IA32_MISC_ENABLE)) - I can usually get noise-free performance measurements, with 99.99% stability (i.e., the variance is roughly 0.01% of the total measurement).

Once turbo is enabled, however, the measurements vary a lot - it gets at least two orders of magnitude worse, and sometimes much worse (i.e., the measurements vary by at least 1% and sometimes up to 10%).

The culprit seems to be not thermal or power-induced variations in turbo on the core running the benchmark, but rather other cores occasionally leaving the halt state, which causes the benchmark core to drop to a lower frequency because the allowed turbo multipliers vary by the number of active cores (35x/33x/32x/31x, respectively for 1/2/3/4 cores active).

So when turbo "kicks in" on any other core. the effective frequency on the benchmark core drops, for e.g., to 3300 MHz when a second core comes online. I would like to disable turbo *only* on the non-benchmark cores, with the idea of keeping the benchmark core in constant idle.

I've tried all of the following:

  1. Using IA32_MISC_ENABLE to disable turbo only on specific cores. The result is that when disabled on any core, turbo is disabled for all cores. I guess that makes sense, since this control is listed as having "package" scope in the developer manuals.
  2. Setting bit 32 of IA32_PERF_CTL MSR, which is supposed to allow per-core control of turbo, exactly for benchmark/characterization reasons. While a readback of the MSR shows the bit as set, it doesn't have any effect on turbo as far as I can tell. It isn't clear to me from the manual, but perhaps this only works on family_model 06_0F (Core 2?).
  3. Using the IA32_HWP_REQUEST MSR to set any/all of the "desired performance", "min/max performance" and "energy performance preference" to various values. At first, writing this MSR didn't seem to have any effect, but once I wrote values to all 4 cores, it suddenly took effect - but at the same values for all cores. I couldn't exactly figure out when/why writes would take effect, but after that I could usually adjust the values by writing a single MSR (e.g., to Core 3), but it would always take effect across all cores. I have HWP enabled.

Any ideas?

 

0 Kudos
4 Replies
McCalpinJohn
Honored Contributor III
1,432 Views

Have a look at MSR_TURBO_RATIO_LIMIT (0x1ad).  This initially contains the maximum Turbo frequency available for using various numbers of cores.   On many systems it is writable, and allows you to specify lower maximum Turbo ratios than the defaults.   I have not tried this on a Skylake processor, but it works as expected on Xeon E5 v3 (Haswell EP), allowing me to set maximum Turbo ratios that are in between the nominal frequency and the Turbo maximum frequency.

You may also want to look to see if MSR_TURBO_ACTIVATION_RATIO (0x64c) exists on your processor.   This will not override the MSR_TURBO_RATIO_LIMIT, but if it is present and programmed, it may change a specific ratio request into a max Turbo ratio request, which can be confusing....

0 Kudos
sbhal1
New Contributor I
1,432 Views

If your processor is Haswell, then try loading acpi_cpufreq kernel module. By setting the /sys/devices/system/cpu/cpu%d/cpufreq/scaling_governor to userspace, you can enable turbo by setting 3101000 for a core in /sys/devices/system/cpu/cpu%d/cpufreq/scaling_setspeed (assuming your CPU speed is 3.1GHz). To disable turbo and run it at highest non-turbo frequency you would need to write 31000000 in the scaling_setspeed file.

Hope this helps.

0 Kudos
Travis_D_
New Contributor II
1,432 Views

sridutt wrote:

If your processor is Haswell, then try loading acpi_cpufreq kernel module. By setting the /sys/devices/system/cpu/cpu%d/cpufreq/scaling_governor to userspace, you can enable turbo by setting 3101000 for a core in /sys/devices/system/cpu/cpu%d/cpufreq/scaling_setspeed (assuming your CPU speed is 3.1GHz). To disable turbo and run it at highest non-turbo frequency you would need to write 31000000 in the scaling_setspeed file.

Yes, I messed around with acpi_cpufreq for a while, but I saw more or less the same behavior: even changes that were written to core-specific files always had effect across the whole package. One interesting note: when I was using acpi_cpufreq the highest turbo frequency was unavailable: I could never get a single CPU to 3.5 GHz, it was always capped at exactly 3.4 GHz. On intel_state, the CPU had no problem going to 3.5 GHz. Food for thought... Perhaps turbo can be more aggressive when HWP is enabled.

0 Kudos
Travis_D_
New Contributor II
1,432 Views

@Dr. Bandwidth - thanks for the suggestion! This works on my box as MSR_TURBO_RATIO_LIMIT is writable. Of course, it doesn't let me do what I wanted anyway (keep one core at the highest turbo), but I'm starting to thing that's going to be impossible anyway, since my assumption that I simply frequency limiting the other CPUs would prevent the "2+ cores" running multiplier taking effect was probably wrong: I assume the multiplier is based on how many cores are unhalted, rather than how many are running in turbo. So only by using an OS that was very, very quiet (i.e., all no non-essential processes running) could I achieve that.

On the other hand, this (as well as the other HWP frequency limiting MSRs I played with earlier), do give me a way to help: I can just set the max turbo ratio for 1 core (normally 35x) to 33x, which is the same as the ration for 2 cores. So I lose 200 MHz of turbo speed, but now it is *very* stable since usually only 1 or 2 cores are ever running. I would have to be very unlucky to have a third core come online (and, in fact, I can make it impossible by isolating 3 out of 4 cpus with isolcpus).

Thanks for your help!

0 Kudos
Reply