I'm testing some SIMD instructions in a Skylake Gold 6148 @ 2.40GHz (20 cores) in a node with two sockets (40 cores). I'm filling all cores with a program executing AVX-512 ADDS, MULS and FMADDS and also setting each core frequency to P_STATE 1 (base frequency, in this case 2.40GHz).
My question is, if the AVX-512 base frequency is 1.6GHz, and I'm making intense use of this AVX512 instructions, why I get a frequency measurements around 2.2GHz? I supposed that I would get a frequency measurements around 1.6GHz, because I set the P_STATE 1, which means NO TURBO, and 2.2GHz could be considered as AVX-512 turbo frequency.
(1) You need to make sure that HWP is not enabled (MSR 0x770) -- once it is enabled, the legacy interfaces to control frequency are ignored. HWP is described in Chapter 14 of Volume 3 of the Intel Architectures SW Developer's Manual.
(2) Assuming that HWP is not enabled, you need to check MSR_TURBO_ACTIVATION_RATIO (0x64c). This was introduced in the Ivy Bridge core and tells the processor to assume that a request for this frequency or higher should be interpreted as a request for maximum Turbo frequency. I have frequently seen this misconfigured by the system BIOS.
Thank you for the answer John,
I forgot some details. I'm using the 'acpi-cpufreq' driver. As far as I know, it works like 'intel_pstate' driver passive mode, where HWP is off. Maybe my best option is to know this driver, and how manages it's frequencies and scaling under the set frequency (and turbo is tuned off).
My recollection is that the "acpi-cpufreq" driver does not allow requests for frequencies between P1 and P0, but it should allow requests for P1. This is where the MSR_TURBO_ACTIVATION_RATIO (0x64c) comes into play. I have seen multiple systems that programmed this incorrectly.
Other platforms set MSR_TURBO_ACTIVATION_RATIO bits 7:0 to 0xff, so this mechanism is never triggered.
The MSR_TURBO_ACTIVATION_RATIO MSR is not included in the Xeon Scalable processor (Skylake Xeon) section of Volume 4 of the Intel SWDM, but the register is readable on my systems. I have not checked to see if it actually does anything. I have a small SKX test cluster with HWP enabled, while the larger SKX cluster has the frequencies pinned to the maximum values by the BIOS (so no frequency controls are supported).