Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

HowTo MSR for Turbo Ratios ?

CyrIng
Novice
10,422 Views

Hello,

My source code ZFreq.c displays the frequencies of the i7 cores

I'm using the MSR registers to read the core ratios multiplied by the current external clock from the SMBIOS.

However whatever the system load is, the MSR IA32_PERF_STATUS never returns the values found in the turbo zone given by MSR_TURBO_RATIO_LIMIT.

To be short IA32_PERF_STATUS never goes above MSR_PLATFORM_INFO.MaxNonTurboRatio

Please help me to program correctly those MSR

 

Thank You

CyrIng

Fr

0 Kudos
1 Solution
Patrick_F_Intel1
Employee
10,191 Views

Hello Cyring,

It depends on what you mean by 'per logical core its non halted activity'.

Usually I look at the 2 fields separately.

1) average non-halted frequency over the interval = TSC_frequency * delta(CPU_CLK_UNHALTED.THREAD) / delta(CPU_CLK_UNHALTED.REF)

2) %of time cpu is unhalted = 100 * delta(CPU_CLK_UNHALTED.REF)/delta(TSC)

Item 1) tells me "when the cpu was running (not halted), what was the average frequency". Item 2) tells me "what % of time was the cpu running".

There is an article http://software.intel.com/en-us/articles/measuring-the-average-unhalted-frequency.

Pat

View solution in original post

0 Kudos
42 Replies
Patrick_F_Intel1
Employee
2,678 Views

And don't forget that on x64 windows you need to start the program with elevated privileges (right click on the program and do 'run as administrator' (or run the program from a cmd.exe windows that you started with 'run as admin' privilege).

0 Kudos
CyrIng
Novice
2,678 Views

Hello Patrick,

Looking at the source code, pcm seems to start in winring0/OlsApiInit.h , an OpenLibSys.org driver from Hiyohiyo.

The entry point is InitOpenLibSys() followed by a serie of GetProcAddress() calls to map the selected DLL functions addresses which return zero in my case : this is the place where I'm stuck

0 Kudos
Patrick_F_Intel1
Employee
2,678 Views

I assume that WinRing0x64.dll is in the same dir as your .exe file?

You can put in debug statements and see where the load process is failing... maybe the LoadLibrary() is failing, maybe one of the GetProcAddress() calls is failing. Maybe you have another (conflicting) version of WinRing0*.h/dll/sys somewhere else? If you have _PHYSICAL_MEMORY_SUPPORT defined in OlsApiInit.h I don't think the memory routines are actually defined in driver/dll.

Pat

0 Kudos
CyrIng
Novice
2,678 Views

With debug statements, the load process fails in LoadLibrary(_T("WinRing0x64.dll"));

Fyi, below the wrapper code:

#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <tchar.h>
#include "OlsApiInit.h"

HMODULE hOpenLibSys = NULL;

BOOL initWinRing0Lib()
{
    	const BOOL result = InitOpenLibSys(&hOpenLibSys);
    	if(result == FALSE) hOpenLibSys = NULL;
    	return result==TRUE;
}

int main()
{
    printf("winring0() = %d\n", initWinRing0Lib());
    return 0;
}

 

and the Linux build command line:

i686-w64-mingw32-gcc -Wall -D_M_X64 -O2 -march=corei7  -c ~/src/Windows/winring0/main.c -o obj/Release/main.o
i686-w64-mingw32-g++  -o bin/Release/winring0.exe obj/Release/main.o  -s  -lkernel32

 

The Windows working directory :

Regards

CyrIng

0 Kudos
Patrick_F_Intel1
Employee
2,678 Views

Can you change the string _T("WinRing0x64.dll") to "WinRing0x64.dll" and see if it works?

Pat

0 Kudos
CyrIng
Novice
2,678 Views

Got it !

LoadLibrary() and GetLastError() were telling me error 193 translated by HRESULT_FROM_WIN32() into 0x800700c1 

My compiler toolchain was 32 bits and it has to be as follow to built a Windows 64 bits app.

x86_64-w64-mingw32-gcc -Wall -O2 -march=corei7  -c ~/src/Windows/winring0/main.c -o obj/Release/main.o
x86_64-w64-mingw32-g++  -o bin/Release/winring0.exe obj/Release/main.o  -s  -lkernel32

The final initialization test code works fine with Windows 7 and 2012 64 bits

#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <tchar.h>
#include "OlsApiInit.h"

HMODULE hOpenLibSys = NULL;

BOOL initWinRing0Lib()
{
	        const BOOL result = InitOpenLibSys(&hOpenLibSys);
        	if(result == FALSE) hOpenLibSys = NULL;
        	return result==TRUE;
}

int main()
{
    if(initWinRing0Lib())
        	DeinitOpenLibSys(&hOpenLibSys);
    return 0;
}

Thanks Patrick for your help.

CyrIng

0 Kudos
McCalpinJohn
Honored Contributor III
2,678 Views

One problem that I have encountered in using MSR IA32_PERF_STATUS to look at the frequency is that you cannot read this counter while in the user-space context.  MSRs can only be read by the kernel, so you have to call the MSR device driver, which may have to set up an inter-processor interrupt to the target processor, and finally the target processor reads its IA32_PERF_STATUS MSR.   In the time between the normal execution of the user code and the execution of the RDMSR command the frequency can change, and I have observed that on our Haswell (Xeon E5 v3) systems it often does change (depending on BIOS options & power-limiting).

This is probably one of the reasons why Intel recommends that you obtain the average frequency over intervals from cycle counters, rather than reading the instantaneous multiplier. If CR4.PCE is set, then you can execute the RDPMC instruction in user space and avoid any perturbations to the system (like a transition into the kernel) that might increase the probability of a frequency change.   Either the fixed-function counters or the programmable counters are useful for this purpose.  I prefer the fixed-function counters since these can be enabled once and require no further kernel driver calls to program.

An irritation that I ran across this week is that (at least on my RHEL 6.5 & 6.6 systems), the "perf stat" command disables the fixed-function performance counters after it uses them, rather than checking their previous configuration and returning them to that state after use.   Stupid *&^!@#%^ software.

0 Kudos
CyrIng
Novice
2,678 Views

Hello John,

True, I also crossed the same conclusion that IA32_PERF_STATUS is not the recommended MSR to measure the frequency

As you can read in my Algorithm , I'm also reading the PMU fixed counters, on a sample period, which are multiplied by the relative frequency ratio read from MSR_PLATFORM_INFO and MSR_TURBO_RATIO_LIMIT

You may get more detail in the source code of XFreq server and the uCycle() function where the frequency computation happens.

After more than a year programming those counters, I find them pretty precised. Programmable counters are also on my next schedule, especially to measure the QPI bandwidth, DRAM, RAPL ...

perf stat like others Linux kernel drivers (such as Watchdog) don't bother with the state of counters when exiting or sharing. That's why I reserved a save area for registers. See function Init_MSR_Nehalem()

I wish you could tell me how my program is running with your Xeon E5 v3 ?

Best regards

CyrIng

0 Kudos
CyrIng
Novice
2,678 Views
Windows 7 64bits is so frustrating. There is nothing difficult to program from scratch a ring0 driver, but running it without the certification stuff is a no go winring0x64.sys is also a painful solution (and a source of virus issues) I won't ask my users to deactivate signature check ... So far I give up to port XFreq to Windows
0 Kudos
CyrIng
Novice
2,678 Views
C7 issue with a 5-3450S IvyBridge: Reading msr MSR_CORE_C7_RESIDENCY (0x3fe) all the time returns a zero value, same with. MSR_PKG_C7_RESIDENCY (0x3fa), while C3, C6 are OK ? Should I drive C7 differently ? Some kind of additional initialization of this counter ? Thanks for any help CyrIng
0 Kudos
Patrick_F_Intel1
Employee
2,678 Views

Hello cyring,

I don't know why C7 is showing up as zero but it shows as zero on my ivybridge laptop as well. Perhaps there isn't enough difference between C6 and C7 on ivybridge so it isn't used. I do see that 'MSR_PKG_CST_CONFIG_CONTROL: Package C-State Limit' reports C7.

On a more personal note, I will soon be leaving Intel and this is probably my last posting. It has been a pleasure sharing with and learning from the users of this forum. Good luck in the future folks,

Pat

0 Kudos
CyrIng
Novice
2,678 Views
OK, Thank you for the tips. Wish you good luck and if you can transfer me your black belt -;) CyrIng
0 Kudos
CyrIng
Novice
2,678 Views

FYI, CoreFreq, a light version of XFreq is available in the GitHub

CoreFreq is based on a Linux Kernel driver which spawns one thread per Core.

Each threads loop reads the msr in a more precised way than in user land.

Regards

CyrIng

0 Kudos
CyrIng
Novice
2,678 Views

Hello,

This is CoreFreq , a Linux Kernel driver which handles the performance monitoring counters, and displays the Core frequencies, c-states, temps & Instructions per second or cycle.

CoreFreq.png

I have a question :

The Base Clock is estimated from an invariant TSC.

How to compute this frequency with a variant TSC, such as Core 2 processor ?

Regards

CyrIng

0 Kudos
Roman_D_Intel
Employee
2,678 Views

CyrIng wrote:

I have a question :

The Base Clock is estimated from an invariant TSC.

How to compute this frequency with a variant TSC, such as Core 2 processor ?

If other methods are not available you can read the nominal frequency from the CPU brand string output by cpuid instruction. Please take a look at the get_frequency_from_cpuid function in Intel PCM.

Thanks,

Roman

0 Kudos
CyrIng
Novice
2,678 Views

Thanks

I have copy/past Intel's code and a few adaptation :

unsigned int Proc_Brand(char *pBrand)
{
	char idString[64]={0x20};
	unsigned int ix=0, jx=0, px=0;
	unsigned int frequency=0, multiplier=0;
	BRAND Brand;

	for(ix=0; ix < 3; ix++)
	{
		asm volatile
		(
			"cpuid"
			: "=a"  (Brand.AX),
			  "=b"  (Brand.BX),
			  "=c"  (Brand.CX),
			  "=d"  (Brand.DX)
			: "a"   (0x80000002 + ix)
		);
		for(jx=0; jx < 4; jx++, px++)
			idString[px]=Brand.AX.Chr[jx];
		for(jx=0; jx < 4; jx++, px++)
			idString[px]=Brand.BX.Chr[jx];
		for(jx=0; jx < 4; jx++, px++)
			idString[px]=Brand.CX.Chr[jx];
		for(jx=0; jx < 4; jx++, px++)
			idString[px]=Brand.DX.Chr[jx];
	}
	for(ix=0; ix < 46; ix++)
		if((idString[ix+1] == 'H') && (idString[ix+2] == 'z'))
		{
			switch(idString[ix])
			{
				case 'M':
					multiplier=1;
				break;
				case 'G':
					multiplier=1000;
				break;
				case 'T':
					multiplier=1000000;
				break;
			}
			break;
		}
	if(multiplier > 0)
	{
	    if(idString[ix-3] == '.')
	    {
		frequency  = (int) (idString[ix-4] - '0') * multiplier;
		frequency += (int) (idString[ix-2] - '0') * (multiplier / 10);
		frequency += (int) (idString[ix-1] - '0') * (multiplier / 100);
	    }
	    else
	    {
		frequency  = (int) (idString[ix-4] - '0') * 1000;
		frequency += (int) (idString[ix-3] - '0') * 100;
		frequency += (int) (idString[ix-2] - '0') * 10;
		frequency += (int) (idString[ix-1] - '0');
		frequency *= frequency;
	    }
	}
	for(ix=jx=0; jx < 48; jx++)
		if(!(idString[jx] == 0x20 && idString[jx+1] == 0x20))
			pBrand[ix++]=idString[jx];

	return(frequency);
}

Roman Dementiev (Intel) wrote:

Quote:

CyrIng wrote:

 

I have a question :

The Base Clock is estimated from an invariant TSC.

How to compute this frequency with a variant TSC, such as Core 2 processor ?

 

 

If other methods are not available you can read the nominal frequency from the CPU brand string output by cpuid instruction. Please take a look at the get_frequency_from_cpuid function in Intel PCM.

Thanks,

Roman

0 Kudos
CyrIng
Novice
2,678 Views

This is the CoreFreq algorithm:

CoreFreq-algorithm.png

0 Kudos
CyrIng
Novice
2,678 Views

CoreFreq v1.2

All-per-CPU

0 Kudos
CyrIng
Novice
2,678 Views
0 Kudos
CyrIng
Novice
2,573 Views

Dual Haswell Processors

CoreFreq_Dual_Haswell.png

0 Kudos
CyrIng
Novice
2,573 Views

Turbo ratio limits

0 Kudos
Reply