topic in Intel® Moderncode for Parallel Architectures

Understanding vPMC behaviour for FFCs accross multiple architecture

Sakthivel_S_ — Wed, 05 Oct 2016 06:37:23 GMT

As I said in another post, I am trying to optimize my huge application performance.. So I am starting up Intel PMU programming. Before entering into my real task, I wrote a standalone program (let say., msr_test) to make sure the PMU counters visibility of 3 FFCs & 4programmable counters. When I run msr_test on a bare metal machine I got all FFCs having some values, if I run same program on my first VM(VM1), I got FFC0 (inst. retired )=<non zero> and remaining FFC1(core counts) and FFC2(ref. core count) are always 0.Then I decided to confirm the behavior with another VM (VM2), Only FFC1 =0 remaining two counters are non 0.Which looks very strange to me, I run this multiple time and I am sure about the about results. What I absorbed is, all three machines are running on different Architecture. Does It make difference on Fixed counters(non-arch)also.

Below are my code sniff. Please check and help to understand the PMC programming.

{ MSR_WRITE, 0x38d, 0x222, 0x00 },      // ia32_perf_fixed_ctr_ctrl 0x222 is user mode for all 3 counters
{ MSR_WRITE, 0x38d, 0x333, 0x00 },     //when I say 0x333 there  I don’t get 0s any of the machines
       

MSR_WRITE is nothing but : write_msr(msrops->ecx, msrops->eax, msrops->edx);
Which inturns		:__asm__ __volatile__("wrmsr" : : "c"(ecx), "a"(eax), "d"(edx));

P.S:

Hypervisor(ESXi5.5) and vpmc.freezMode =”GUEST”

Both machines having same OS(Linux RHEL).

HW details: Bare metal code name= 06_3AH,VM1=06_2CH,VM2=06_2DH

Thanks

Sakthivel S OM

can pls some take a look on

Sakthivel_S_ — Fri, 07 Oct 2016 07:33:54 GMT

can pls some take a look on it .. and provide clarification?

Did you check to make sure

McCalpinJohn — Fri, 07 Oct 2016 12:37:48 GMT

Did you check to make sure that the OS (or hypervisor) was not changing your programming of the Fixed-Function Counter control register? (MSR 0x38d)

The NMI Watchdog often uses a fixed-function counter, but that may be disabled by default inside VMs.

The Linux "perf" utility sometimes uses the fixed-function counters for cycle counting, and has the irritating habit of disabling the counters after using them.

Sakthivel_S_ — Fri, 07 Oct 2016 13:57:40 GMT

Thanks for you valuable feedback. i just verified the points u denoted here.

first, I have printed FFC control register(0x38d) just before exiting the program, i got the proper configured value also there is No perf utility installed on my machine.

I have disabled NMI on VMs and tried the same program , but there is no change in the result(Please find below results).

FFC control: 546
FFC 0 instr retired: 6691
FFC 1 core cycles: 0 <---
FFC 2 ref cycles: 37601

-Sakthivel S OM

This is outside of my area of

McCalpinJohn — Fri, 07 Oct 2016 17:38:51 GMT

This is outside of my area of expertise, but I believe that at least some VMs run at a different ring level than you might expect. This is discussed in the "Software-Based Virtualization" section of https://en.wikipedia.org/wiki/X86_virtualization, for example.

I noticed that you showed the results of setting MSR 0x38d to 0x222 and 0x333 --- what happens with 0x111?

instr retired: 231#012

Sakthivel_S_ — Sat, 08 Oct 2016 09:07:37 GMT

instr retired: 231#012
core cyc: 0#012

ref cyc: #011 0#012

These are the values with 0x111 on FFC control register(0x38d).

As you explained earlier, if FFC is used by NMI or perf utility and reset by them, in this case we can expect all 3 FFCs values are 0. but i never seen 0 in for FFC 1.

Sakthivel_S_ — Sat, 08 Oct 2016 09:09:14 GMT

instr retired: 231
core cyc: 0

ref cyc: 0

These are the values with 0x111 on FFC control register(0x38d).

As you explained earlier, if FFC is used by NMI or perf utility and reset by them, in this case we can expect all 3 FFCs values are 0. but i never seen 0 in for FFC 1.