topic <quote> in Software Tuning, Performance Optimization & Platform Monitoring

isCoreOnline will always return false if /proc/cpuinfo doesn't print "physical id" and / or "core id"

Valentin_Bondzio — Mon, 22 Jun 2015 08:41:18 GMT

Hi Guys,

just FYI:

# ./pcm.x 1 -i=1

Intel(r) Performance Counter Monitor V2.8 (2014-12-18 12:52:39 +0100 ID=ba39a89)

Copyright (c) 2009-2014 Intel Corporation

Number of physical cores: 1
Number of logical cores: 2
Number of online logical cores: 2
Threads (logical cores) per physical core: 2
Num sockets: 1
Physical cores per socket: 1
Core PMU (perfmon) version: 1
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Nominal core frequency: 2800000000 Hz
Delay: 1

Detected Intel(R) Xeon(R) CPU X5660 @ 2.80GHz "Intel(r) microarchitecture codename Westmere/Clarkdale"
terminate called after throwing an instance of 'std::exception'
what(): std::exception
DEBUG: caught signal to interrupt (Aborted).
Cleaning up
Zeroed PMU registers
Freeing up all RMIDs

It turns out that isCoreOnline returned false for every processor because /proc/cpuinfo on this machine looked like:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 37
model name : Intel(R) Xeon(R) CPU X5660 @ 2.80GHz
stepping : 1
cpu MHz : 2799.204
cache size : 12288 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat
bogomips : 5598.40
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

A simple workaround for this case was to replace:

return (topology[os_core_id].os_id != -1) && (topology[os_core_id].core_id != -1) && (topology[os_core_id].socket != -1);

with:

return 1;

But you might want to test for that case (if it is a supported configuration) and print a better error.

Cheers,

Valentin

thanks for the report. What

Roman_D_Intel — Tue, 23 Jun 2015 09:13:28 GMT

thanks for the report. What is the Linux kernel version you are running? I would like to learn why the physical ids / core ids are missing.

Best regards,

Roman

Uhps, thought I posted that:

Valentin_Bondzio — Tue, 23 Jun 2015 11:00:59 GMT

Uhps, thought I posted that:

# uname -a
Linux foo 2.6.32.59-0.7-default #1 SMP 2012-07-13 15:50:56 +0200 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1

Two other machines with the same version of SLES / kernel did not exhibit the same behavior (cpuinfo did contain phys and core id). I don't have direct access to the systems, I just know that all 3 systems are on different HW,

edit while writing: I was just told that this was a VM (on ESXi), even though I had asked to compile / test on physical HW only. This wouldn't have worked in any case (vPMC needs to be explicitly enabled for one) but now that it seems to be down to the presentation of the CPU by the virtual hardware, I can check internally why this is. I'll follow up.

Cheers,

Valentin

Valentin,

Roman_D_Intel — Tue, 23 Jun 2015 11:11:53 GMT

Valentin,

pcm needs access to performance monitoring units through MSR driver or through Linux perf (enable in the PCM Makefile). Providing this access is a challenge in a virtualized environment.

Thanks,

Roman

Actually, according to the

Valentin_Bondzio — Tue, 23 Jun 2015 11:13:00 GMT

Actually, according to the Linux source, the behavior is correct for one core per socket configurations. VMware virtual machines are per default created with coresPerSocket =1 and each additional vCPU increases the vSocket count, hence what we see here.

Cheers,

Valentin

edit: P.S.
This fails before MSRs are even read though, if vPMC is enabled for a VM with just one vCPU (on core / socket) it will still run into this exception.

Roman_D_Intel — Tue, 23 Jun 2015 11:21:00 GMT

<quote>

the behavior is correct for one core per socket configurations

</quote>

on a system with many sockets how to find processor (socket) id of a core if the "physical id" is missing?

Thanks,

Roman

Generally PCM has some amount

Roman_D_Intel — Tue, 23 Jun 2015 11:29:00 GMT

Generally PCM has some assumptions with regards to uncore and socket topology that might be broken under virtualization (for example that each socket owns an own memory controller).

I wouldn't be able to answer

Valentin_Bondzio — Tue, 23 Jun 2015 11:39:29 GMT

I wouldn't be able to answer that ... I'm just referring to:

/*
 *	Get CPU information for use by the procfs.
 */
static void show_cpuinfo_core(struct seq_file *m, struct cpuinfo_x86 *c,
			      unsigned int cpu)
{
#ifdef CONFIG_SMP
	if (c->x86_max_cores * smp_num_siblings > 1) {
		seq_printf(m, "physical id\t: %d\n", c->phys_proc_id);
		seq_printf(m, "siblings\t: %d\n",
			   cpumask_weight(cpu_core_mask(cpu)));
		seq_printf(m, "core id\t\t: %d\n", c->cpu_core_id);
		seq_printf(m, "cpu cores\t: %d\n", c->booted_cores);
		seq_printf(m, "apicid\t\t: %d\n", c->apicid);
		seq_printf(m, "initial apicid\t: %d\n", c->initial_apicid);
	}
#endif
}

I'd guess directly from the APIC ID? But I really wouldn't know, not procfs as far as I can tell.

Cheers,

Valentin

Quote:Roman Dementiev (Intel)

Valentin_Bondzio — Tue, 23 Jun 2015 11:45:52 GMT

Roman Dementiev (Intel) wrote:

Generally PCM has some assumptions with regards to uncore and socket topology that might be broken under virtualization (for example that each socket owns an own memory controller).

I agree, VM sizing / vNUMA configuration matter a lot and should be made as equal to the underlying HW as possible + the vPMC modes and (e.g.) ESXi scheduling need to be understood. https://labs.vmware.com/vmtj/methodology-for-performance-analysis-of-vmware-vsphere-under-tier-1-applications is a very good discussion regarding analysis on VMware VMs.

All I'm saying is that it would still fail on a physical single core per socket system, if those still exist :-)

However, it would be nice if PMC could detect that it is running on a VM an print a warning.

Cheers,

Valentin

you are right. We will

Roman_D_Intel — Tue, 23 Jun 2015 11:57:00 GMT

you are right. We will provide a special handling for such machines (probably someone still have them :-).

Do you know how to reliably detect that the program is running on a VM?

Thanks,

Roman

We actually have it

Valentin_Bondzio — Tue, 23 Jun 2015 12:19:00 GMT

We actually have it documented for VMware VMs :-) For a more general approach I'd probably look at SMBIOS/DMI information, I found a good overview here.

There is of course the possibility that e.g. the native SMBIOS configuration is (partially) passed through from the host (or faked) but that is a very very rare case and only required for ancient software or other legacy reasons (or someone wants to hide the fact that something is running in a VM, who knows).

Note that thanks to technology like vPMC (if enabled), many events can still be very beneficial. So please only warn that some counters might not be available due to configuration or availability of virtual hardware features / different from running on bare metal. (as opposed to to just exiting for example)

Thanks for being so responsive to feedback!

Valentin