topic Prabhu, in Software Tuning, Performance Optimization & Platform Monitoring

Intel Performance Counter Monitor - Can't access PCI configuration space

Prabhu_T_ — Thu, 13 Jun 2013 13:15:32 GMT

Hi,

I'm trying to run the latest PCM 2.5 on a machine with Intel(R) Xeon(R) CPU E5-2670 (Sandy Bridge), but PCM gives the below message and do not show any memory read/write access in the report.

Can not access SNB-EP (Jaketown) PCI configuration space. Access to uncore counters (memory and QPI bandwidth) is disabled.
You must be root to access these SNB-EP counters in PCM

Can you please guide me what am I missing here.

1. I'm running PCM as root and also looked at the DELL BIOS options for performance counters related to PCI but could not find any to enable or disable

Thanks, Prabhu

It is related to Bios support

Bernard — Thu, 13 Jun 2013 15:38:19 GMT

It is related to Bios support.I do not think if in your case such a functionality is exposed to end user.Try to contact either DELL or Bios vendor.

The "lspci" command under

McCalpinJohn — Thu, 13 Jun 2013 15:57:19 GMT

The "lspci" command under Linux will list all of the PCI configuration space ranges that the BIOS has made accessible.
These can be cross-referenced with the list in the "Intel® Xeon® Processor E5-2600 Product Family Uncore Performance Monitoring Guide"(Reference Number: 327043-001), to see exactly which devices and functions are missing. This information will be helpful if you contact the system vendor about BIOS updates to enable these functions.

And if you are on Windows you

Bernard — Thu, 13 Jun 2013 17:11:03 GMT

And if you are on Windows you can use local kernel debugger to inspect pci configuration space with !pci command(tested on Win XP).

Thank you for your

Prabhu_T_ — Fri, 14 Jun 2013 04:51:25 GMT

Thank you for your suggestions Mr.John D. McCalpin and Mr. iliyapolak. I will try to contact the vendor for a BIOS update if available. But I have one more question, how is the Intel vtune amplifier 2013 is able to collect data for read/write of memory(Memory Access or Bandwidth analysis) and PCM cannot collect due to the above issue. If bios do not allow then VTune also shouldn't be collecting, Am I missing something ?. Please see below, the read and write is 0 for PCM.

-------------------------------------------------------------------------------------------------------------------

TOTAL * 0.15 1.05 0.15 1.15 511 K 1973 K 0.74 0.96 0.02 0.01 0.00 0.00 N/A

- Prabhu

Hi Prabhu,

Roman_D_Intel — Fri, 14 Jun 2013 08:57:50 GMT

Hi Prabhu,

could you please post the output of lspci command here to let me understand the issue better?

Thanks,

Roman

Hi Roman,

Prabhu_T_ — Fri, 14 Jun 2013 13:08:47 GMT

Hi Roman,

Please see below for the output of lscpi command:

~/PCM2.5 # lspci
00:00.0 Host bridge: Intel Corporation Sandy Bridge DMI2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Sandy Bridge IIO PCI Express Root Port 1a (rev 07)
00:02.0 PCI bridge: Intel Corporation Sandy Bridge IIO PCI Express Root Port 2a (rev 07)
00:02.2 PCI bridge: Intel Corporation Sandy Bridge IIO PCI Express Root Port 2c (rev 07)
00:03.0 PCI bridge: Intel Corporation Sandy Bridge IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
00:03.2 PCI bridge: Intel Corporation Sandy Bridge IIO PCI Express Root Port 3c (rev 07)
00:05.0 System peripheral: Intel Corporation Sandy Bridge Address Map, VTd_Misc, System Management (rev 07)
00:05.2 System peripheral: Intel Corporation Sandy Bridge Control Status and Global Errors (rev 07)
00:11.0 PCI bridge: Intel Corporation Patsburg PCI Express Virtual Root Port (rev 05)
00:16.0 Communication controller: Intel Corporation Patsburg HECI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation Patsburg HECI Controller #2 (rev 05)
00:1a.0 USB Controller: Intel Corporation Patsburg USB2 Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation Patsburg PCI Express Root Port 1 (rev b5)
00:1c.7 PCI bridge: Intel Corporation Patsburg PCI Express Root Port 8 (rev b5)
00:1d.0 USB Controller: Intel Corporation Patsburg USB2 Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation Device 1d41 (rev 05)
00:1f.2 SATA controller: Intel Corporation Patsburg 6 Port SATA AHCI Controller (rev 05)
01:00.0 Ethernet controller: Intel Corporation Device 1521 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Device 1521 (rev 01)
01:00.2 Ethernet controller: Intel Corporation Device 1521 (rev 01)
01:00.3 Ethernet controller: Intel Corporation Device 1521 (rev 01)
02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS TB (rev 01)
08:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
09:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
09:01.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
0a:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB]
0b:00.0 VGA compatible controller: Matrox Graphics, Inc. G200eR2
3f:08.0 System peripheral: Intel Corporation Sandy Bridge QPI Link 0 (rev 07)
3f:09.0 System peripheral: Intel Corporation Sandy Bridge QPI Link 1 (rev 07)
3f:0a.0 System peripheral: Intel Corporation Sandy Bridge Power Control Unit 0 (rev 07)
3f:0a.1 System peripheral: Intel Corporation Sandy Bridge Power Control Unit 1 (rev 07)
3f:0a.2 System peripheral: Intel Corporation Sandy Bridge Power Control Unit 2 (rev 07)
3f:0a.3 System peripheral: Intel Corporation Device 3cd0 (rev 07)
3f:0b.0 System peripheral: Intel Corporation Sandy Bridge Interrupt Control Registers (rev 07)
3f:0b.3 System peripheral: Intel Corporation Sandy Bridge Semaphore and Scratchpad Configuration Registers (rev 07)
3f:0c.0 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0c.1 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0c.2 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0c.3 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0c.6 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 0 (rev 07)
3f:0c.7 System peripheral: Intel Corporation Sandy Bridge System Address Decoder (rev 07)
3f:0d.0 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0d.1 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0d.2 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0d.3 System peripheral: Intel Corporation Device 3ce8 (rev 07)
3f:0d.6 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 1 (rev 07)
3f:0e.0 System peripheral: Intel Corporation Sandy Bridge Processor Home Agent (rev 07)
3f:0e.1 Performance counters: Intel Corporation Sandy Bridge Processor Home Agent Performance Monitoring (rev 07)
3f:0f.0 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Registers (rev 07)
3f:0f.1 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller RAS Registers (rev 07)
3f:0f.2 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 0 (rev 07)
3f:0f.3 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 1 (rev 07)
3f:0f.4 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 2 (rev 07)
3f:0f.5 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 3 (rev 07)
3f:0f.6 System peripheral: Intel Corporation Device 3cae (rev 07)
3f:10.0 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
3f:10.1 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
3f:10.2 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 0 (rev 07)
3f:10.3 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 1 (rev 07)
3f:10.4 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
3f:10.5 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
3f:10.6 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 2 (rev 07)
3f:10.7 System peripheral: Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 3 (rev 07)
3f:11.0 System peripheral: Intel Corporation Device 3cb8 (rev 07)
3f:13.0 System peripheral: Intel Corporation Sandy Bridge R2PCIe (rev 07)
3f:13.1 Performance counters: Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor (rev 07)
3f:13.4 Performance counters: Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers (rev 07)
3f:13.5 Performance counters: Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
3f:13.6 System peripheral: Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)

------

Thanks,Prabhu

Prabhu,

Roman_D_Intel — Fri, 14 Jun 2013 14:08:28 GMT

Prabhu,

does your system have single processor (socket) ?

Thanks,

Roman

>?>>Can not access SNB-EP

Bernard — Fri, 14 Jun 2013 15:25:07 GMT

>?>>Can not access SNB-EP (Jaketown) PCI configuration space. Access to uncore counters (memory and QPI bandwidth)>>>

Does it mean device 8 and 9 cannot be read?

Roman, The system has single

Prabhu_T_ — Fri, 14 Jun 2013 16:27:52 GMT

Roman, The system has single processor only.

Thanks,

Prabhu

>>> Intel vtune amplifier

Bernard — Fri, 14 Jun 2013 16:48:16 GMT

>>> Intel vtune amplifier 2013 is able to collect data for read/write of memory(Memory Access or Bandwidth analysis) and PCM cannot collect due to the above issue>>>

Do you mean to access pci address space?

Prabhu,

Roman_D_Intel — Mon, 17 Jun 2013 07:08:26 GMT

Prabhu,

it seems that some low-level topology registers on your system have not been setup correctly by the BIOS. I will provide you a software workaround for PCM shortly.

Thanks,

Roman

Prabhu,

Roman_D_Intel — Mon, 17 Jun 2013 12:39:00 GMT

Prabhu,

could you please apply the attached patch.txt ?

[bash]

cd IntelPerformanceCounterMonitorV2.5

patch < patch.txt

[/bash]

It should enable the integrated memory controller statistics. The QPI statistics are still hidden by your BIOS. But probably you don't need QPI stats anyway because you have a single socket system (QPI links usually connect processors/sockets).

Please try it and let me know if memory bandwidth counters work or not. You can generate memory traffic with the memoptest utility in PCM:

[bash]

cd IntelPerformanceCounterMonitorV2.5

make memoptest

./memoptest 0

[/bash]

Thanks,

Roman

Hi Roman,

Prabhu_T_ — Tue, 18 Jun 2013 06:54:00 GMT

Hi Roman,

Thanks, I have installed your patch. Below is the output of ./memoptest 0. But read and write from memory controller in PCM output did not change.

Elements data size: 203125 KB
Reading memory
Bandwidth: 2446.06 MByte/sec
Reading memory
Reading memory
Bandwidth: 1854.13 MByte/sec

I have build the entire PCM again and I have expected READ/WRITE in the output of the pcm.x would show the read and write from memory controller(still shows zero). But I think this patch has fixed the "Instructions per nominal CPU cycle" marked in the below output of pcm and don't know if the patch is relavent to this.

-------------------------------------------------------------------------------------------------------------------
TOTAL * 1.10 0.95 1.15 1.15 4489 K 11 M 0.60 0.95 0.02 0.01 0.00 0.00 N/A

Instructions retired: 45 G ; Active cycles: 48 G ; Time (TSC): 2610 Mticks ; C0 (active,non-halted) core residency: 100.00 %

C1 core residency: 0.00 %; C3 core residency: 0.00 %; C6 core residency: 0.00 %; C7 core residency: 0.00 %
C2 package residency: 0.00 %; C3 package residency: 0.00 %; C6 package residency: 0.00 %; C7 package residency: 0.00 %

PHYSICAL CORE IPC : 1.91 => corresponds to 47.70 % utilization for cores in active state
Instructions per nominal CPU cycle: 2.20 => corresponds to 55.04 % core utilization over time interval

Thanks, Prabhu

@Roman

Bernard — Tue, 18 Jun 2013 07:12:59 GMT

@Roman

what x86 instruction(s) are used to read memory? Is this rep movsb?

Prabhu,

Roman_D_Intel — Tue, 18 Jun 2013 08:33:43 GMT

Prabhu,

Could you please post the full output for

[plain]

./pcm.x "sleep 1"

lspci -vvv

[/plain]

Thanks,

Roman

Since it is a lot of data you

Roman_D_Intel — Tue, 18 Jun 2013 08:38:35 GMT

Since it is a lot of data you can better attach it to your post. Thanks, Roman

>>>Instructions per nominal

Bernard — Tue, 18 Jun 2013 10:45:00 GMT

>>>Instructions per nominal CPU cycle: 2.20 => corresponds to 55.04 % core utilization over time interval>>>

It could be interested to see the exact breakdown of executed instructions.It can be provided by VTune.

@iliyapolak

Roman_D_Intel — Tue, 18 Jun 2013 14:27:00 GMT

@iliyapolak

I use C++ std::find scan operation on array of structures (integer key + data). It does a few accesses per cache line on the keys (each access for a key is 4-bytes). I believe the compiler should generate normal integer mov instructions without the rep prefix in this case. I am not saying it is the most efficient way to generate read traffic (but a simple one using higher-level C++).

Roman

Thanks for answer.I thought

Bernard — Tue, 18 Jun 2013 16:28:26 GMT

Thanks for answer.I thought that you were using memcpy() instruction to move data from/to large int arrays. Afaik memcpy() is translated to rep movsd when the type is int.