Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

[PCM] QPI traffic reported all zeros

Thomas_W_Intel
Employee
3,728 Views

Zheng L. posted:

Hello everyone, I try to get some data from the Intel Xeon E5-2687W by using PCU. Beause of the project reason, we are mainly interested finding in how the multi-thread using QPI to get the reading from the PCI card may effect the system. However, The incoming data traffic of QPI are always 0 and the outgoing data traffic are always 0 too. And I get some weird reading to. Is that possible that the reading is wrong?

Also I have two screen shots of that, but I find that the forum can not upload the picture. Is there any way that I can upload the picture so someone can help me analyze that?

Thank you very much.

Might it be that you have a second instance of PCM running? It might also be one that was not cleanly shut down?

 

0 Kudos
49 Replies
Zheng_Luo
Beginner
989 Views

John D. McCalpin wrote:

We are running Dell DCS8000 systems, so the BIOS is likely different than yours.   I don't know what command was used to enable the PCI configuration space areas for the QPI counters and was not able to find it in the list of options that I checked.   It is probably best to follow up with Dell.

Thank you very much.

0 Kudos
Zheng_Luo
Beginner
989 Views

Zheng Luo wrote:

Quote:

Roman Dementiev (Intel)wrote:

Zheng Luo,

This version of BIOS seems to be much better. Your BIOS settings changes should not impact QPI perfmon device visibility. Could you please send me lspci output?

I noticed in your PCM output the line "Number of PCM instances: 2". It may happen that you run two instances of PCM or sometime ago a PCM instance has been killed unexpectedly such that PCM instance counting becomes broken (this may also break QPI statistics display in this version of PCM: to be fixed in the next release). To clean the instance counting please do the following:

1. stop all pcm instances

2. as root: rm -rf /dev/shm/sem.*Intel*

3. start pcm again

Please share the output of pcm started after you did these operation.

Thanks for your cooperation,

Roman

Thank you very much Roman,

Now the PCM seems working now, at least there is no arbitrary value in the regarding to QPI reading. I will do more test with that. By the way, how do you close the PCM instance? I just use Ctrl + C to close the program...

Also I use the command 

numactl --cpunodebind=0 --membind=0 ./lat_mem_rd -t 1024

numactl --cpunodebind=0 --membind=1 ./lat_mem_rd -t 1024

mentioned in http://software.intel.com/en-us/forums/topic/280235#comment-1755207 to generated traffic in the QPI, but all the data that I read are zeros, this time there is no arbitrary values, but all the QPI data is zero. Is there any other way to generate traffic in the QPI so that I may see the non-zero data? 

Zheng Luo

0 Kudos
Roman_D_Intel
Employee
989 Views

Zheng Luo,

yes, using Ctrl-C to stop PCM is fine.

Could you share your current output from pcm.x?

You can also try single threaded memory bandwidth test included in PCM:

make memoptest

# reading

numactl --cpunodebind=0 --membind=1 ./memoptest 0

# writing

numactl --cpunodebind=0 --membind=1 ./memoptest 1

# writing using non-temporal streaming stores

numactl --cpunodebind=0 --membind=1 ./memoptest 2

Thanks,

Roman

0 Kudos
Roman_D_Intel
Employee
989 Views

Just had seen the copy of your post above with the output:

ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P1_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P1_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2).

0 Kudos
Zheng_Luo
Beginner
989 Views

Roman Dementiev (Intel) wrote:

Just had seen the copy of your post above with the output:

ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P1_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2). ERROR: QPI LL counter programming seems not to work. Q_P1_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2).

Hi Roman,

If I get these error message, does that mean the QPI counter still not working? Therefore, I got all the zero result. And those probems are still caused by the BIOS?

Zheng Luo 

0 Kudos
Roman_D_Intel
Employee
989 Views

correct

0 Kudos
McCalpinJohn
Honored Contributor III
989 Views

You can tell whether the QPI link-layer performance counters are accessible from the lspci output.   The text description may vary, but for each socket (in your case these will start with "1f:" and "3f:") the output of lspci should include lines like:

1f:08.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 0 Performance Monitor (rev 07)
1f:09.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 1 Performance Monitor (rev 07)
3f:08.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 0 Performance Monitor (rev 07)
3f:09.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 1 Performance Monitor (rev 07)

The first 7 characters are all that matters -- the first field defines the "bus" (in this case the bus number corresponds to a particular socket), the second field defines the "slot" (sometimes referred to as the "device"), and the third field defines the "function".    For each socket, slot 8, function 2 corresponds to the QPI interface 0 link layer the hardware performance counters, while slot 8, function 2 corresponds to the QPI interface 1 link layer hardware performance counters.

The QPI link layer counters have additional functionality (for "mask" and "match" functions) under devices 8/9, function 6.   I don't know if the Intel PCM or VTune tools use these, but from reading the documentation (Xeon E5-2600 uncore performance monitoring guide) they like they could be useful for some detailed analyses.

With the latest BIOS updates on our systems, we were able to use the "setupbios" utility from Dell to enable access to these PCI configuration space areas.  The text descriptions from lspci don't make any sense -- they don't include the word "performance" or the acronym "QPI" -- but the functions work fine. 

Attempts to read devices/functions in PCI configuration space that are unsupported or non-existent will result in a "master abort" response (all bits set -- i.e., 0xFFFFFFFF).   Most software recognizes this case and returns zero values for the counters rather than aborting execution of the performance monitoring program.

0 Kudos
Zheng_Luo
Beginner
989 Views

Roman Dementiev (Intel) wrote:

correct

OK, I see, I will contact Dell to see if they can give any help. Thank you very much.

Zheng Luo

0 Kudos
Zheng_Luo
Beginner
989 Views

Roman Dementiev (Intel) wrote:

correct

Thank you for the help Roman. I will try to contact Dell to see if I can get any help from there.

Zheng Luo

0 Kudos
Zheng_Luo
Beginner
989 Views

John D. McCalpin wrote:

You can tell whether the QPI link-layer performance counters are accessible from the lspci output.   The text description may vary, but for each socket (in your case these will start with "1f:" and "3f:") the output of lspci should include lines like:

1f:08.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 0 Performance Monitor (rev 07)
1f:09.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 1 Performance Monitor (rev 07)
3f:08.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 0 Performance Monitor (rev 07)
3f:09.2 Performance counters: Intel Corporation Sandy Bridge QPI Port 1 Performance Monitor (rev 07)

The first 7 characters are all that matters -- the first field defines the "bus" (in this case the bus number corresponds to a particular socket), the second field defines the "slot" (sometimes referred to as the "device"), and the third field defines the "function".    For each socket, slot 8, function 2 corresponds to the QPI interface 0 link layer the hardware performance counters, while slot 8, function 2 corresponds to the QPI interface 1 link layer hardware performance counters.

The QPI link layer counters have additional functionality (for "mask" and "match" functions) under devices 8/9, function 6.   I don't know if the Intel PCM or VTune tools use these, but from reading the documentation (Xeon E5-2600 uncore performance monitoring guide) they like they could be useful for some detailed analyses.

With the latest BIOS updates on our systems, we were able to use the "setupbios" utility from Dell to enable access to these PCI configuration space areas.  The text descriptions from lspci don't make any sense -- they don't include the word "performance" or the acronym "QPI" -- but the functions work fine. 

Attempts to read devices/functions in PCI configuration space that are unsupported or non-existent will result in a "master abort" response (all bits set -- i.e., 0xFFFFFFFF).   Most software recognizes this case and returns zero values for the counters rather than aborting execution of the performance monitoring program.

Thanks for you reply John. In your reply you mentioned that "Attempts to read devices/functions in PCI configuration space that are unsupported or non-existent will result in a "master abort" response (all bits set -- i.e., 0xFFFFFFFF).   Most software recognizes this case and returns zero values for the counters rather than aborting execution of the performance monitoring program". Therefore, it is the reason that I get all the zero values? 

0 Kudos
McCalpinJohn
Honored Contributor III
989 Views

Yes -- your "lspci" output did not list the device 8/9, function 2 entries, and your previous posting showed that you were getting the master abort response:

         "ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2).

The Intel PCM software recognizes this case and simply fills the count fields with zeros.  Perhaps an "N/A" field would be more appropriate, but that would still leave you wondering if the field was "Not Available" (as in your case) or "Not Applicable" (for example on a single-socket chip that has no QPI interfaces).

0 Kudos
Zheng_Luo
Beginner
989 Views

John D. McCalpin wrote:

Yes -- your "lspci" output did not list the device 8/9, function 2 entries, and your previous posting showed that you were getting the master abort response:

         "ERROR: QPI LL counter programming seems not to work. Q_P0_PCI_PMON_BOX_CTL=0xffffffff Please see BIOS options to enable the export of performance monitoring devices (devices 8 and 9: function 2).

The Intel PCM software recognizes this case and simply fills the count fields with zeros.  Perhaps an "N/A" field would be more appropriate, but that would still leave you wondering if the field was "Not Available" (as in your case) or "Not Applicable" (for example on a single-socket chip that has no QPI interfaces).

I see it. Thank you very much for your explanation. I found something in the forum may be helpful, and I will try that. http://software.intel.com/en-us/forums/topic/385194  He asked Dell to provide an experimental BIOS, and that actually works.

Zheng Luo

0 Kudos
Sonal_A_
Beginner
989 Views

Hello!

I am trying to use Intel PCM to measure the L3 cache misses for a program. I create the PCM object in my program as instructed at https://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization.

I compile my file using :
icc -O3 -o myfile myfile.cpp ../PCM/cpucounters.cpp ../PCM/msr.cpp ../PCM/pci.cpp ../PCM/client_bw.cpp -AVX -xhost

When I execute, I get the output as shown in 436792

As you can see the QPI link layer performance counters are possible not accessible.

When I do a > lspci | grep QPI, I get the following output:
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)

The model of my machine is: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz

Please suggest as to how to get around this problem.

0 Kudos
McCalpinJohn
Honored Contributor III
989 Views

As you can see from the prior comments in this forum thread, your BIOS is not exposing the QPI link-layer counters.  On your system these would be devices 7f:08.2, 7f:09.2, ff:08.2, ff:09.2, with associated "mask/match" controls at 7f:08.6, 7f:09.6, ff08.6, ff09.6.

Your BIOS may or may not actually support these devices.  As I noted above, once I received the BIOS upgrade that supported these devices I still could not see them.  I had to change a BIOS setting and reboot to actually make them visible.

So you need to work with your system vendor to find out if your BIOS supports these devices, and if so, what BIOS option is required to actually enable them.

0 Kudos
Daniel_S_3
Beginner
989 Views

Hey!

I've got a related issue with a SM quad socket system (X9QRi-F) and 4x Xeon E5-4620 v2.
I checked already the BIOS settings for related options to enable/disable the QPI LL counters, but without success.
I'm affraid that the system does not support the QPI link-layer counters, as mentioned in previous posts, but maybe you can give me hint whether there is an issue with the pcm tool or not ... I attached the output of pcm and lspci.

The lspci output contains a line like

Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)

for each socket. Can I assume, that the QPI LL counters are available or which kind of monitoring device is listed here?

Thank you for your help!

Kind regards,

Daniel S.

0 Kudos
Roman_D_Intel
Employee
989 Views

Daniel,

Most probably you have this issue: https://software.intel.com/en-us/articles/bios-preventing-access-to-qpi-performance-counters

It hides these registers (which are also missing on your system according to the lspci output you have provided):

QPI0 Port 0 PMON Registers D8:F2

QPI0 Port 1 PMON Registers D9:F2

QPI1 Port 2 PMON Registers D24:F2

QPI0 Mask/Match Port 0 PMON Registers D8:F6

QPI0 Mask/Match Port 1 PMON Registers D9:F6

QPI1 Mask/Match Port 2 PMON Registers D24:F6

Roman

0 Kudos
Mrunal_G_
Beginner
989 Views

Hi,

I was looking at the pcm-numa.x output from 2.8 release. I  want to understand what the column names Local and Remote DRAM access means. For e.g. I am running a database workload and trying to understand how much traffic crosses across sockets in my 4 socket machine. 

 Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz "Intel(r) microarchitecture codename Ivy Bridge-EP/EN/EX/Ivytown"

My question is how do I interpret these numbers for local and remote dram access. What does 268K and 187K means exactly in the first line below. It would have been better to quantify the QPI traffic, but as this post states, I am also getting 0, as the qpi traffic always, so I plan to look at the BIOS related settings today with my sys admin. 

If I can not get QPI numbers directly, Is there any way I can find out what is the cumulative traffic across sockets, using the cumulative Remote DRAM access? What is considered a DRAM access and what size granularity it is, because there could be also prefetching in sequential access case, but  in a random access case the access could be different such that only a single page is touched.  

Please help me understand these numbers to make some better interpretation of my database workload. I have two cases here, one is a scan operator, which I expect to have a sequential access if it comes from remote dram, and a hash table lookup kind of access, which could be a random remote dram kind of access during probe phase in a remote dram containing hash table. 

I tried looking at the code below, but could not understand it. 

https://github.com/erikarn/intel-pcm/blob/master/src/pcm-numa.cpp

 

Core | IPC  | Instructions | Cycles  |  Local DRAM accesses | Remote DRAM Accesses 
   0   0.16         79 M      485 M       268 K               187 K              
   1   0.13         59 M      455 M       262 K               139 K              
   2   0.13         60 M      452 M       263 K               137 K              
   3   0.13         58 M      447 M       257 K               135 K              
   4   0.13         59 M      448 M       255 K               138 K              

0 Kudos
Mrunal_G_
Beginner
989 Views

Ok. So I tried checking if BIOS has some parameters that I can enable for QPI. I saw in QPI configuration QPI Link0s and QPI Link0p, and on enabling them the server refused to even show bios menu. I checked Internet for these parameters and they seem to be harmless from readings on forums. 

Apart from this I could not understand which parameters to play with in BIOS for enabling the QPI monitoring support. The motherboard is following.

    product: X9QR7-TF+/X9QRi-F+ (070715D9)
    vendor: Supermicro

If you have any insight to enable monitoring this, that would be great. I am pasting output of lscpi below.  and ls -al /dev/mem

 ls -al /dev/mem  --->
crw-r----- 1 root kmem 1, 1 Feb 12 10:02 /dev/mem

lspci --->

 

00:00.0 Host bridge: Intel Corporation Xeon E5 v2/Core i7 DMI2 (rev 04)

00:02.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
04:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
3f:08.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
3f:09.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
3f:0a.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
3f:0a.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
3f:0a.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
3f:0a.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
3f:0b.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0b.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0e.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0f.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
3f:0f.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
3f:0f.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:10.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
3f:10.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
3f:10.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
3f:10.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
3f:10.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
3f:10.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
3f:10.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
3f:10.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
3f:13.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
3f:13.5 Performance counters: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
3f:16.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 System Address Decoder (rev 04)
3f:16.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
3f:16.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
3f:1c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
3f:1c.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
3f:1d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
3f:1d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
3f:1d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
3f:1e.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
3f:1e.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
3f:1e.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
3f:1e.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
3f:1e.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
3f:1e.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
3f:1e.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)
40:01.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
40:04.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
40:04.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
40:04.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
40:04.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
40:04.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
40:04.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
40:04.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
40:04.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
40:05.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
40:05.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 IIO RAS (rev 04)
40:05.4 PIC: Intel Corporation Xeon E5 v2/Core i7 IOAPIC (rev 04)
41:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 05)
7f:08.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
7f:09.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
7f:0a.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
7f:0a.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
7f:0a.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
7f:0a.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
7f:0b.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
7f:0b.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
7f:0c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
7f:0e.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
7f:0f.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
7f:0f.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
7f:0f.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:10.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
7f:10.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
7f:10.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
7f:10.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
7f:10.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
7f:10.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
7f:10.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
7f:10.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
7f:13.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
7f:13.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
7f:13.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
7f:13.5 Performance counters: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
7f:16.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 System Address Decoder (rev 04)
7f:16.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
7f:16.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
7f:1c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
7f:1c.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
7f:1d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
7f:1d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
7f:1d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
7f:1e.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
7f:1e.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
7f:1e.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
7f:1e.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
7f:1e.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
7f:1e.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
7f:1e.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)
80:04.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
80:04.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
80:04.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
80:04.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
80:04.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
80:04.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
80:04.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
80:04.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
80:05.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
80:05.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 IIO RAS (rev 04)
80:05.4 PIC: Intel Corporation Xeon E5 v2/Core i7 IOAPIC (rev 04)
bf:08.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
bf:09.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
bf:0a.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
bf:0a.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
bf:0a.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
bf:0a.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
bf:0b.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
bf:0b.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
bf:0c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0c.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0c.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0c.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0c.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0c.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
bf:0e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
bf:0e.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
bf:0f.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
bf:0f.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
bf:0f.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
bf:0f.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
bf:0f.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
bf:0f.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
bf:10.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
bf:10.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
bf:10.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
bf:10.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
bf:10.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
bf:10.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
bf:10.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
bf:10.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
bf:13.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
bf:13.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
bf:13.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
bf:13.5 Performance counters: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
bf:16.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 System Address Decoder (rev 04)
bf:16.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
bf:16.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
bf:1c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
bf:1c.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
bf:1d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
bf:1d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
bf:1d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
bf:1d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
bf:1d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
bf:1d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
bf:1e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
bf:1e.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
bf:1e.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
bf:1e.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
bf:1e.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
bf:1e.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
bf:1e.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
bf:1e.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)
c0:02.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
c0:02.2 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
c0:03.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
c0:04.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
c0:04.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
c0:04.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
c0:04.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
c0:04.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
c0:04.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
c0:04.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
c0:04.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
c0:05.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
c0:05.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 IIO RAS (rev 04)
c0:05.4 PIC: Intel Corporation Xeon E5 v2/Core i7 IOAPIC (rev 04)
c1:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
c1:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
c3:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
c4:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
ff:08.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
ff:09.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
ff:0a.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
ff:0a.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
ff:0a.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
ff:0a.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
ff:0b.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
ff:0b.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 UBOX Registers (rev 04)
ff:0c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0c.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0c.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0c.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0c.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0c.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Unicast Registers (rev 04)
ff:0e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
ff:0e.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
ff:0f.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
ff:0f.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
ff:0f.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
ff:0f.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
ff:0f.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
ff:0f.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
ff:10.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
ff:10.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
ff:10.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
ff:10.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
ff:10.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
ff:10.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
ff:10.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
ff:10.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
ff:13.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
ff:13.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 R2PCIe (rev 04)
ff:13.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
ff:13.5 Performance counters: Intel Corporation Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
ff:16.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 System Address Decoder (rev 04)
ff:16.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
ff:16.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
ff:1c.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
ff:1c.1 Performance counters: Intel Corporation Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
ff:1d.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
ff:1d.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
ff:1d.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
ff:1d.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
ff:1d.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
ff:1d.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
ff:1e.0 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
ff:1e.1 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
ff:1e.2 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
ff:1e.3 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
ff:1e.4 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
ff:1e.5 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
ff:1e.6 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
ff:1e.7 System peripheral: Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)

 

 

 

 

 

 

0 Kudos
Otto_B_Intel
Employee
989 Views

Are you sure you have the latest BIOS installed? Sometimes the vendor adds features in newer BIOS releases but this is no guarantee. When flashing your BIOS please make sure you follow proper procedure, things can go very wrong if you do not do so.

0 Kudos
Mrunal_G_
Beginner
942 Views

I do not have a control over the bios upgrade as the machine I am dealing is monitored by a sys admin and he does the maintenance work. We reverted back to the default factory setting and now the machine seems to boot and I can run the pcm-numa.x command now, though earlier I would use ctrl+c to kill it to get a log, without sudo kill, but now since the reboot It needs me to do a sudo kill else it ignores the ctrl+c totally. 

so I think I will not be able to get anything out of bios since I am working towards a deadline and the sys admin schedule might not allow upgrade before that, but that is also not the main problem, even if we do an upgrade we are not sure we will resolve the qpi showing correct values in pcm tool output. So for time being, I want to see, if I can make the numbers in remote and local memory access make some sense. 

I am running some experiments using memory mapped io and trying to see the affinity of data access with respect to memory, but the numbers on local and remote are not matching my expectations. For e.g. when I expect minimal remote memory access when my task is parallelizable and I expect memory mapped io to map data affinity to sockets to get minimal remote memory access, but I do not see that. 

That is the reason, I would highly appreciate if some body could tell me the meaning of these numbers and how to interpret them, the way I asked in my first post here. Please let me know if any insights into interpreting them. 

Thank you very much. 

0 Kudos
McCalpinJohn
Honored Contributor III
942 Views

As I noted above in https://software.intel.com/en-us/forums/topic/473955#comment-1756915, you need devices 8.2 and 9.2 on each socket (buses 1f, 7f, bf, ff) to get the QPI link-layer counters, and those are clearly not present in the lspci output.

There are other counters that can identify local and remote DRAM accesses.  Several of these have bugs on Sandy Bridge -- I don't know if they have been fixed in Ivy Bridge.   Intel's VTune includes (partial) workarounds for the bugs on Sandy Bridge platforms -- I don't know if PCM does this (or if it is necessary).

The "local DRAM accesses" and "remote DRAM accesses" are almost certainly counting cache line transfers.

Core | IPC  | Instructions | Cycles  |  Local DRAM accesses | Remote DRAM Accesses 
   0   0.16         79 M      485 M       268 K               187 K              
   1   0.13         59 M      455 M       262 K               139 K              
   2   0.13         60 M      452 M       263 K               137 K              
   3   0.13         58 M      447 M       257 K               135 K              
   4   0.13         59 M      448 M       255 K               138 K              

The IPC is very low here -- indicating a lot of stalls.  The local access rate for these 5 cores is about 1.305 Million cache lines 83.52 million bytes.  Assuming 2.4 GHz operation, the 485 million cycles is 0.2 seconds, so the local bandwidth is 83.5 MB/0.2 seconds = 413 MB/s, which is very low.  Remote bandwidth is lower -- about 235 MB/s.

Another way to look at this is that these 5 threads have 1.3 M local DRAM accesses in 485 M cycles, or one access per 372 cycles.    At 2.4 GHz, this is one access every 155 ns.   I am guessing that there are other threads running at the same time, and their references need to be added into the formula.   The remote access rate of 0.74 M accesses in 485 M cycles is one access every 653 cycles, or one per 272 ns.  

All of these access rates seem slow, but the very low IPC suggests lots of stalls, so memory accesses seem like the right place to be looking.  You will want to look for remote vs local DRAM access rates across all the cores in the system to see if there is an imbalance that is slowing things down.  You might also want to look at the DRAM traffic counters on each chip to see if the accesses are close to uniformly distributed.

On the Xeon E5-46xx v1 (Sandy Bridge) systems I saw poor sustained bandwidth between sockets, but never came up with a good explanation.   These values look more latency-limited than bandwidth-limited, but more data is needed.

0 Kudos
Reply