Accessing uncore_qpi in Intel E5-4610 V2

[dhumal__akshay]

Hello all,

I have a 4 socket system each with Intel(R) Xeon(R) CPU E5-4610 v2 @ 2.30GHz (Ivybridge EP).  I used this command to find all the available PMU in my system.

Command : ls -l /sys/bus/event_source/devices/

Output: 

lrwxrwxrwx 1 root root 0 Jan  3 18:14 breakpoint -> ../../../devices/breakpoint
lrwxrwxrwx 1 root root 0 Jan  3 18:14 cpu -> ../../../devices/cpu
lrwxrwxrwx 1 root root 0 Jan  3 18:14 cstate_core -> ../../../devices/cstate_core
lrwxrwxrwx 1 root root 0 Jan  3 18:14 cstate_pkg -> ../../../devices/cstate_pkg
lrwxrwxrwx 1 root root 0 Jan  3 18:14 intel_bts -> ../../../devices/intel_bts
lrwxrwxrwx 1 root root 0 Jan  3 18:14 msr -> ../../../devices/msr
lrwxrwxrwx 1 root root 0 Jan  3 18:14 power -> ../../../devices/power
lrwxrwxrwx 1 root root 0 Jan  3 18:14 software -> ../../../devices/software
lrwxrwxrwx 1 root root 0 Jan  3 18:14 tracepoint -> ../../../devices/tracepoint
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_0 -> ../../../devices/uncore_cbox_0
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_1 -> ../../../devices/uncore_cbox_1
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_2 -> ../../../devices/uncore_cbox_2
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_3 -> ../../../devices/uncore_cbox_3
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_4 -> ../../../devices/uncore_cbox_4
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_5 -> ../../../devices/uncore_cbox_5
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_6 -> ../../../devices/uncore_cbox_6
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_cbox_7 -> ../../../devices/uncore_cbox_7
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_ha_0 -> ../../../devices/uncore_ha_0
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_0 -> ../../../devices/uncore_imc_0
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_1 -> ../../../devices/uncore_imc_1
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_2 -> ../../../devices/uncore_imc_2
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_3 -> ../../../devices/uncore_imc_3
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_4 -> ../../../devices/uncore_imc_4
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_5 -> ../../../devices/uncore_imc_5
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_6 -> ../../../devices/uncore_imc_6
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_imc_7 -> ../../../devices/uncore_imc_7
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_pcu -> ../../../devices/uncore_pcu
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_r2pcie -> ../../../devices/uncore_r2pcie
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_r3qpi_0 -> ../../../devices/uncore_r3qpi_0
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_r3qpi_1 -> ../../../devices/uncore_r3qpi_1
lrwxrwxrwx 1 root root 0 Jan  3 18:14 uncore_ubox -> ../../../devices/uncore_ubox

However it does not show the uncore_qpi_0/uncore_qpi_1 folder. I am using Linux server3 4.4.0-104-generic. How can I access uncore_qpi counters?

Thanks and regards

Akshay Dhumal

[McCalpinJohn · Accepted solution]

For historical reasons, some BIOS's hide the QPI link layer counters... I think that this was due to a misunderstanding of some Intel documentation, but that is a longer story....

I have one system that has a BIOS option to enable access to the counters -- look for any BIOS option that mentions performance counters.

For another system, we had to request a new BIOS from the vendor to correct this problem.
 

[dhumal__akshay]

Thank you very much  for this information. My system is Intel server board S4600LH with BIOS SE5C600.86B.02.03.0003. It does not show anything related to performance counters. I will try new BIOS version. Does the newest BIOS version available for my system would solve the problem? 

Thanks and regards,

Akshay 

[McCalpinJohn]

Every BIOS for every board for every vendor is a little bit different, so there is no way to know what they all do.   I would certainly recommend checking for the latest update for your system and then looking for the PCI configuration space registers for the QPI Link Layer counters.

The easiest way to check is with the "lspci" command.  There will be two entries for each socket in your system if the BIOS does not hide them.  On a two-socket Xeon E5 v3 system, the lspci output for these devices looks like:

7f:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)

7f:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)

ff:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)

ff:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)

The output should be similar on a Xeon E5 4xxx v2 -- the device [8,9] (for links 0,1) and function [2] should be the same, while the bus numbers will depend on how many sockets you have populated.  A four-socket server should have its uncore devices on buses 3f, 7f, bf, ff, if I recall correctly.

[dhumal__akshay]

Thank you for the information. My system does not display Performance counter but displays system peripherals.

lspci | grep QPI:

3f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
3f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)

I have even updated by BIOS to the latest version and checked BIOS for options related to performance counters.  However I didn't  find any option for doing this. 

However when I used libpfm-4.8 tool and listed all events using showevtinfo, it showed me this:

Detected PMU models:
    [18, ix86arch, "Intel X86 architectural PMU", 7 events, 1 max encoding, 7 counters, core PMU]
    [51, perf, "perf_events generic PMU", 80 events, 1 max encoding, 0 counters, OS generic PMU]
    [71, snb_ep, "Intel Sandy Bridge EP", 83 events, 2 max encoding, 11 counters, core PMU]
    [80, snbep_unc_cbo0, "Intel Sandy Bridge-EP C-Box 0 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [81, snbep_unc_cbo1, "Intel Sandy Bridge-EP C-Box 1 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [82, snbep_unc_cbo2, "Intel Sandy Bridge-EP C-Box 2 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [83, snbep_unc_cbo3, "Intel Sandy Bridge-EP C-Box 3 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [84, snbep_unc_cbo4, "Intel Sandy Bridge-EP C-Box 4 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [85, snbep_unc_cbo5, "Intel Sandy Bridge-EP C-Box 5 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [86, snbep_unc_cbo6, "Intel Sandy Bridge-EP C-Box 6 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [87, snbep_unc_cbo7, "Intel Sandy Bridge-EP C-Box 7 uncore", 22 events, 2 max encoding, 4 counters, uncore PMU]
    [88, snbep_unc_ha, "Intel Sandy Bridge-EP HA uncore", 22 events, 3 max encoding, 4 counters, uncore PMU]
    [89, snbep_unc_imc0, "Intel Sandy Bridge-EP IMC0 uncore", 25 events, 1 max encoding, 5 counters, uncore PMU]
    [90, snbep_unc_imc1, "Intel Sandy Bridge-EP IMC1 uncore", 25 events, 1 max encoding, 5 counters, uncore PMU]
    [91, snbep_unc_imc2, "Intel Sandy Bridge-EP IMC2 uncore", 25 events, 1 max encoding, 5 counters, uncore PMU]
    [92, snbep_unc_imc3, "Intel Sandy Bridge-EP IMC3 uncore", 25 events, 1 max encoding, 5 counters, uncore PMU]
    [93, snbep_unc_pcu, "Intel Sandy Bridge-EP PCU uncore", 37 events, 2 max encoding, 4 counters, uncore PMU]
    [94, snbep_unc_qpi0, "Intel Sandy Bridge-EP QPI0 uncore", 38 events, 3 max encoding, 4 counters, uncore PMU]
    [95, snbep_unc_qpi1, "Intel Sandy Bridge-EP QPI1 uncore", 38 events, 3 max encoding, 4 counters, uncore PMU]
    [96, snbep_unc_ubo, "Intel Sandy Bridge-EP U-Box uncore", 2 events, 1 max encoding, 3 counters, uncore PMU]
    [97, snbep_unc_r2pcie, "Intel Sandy Bridge-EP R2PCIe uncore", 10 events, 1 max encoding, 4 counters, uncore PMU]
    [98, snbep_unc_r3qpi0, "Intel Sandy Bridge-EP R3QPI0 uncore", 21 events, 1 max encoding, 3 counters, uncore PMU]
    [99, snbep_unc_r3qpi1, "Intel Sandy Bridge-EP R3QPI1 uncore", 21 events, 1 max encoding, 3 counters, uncore PMU]
    [110, rapl, "Intel RAPL", 3 events, 1 max encoding, 3 counters, uncore PMU]
    [114, perf_raw, "perf_events raw PMU", 1 events, 1 max encoding, 0 counters, OS generic PMU]
Total events: 9021 available, 639 supported

It also listed all possible events to get QPI_link bandwidth and QPI_link utilization as mentioned in this Intel reference material for uncore performance monitoring for E5 processors:  https://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf. Attaching the Intel guide snapshot list.

When I collect data from these events I still get zero values. Can I trust the information which can be extracted from these counters(displayed by libpfm)? How it is possible that libpfm shows data for uncore_qpi PMU, but I am still unable to get data ?

Thanks and regards

Akshay

[McCalpinJohn]

There is certainly no guarantee that your the BIOS (even the most recent available) for your system properly exposes the QPI Link Layer counters.  I just recommended "lspci" as the easiest way to check.  (By the way, the text returned by "lspci" is not authoritative, so I recommend against using it for searches (e.g., looking for "QPI" in the text descriptions).  The easiest search on your system would be

lspci | grep "^.f:0[89]"

Then look at this more limited output to see if function 2 is available.  (I don't expect it to be there, but if you get another BIOS update, or you find other BIOS options to try changing, this is the quickest way to see if the QPI Link Layer performance counters have been made visible.)

[dhumal__akshay]

McCalpin, John wrote:

There is certainly no guarantee that your the BIOS (even the most recent available) for your system properly exposes the QPI Link Layer counters.  I just recommended "lspci" as the easiest way to check.  (By the way, the text returned by "lspci" is not authoritative, so I recommend against using it for searches (e.g., looking for "QPI" in the text descriptions).  The easiest search on your system would be

lspci | grep "^.f:0[89]"

Then look at this more limited output to see if function 2 is available.  (I don't expect it to be there, but if you get another BIOS update, or you find other BIOS options to try changing, this is the quickest way to see if the QPI Link Layer performance counters have been made visible.)

My output is still the same 

lspci | grep "^.f:0[89]"
3f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
3f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)

Can you clarify what is function 2 here ? 

[McCalpinJohn]

Function 2 is where the QPI Link Layer performance counters are located.  This is described in Section 2.8 of the Intel document titled "Intel Xeon Processor E5 v2 and E7 v2 Product Families Uncore Performance Monitoring Reference Manual", document 329468.

[dhumal__akshay]

Yes I checked it. I have a small question. For my product Intel server board S4600LH, Intel provides firmware update with following:

System BIOS  - 02.06.0005
ME Firmware  - 02.01.07.328
BMC Firmware - 01.27.9958
FRUSDR       - 0.23

My system ME firmware was same. I only updated my BIOS. Do I also need to update BMC firmware and FRUSDR?

Since my  BIOS is not showing any options to fetch performance counters related to QPI, so do I understand that, in my system Intel server board S4600LH (E5-4610 V2), with the latest BIOS installed as per the vendor, it is not possible to fetch QPI related counters.

Hope there exists some way. Any sort of help is most appreciated.

Regards

Akshay Dhumal

[McCalpinJohn]

This should be controlled by the system BIOS only, but I am not an expert on these matters.  (Actually, I have never even met an expert in these matters for recent systems, despite having spent a fair amount of time looking for such people...)