Community
cancel
Showing results for 
Search instead for 
Did you mean: 
surim
Beginner
67 Views

BIOS detects only 2 of 3 QPI ports on Intel Xeon E7-8870 v3

Hi all,

I have trouble with two problems.

First,

We have an Intel Haswell EX E7-8870 v3 machine. It has 4 sockets and specs say 3 QPI ports supported for each socket.
We want to monitor QPI links (the number of accesses to each node from a single node).
However, I found only two QPI ports detected when I run pcm-numa.x in Intel PCM V2.11. It says that the third QPI link was not found and possibly BIOS hides the device, but I cannot find any BIOS setting option to enable the third QPI link.
Here is output.

 Intel(r) Performance Counter Monitor: NUMA monitoring utility 
 Copyright (c) 2009-2016 Intel Corporation

Number of physical cores: 72
Number of logical cores: 144
Number of online logical cores: 144
Threads (logical cores) per physical core: 2
Num sockets: 4
Physical cores per socket: 18
Core PMU (perfmon) version: 3
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2100000000 Hz
Package thermal spec power: 140 Watt; Package minimum power: 69 Watt; Package maximum power: 308 Watt; 
ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 0. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing.
Socket 0: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected.
ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 1. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing.
Socket 1: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected.
ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 2. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing.
Socket 2: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected.
ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 3. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing.
Socket 3: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected.
Trying to use Linux perf events...
Can not use Linux perf because OffcoreResponse counter usage requested. Falling-back to direct PMU programming.
Socket 0
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Socket 1
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Socket 2
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Socket 3
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)

Detected Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz "Intel(r) microarchitecture codename Haswell-EP/EN/EX"
Update every 1.0 seconds
Time elapsed: 1015 ms
Core | IPC  | Instructions | Cycles  |  Local DRAM accesses | Remote DRAM Accesses 
   0   0.33       1988 K     5973 K       348                 650                
   1   0.23       1175 K     5135 K       227                  71                
   2   0.27        147 K      552 K        22                   9                
   3   0.36        399 K     1095 K        24                  11                
   4   0.25        154 K      614 K        14                   6                
   5   0.32        149 K      467 K        22                   5                
   6   0.39        216 K      556 K        42                  10                
   7   0.32        150 K      473 K        24                   6                
   8   0.25        173 K      694 K        16                   6                
   9   0.44        559 K     1257 K        37                  66                
  10   0.41        151 K      364 K        18                   4                
  11   0.44       2018 K     4564 K       109                 697                
  12   0.43       2385 K     5528 K       111                1622                
  13   0.24        192 K      788 K        32                  11                
  14   0.31        132 K      425 K         9                   0                
  15   0.30        114 K      388 K        11                   1                
  16   0.33        106 K      318 K        12                   0                
  17   0.30        107 K      363 K         9                   6                
  18   0.29        109 K      383 K        31                  28                
  19   0.29        121 K      415 K        44                 104                
  20   0.33        109 K      328 K         9                  17                
  21   0.33        109 K      332 K         7                  20                
  22   0.30        108 K      366 K        30                  21                
  23   0.32        109 K      344 K         7                  18                
  24   0.33        108 K      330 K         8                  18                
  25   0.33        108 K      326 K         7                  18                
  26   0.30        115 K      382 K        21                  22                
  27   0.33        109 K      335 K         8                  15                
  28   0.33        109 K      337 K         7                  17                
  29   0.32        109 K      338 K         7                  14                
  30   0.32        108 K      340 K         7                  19                
  31   0.33        110 K      336 K        11                  14                
  32   0.33        110 K      339 K         9                  17                
  33   0.32        113 K      356 K         8                  14                
  34   0.76        483 K      634 K        43                  37                
  35   0.41        266 K      653 K        28                  40                
  36   0.26        114 K      433 K        24                  71                
  37   0.32        110 K      349 K        12                  24                
  38   0.33        110 K      332 K        11                  16                
  39   0.33        110 K      333 K        13                  19                
  40   0.33        110 K      337 K         9                  17                
  41   0.32        108 K      338 K         8                  19                
  42   0.32        110 K      343 K        10                  16                
  43   0.28        110 K      400 K        10                  18                
  44   0.54        208 K      385 K        14                  22                
  45   0.33        109 K      335 K        15                  27                
  46   0.34        110 K      328 K        13                  17                
  47   0.32        110 K      345 K         9                  16                
  48   0.32        111 K      347 K        10                  18                
  49   0.33        110 K      336 K         8                  16                
  50   0.33        107 K      323 K        11                  18                
  51   0.32        110 K      345 K        11                  20                
  52   0.30        110 K      367 K         8                  18                
  53   0.30        110 K      363 K         4                  21                
  54   0.38        116 K      308 K        57                 108                
  55   0.53        301 K      564 K        54                 127                
  56   0.34        111 K      323 K        26                   1                
  57   0.35        233 K      665 K        46                  53                
  58   0.34        109 K      325 K        37                   0                
  59   0.34        111 K      331 K        30                   0                
  60   0.32        111 K      344 K        33                   0                
  61   0.32        107 K      332 K        26                   0                
  62   0.33        109 K      336 K        33                   0                
  63   0.32        109 K      338 K        28                   0                
  64   0.34        112 K      331 K        32                   0                
  65   0.29        119 K      409 K        49                   4                
  66   0.33        109 K      336 K        28                   0                
  67   0.33        111 K      334 K        35                   0                
  68   0.34        109 K      324 K        34                   0                
  69   0.33        111 K      338 K        34                   0                
  70   0.56        113 K      201 K        48                   1                
  71   0.28        108 K      387 K        29                   7                
  72   0.12        108 K      946 K        18                  14                
  73   0.17        197 K     1166 K        75                  20                
  74   0.38        144 K      382 K        15                  10                
  75   0.43        375 K      881 K        26                  45                
  76   0.38        110 K      286 K        12                   0                
  77   0.75        399 K      529 K        30                   3                
  78   0.39        108 K      276 K         9                   0                
  79   0.39        163 K      423 K        19                   7                
  80   0.39        108 K      278 K         8                   0                
  81   0.39        693 K     1786 K        30                 168                
  82   0.40       3040 K     7519 K        69                1549                
  83   0.39       2680 K     6881 K        60                1751                
  84   0.38        106 K      282 K        18                   1                
  85   0.59        203 K      346 K         8                   4                
  86   0.45        109 K      244 K         8                   0                
  87   0.43        110 K      257 K         8                   0                
  88   0.46        109 K      239 K         8                   0                
  89   0.41        119 K      290 K         8                   9                
  90   0.37        112 K      306 K        11                  30                
  91   0.41        111 K      275 K         7                  23                
  92   0.44        114 K      256 K        10                  21                
  93   0.43        113 K      262 K         7                  18                
  94   0.41        109 K      265 K         8                  19                
  95   0.45        112 K      251 K         9                  17                
  96   0.44        112 K      253 K         8                  18                
  97   0.44        113 K      257 K         8                  19                
  98   0.44        110 K      249 K         7                  18                
  99   0.44        111 K      256 K         9                  18                
 100   0.44        113 K      259 K         7                  19                
 101   0.43        112 K      263 K         7                  20                
 102   0.42        112 K      265 K         8                  22                
 103   0.43        113 K      266 K         8                  28                
 104   0.45        114 K      251 K         9                  17                
 105   0.44        114 K      261 K         9                  20                
 106   0.42        112 K      270 K         9                  18                
 107   0.35        122 K      351 K         7                  39                
 108   0.37        112 K      302 K        14                  39                
 109   0.44        114 K      259 K         8                  21                
 110   0.46        114 K      250 K         8                  15                
 111   0.44        112 K      253 K         8                  21                
 112   0.43        114 K      263 K        11                  20                
 113   0.45        113 K      251 K         8                  20                
 114   0.45        114 K      256 K         9                  18                
 115   0.36        146 K      406 K        17                  30                
 116   0.44        114 K      261 K        11                  18                
 117   0.45        112 K      252 K        11                  22                
 118   0.45        114 K      254 K         9                  18                
 119   0.43        115 K      265 K        10                  21                
 120   0.44        114 K      261 K        13                  20                
 121   0.43        114 K      268 K         9                  20                
 122   0.44        114 K      259 K        11                  20                
 123   0.44        115 K      261 K         9                  21                
 124   0.76        248 K      326 K        12                  21                
 125   0.40        122 K      309 K        10                  34                
 126   0.45        111 K      246 K        37                  37                
 127   0.44        113 K      260 K        37                   1                
 128   0.46        112 K      241 K        34                   3                
 129   0.40        113 K      287 K        28                   2                
 130   0.46        115 K      252 K        31                   0                
 131   0.45        115 K      258 K        30                   0                
 132   0.43        115 K      266 K        32                   0                
 133   0.45        115 K      256 K        31                   0                
 134   0.44        112 K      254 K        33                   0                
 135   0.44        114 K      260 K        32                   0                
 136   0.46        115 K      252 K        30                   0                
 137   0.43        113 K      263 K        27                   0                
 138   0.44        114 K      262 K        30                   0                
 139   0.44        115 K      261 K        32                   0                
 140   0.46        115 K      250 K        30                   0                
 141   0.44        114 K      257 K        25                   0                
 142   0.57        117 K      207 K        42                   0                
 143   0.24        212 K      873 K        87                 124                
-------------------------------------------------------------------------------------------------------------------
   *   0.37         32 M       87 M      3631                8859                

OS: Ubuntu 14.04.4 LTS
Linux version: 4.4.5
Super Micro X10QBL-CT BIOS Version: 1.0b

I found BIOS updates(version 2.0), but I'm not sure that it is the BIOS problem so I need to upgrade the BIOS version.
Here is how linux finds the devices(ls result from /sys/bus/event_source/devices).

/sys/bus/event_source/devices $ ls
breakpoint   uncore_cbox_0   uncore_cbox_2  uncore_imc_0  uncore_qpi_1
cpu          uncore_cbox_1   uncore_cbox_3  uncore_imc_1  uncore_r2pcie
cstate_core  uncore_cbox_10  uncore_cbox_4  uncore_imc_2  uncore_r3qpi_0
cstate_pkg   uncore_cbox_11  uncore_cbox_5  uncore_imc_3  uncore_r3qpi_2
intel_bts    uncore_cbox_12  uncore_cbox_6  uncore_imc_4  uncore_sbox_0
intel_cqm    uncore_cbox_13  uncore_cbox_7  uncore_imc_5  uncore_sbox_1
msr          uncore_cbox_14  uncore_cbox_8  uncore_imc_6  uncore_sbox_2
power        uncore_cbox_15  uncore_cbox_9  uncore_imc_7  uncore_sbox_3
software     uncore_cbox_16  uncore_ha_0    uncore_pcu    uncore_ubox
tracepoint   uncore_cbox_17  uncore_ha_1    uncore_qpi_0

It shows only 'uncore_qpi_0' and 'uncore_qpi_1' detected.

 

Second,

We want to figure out how to monitor the performance counters which requires to set MATCH/MASK registers by using linux perf_event.
By using perf_even_open syscall, we can monitor the counters with setting perf_event_attr.
As I understand for the programming method to get the events with MATCH/MASK flags, config1 and config2 fields are required to be set.
For example, the derived event 'DRS_DataC_FROM_QPI_TO_NODEx' needs CTO_COUNT event with Q_Py_PCI_PMON_PKT_z_MATCH0 and Q_Py_PCI_PMON_PKT_z_MASK0 registers to specify the node id. (I checked Intel Xeon Processor E5 and E7 v3 Family Uncore Performance Monitoring Reference Manual)
So I checked the format for the event, mask0, and match0 as follows:

/sys/bus/event_source/devices/uncore_qpi_0/format $ cat event && cat mask0 && cat match0
config:0-7,21
config2:0-31
config1:0-31

Then I set the values to the corresponding bits of config, config1, and config2, but I got 0, that is, I cannot get the monitored results from the counter.
Does anyone know the reason why I cannot monitor only the events which requires additional registers?

 

My questions are:

  1. How can I make all the three QPI ports detected on our Intel Xeon E7-8870 v3 machine? Is it the problem of BIOS version? If it is, what is the BIOS setting option to enable the third QPI port?
  2. To monitor the performance counters with additional registers such as MATCH/MASK registers by using linux perf_event_open syscall, are there additional fields to be set in struct perf_event_attr? or Is there the restriction of the number of uncore performance counters which are monitored at the same time? or any other reason why I cannot get the monitored result even I set config, config2, and config1 fields to set the additional registers?

Thanks in advance.

0 Kudos
0 Replies
Reply