- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
I have trouble with two problems.
First,
We have an Intel Haswell EX E7-8870 v3 machine. It has 4 sockets and specs say 3 QPI ports supported for each socket.
We want to monitor QPI links (the number of accesses to each node from a single node).
However, I found only two QPI ports detected when I run pcm-numa.x in Intel PCM V2.11. It says that the third QPI link was not found and possibly BIOS hides the device, but I cannot find any BIOS setting option to enable the third QPI link.
Here is output.
Intel(r) Performance Counter Monitor: NUMA monitoring utility Copyright (c) 2009-2016 Intel Corporation Number of physical cores: 72 Number of logical cores: 144 Number of online logical cores: 144 Threads (logical cores) per physical core: 2 Num sockets: 4 Physical cores per socket: 18 Core PMU (perfmon) version: 3 Number of core PMU generic (programmable) counters: 4 Width of generic (programmable) counters: 48 bits Number of core PMU fixed counters: 3 Width of fixed counters: 48 bits Nominal core frequency: 2100000000 Hz Package thermal spec power: 140 Watt; Package minimum power: 69 Watt; Package maximum power: 308 Watt; ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 0. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing. Socket 0: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected. ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 1. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing. Socket 1: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected. ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 2. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing. Socket 2: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected. ERROR: QPI LL performance monitoring device for the third QPI link was not found on Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz processor in socket 3. Possibly BIOS hides the device. The QPI statistics will be incomplete or missing. Socket 3: 2 memory controllers detected with total number of 8 channels. 2 QPI ports detected. Trying to use Linux perf events... Can not use Linux perf because OffcoreResponse counter usage requested. Falling-back to direct PMU programming. Socket 0 Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second) Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second) Socket 1 Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second) Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second) Socket 2 Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second) Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second) Socket 3 Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second) Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second) Detected Intel(R) Xeon(R) CPU E7-8870 v3 @ 2.10GHz "Intel(r) microarchitecture codename Haswell-EP/EN/EX" Update every 1.0 seconds Time elapsed: 1015 ms Core | IPC | Instructions | Cycles | Local DRAM accesses | Remote DRAM Accesses 0 0.33 1988 K 5973 K 348 650 1 0.23 1175 K 5135 K 227 71 2 0.27 147 K 552 K 22 9 3 0.36 399 K 1095 K 24 11 4 0.25 154 K 614 K 14 6 5 0.32 149 K 467 K 22 5 6 0.39 216 K 556 K 42 10 7 0.32 150 K 473 K 24 6 8 0.25 173 K 694 K 16 6 9 0.44 559 K 1257 K 37 66 10 0.41 151 K 364 K 18 4 11 0.44 2018 K 4564 K 109 697 12 0.43 2385 K 5528 K 111 1622 13 0.24 192 K 788 K 32 11 14 0.31 132 K 425 K 9 0 15 0.30 114 K 388 K 11 1 16 0.33 106 K 318 K 12 0 17 0.30 107 K 363 K 9 6 18 0.29 109 K 383 K 31 28 19 0.29 121 K 415 K 44 104 20 0.33 109 K 328 K 9 17 21 0.33 109 K 332 K 7 20 22 0.30 108 K 366 K 30 21 23 0.32 109 K 344 K 7 18 24 0.33 108 K 330 K 8 18 25 0.33 108 K 326 K 7 18 26 0.30 115 K 382 K 21 22 27 0.33 109 K 335 K 8 15 28 0.33 109 K 337 K 7 17 29 0.32 109 K 338 K 7 14 30 0.32 108 K 340 K 7 19 31 0.33 110 K 336 K 11 14 32 0.33 110 K 339 K 9 17 33 0.32 113 K 356 K 8 14 34 0.76 483 K 634 K 43 37 35 0.41 266 K 653 K 28 40 36 0.26 114 K 433 K 24 71 37 0.32 110 K 349 K 12 24 38 0.33 110 K 332 K 11 16 39 0.33 110 K 333 K 13 19 40 0.33 110 K 337 K 9 17 41 0.32 108 K 338 K 8 19 42 0.32 110 K 343 K 10 16 43 0.28 110 K 400 K 10 18 44 0.54 208 K 385 K 14 22 45 0.33 109 K 335 K 15 27 46 0.34 110 K 328 K 13 17 47 0.32 110 K 345 K 9 16 48 0.32 111 K 347 K 10 18 49 0.33 110 K 336 K 8 16 50 0.33 107 K 323 K 11 18 51 0.32 110 K 345 K 11 20 52 0.30 110 K 367 K 8 18 53 0.30 110 K 363 K 4 21 54 0.38 116 K 308 K 57 108 55 0.53 301 K 564 K 54 127 56 0.34 111 K 323 K 26 1 57 0.35 233 K 665 K 46 53 58 0.34 109 K 325 K 37 0 59 0.34 111 K 331 K 30 0 60 0.32 111 K 344 K 33 0 61 0.32 107 K 332 K 26 0 62 0.33 109 K 336 K 33 0 63 0.32 109 K 338 K 28 0 64 0.34 112 K 331 K 32 0 65 0.29 119 K 409 K 49 4 66 0.33 109 K 336 K 28 0 67 0.33 111 K 334 K 35 0 68 0.34 109 K 324 K 34 0 69 0.33 111 K 338 K 34 0 70 0.56 113 K 201 K 48 1 71 0.28 108 K 387 K 29 7 72 0.12 108 K 946 K 18 14 73 0.17 197 K 1166 K 75 20 74 0.38 144 K 382 K 15 10 75 0.43 375 K 881 K 26 45 76 0.38 110 K 286 K 12 0 77 0.75 399 K 529 K 30 3 78 0.39 108 K 276 K 9 0 79 0.39 163 K 423 K 19 7 80 0.39 108 K 278 K 8 0 81 0.39 693 K 1786 K 30 168 82 0.40 3040 K 7519 K 69 1549 83 0.39 2680 K 6881 K 60 1751 84 0.38 106 K 282 K 18 1 85 0.59 203 K 346 K 8 4 86 0.45 109 K 244 K 8 0 87 0.43 110 K 257 K 8 0 88 0.46 109 K 239 K 8 0 89 0.41 119 K 290 K 8 9 90 0.37 112 K 306 K 11 30 91 0.41 111 K 275 K 7 23 92 0.44 114 K 256 K 10 21 93 0.43 113 K 262 K 7 18 94 0.41 109 K 265 K 8 19 95 0.45 112 K 251 K 9 17 96 0.44 112 K 253 K 8 18 97 0.44 113 K 257 K 8 19 98 0.44 110 K 249 K 7 18 99 0.44 111 K 256 K 9 18 100 0.44 113 K 259 K 7 19 101 0.43 112 K 263 K 7 20 102 0.42 112 K 265 K 8 22 103 0.43 113 K 266 K 8 28 104 0.45 114 K 251 K 9 17 105 0.44 114 K 261 K 9 20 106 0.42 112 K 270 K 9 18 107 0.35 122 K 351 K 7 39 108 0.37 112 K 302 K 14 39 109 0.44 114 K 259 K 8 21 110 0.46 114 K 250 K 8 15 111 0.44 112 K 253 K 8 21 112 0.43 114 K 263 K 11 20 113 0.45 113 K 251 K 8 20 114 0.45 114 K 256 K 9 18 115 0.36 146 K 406 K 17 30 116 0.44 114 K 261 K 11 18 117 0.45 112 K 252 K 11 22 118 0.45 114 K 254 K 9 18 119 0.43 115 K 265 K 10 21 120 0.44 114 K 261 K 13 20 121 0.43 114 K 268 K 9 20 122 0.44 114 K 259 K 11 20 123 0.44 115 K 261 K 9 21 124 0.76 248 K 326 K 12 21 125 0.40 122 K 309 K 10 34 126 0.45 111 K 246 K 37 37 127 0.44 113 K 260 K 37 1 128 0.46 112 K 241 K 34 3 129 0.40 113 K 287 K 28 2 130 0.46 115 K 252 K 31 0 131 0.45 115 K 258 K 30 0 132 0.43 115 K 266 K 32 0 133 0.45 115 K 256 K 31 0 134 0.44 112 K 254 K 33 0 135 0.44 114 K 260 K 32 0 136 0.46 115 K 252 K 30 0 137 0.43 113 K 263 K 27 0 138 0.44 114 K 262 K 30 0 139 0.44 115 K 261 K 32 0 140 0.46 115 K 250 K 30 0 141 0.44 114 K 257 K 25 0 142 0.57 117 K 207 K 42 0 143 0.24 212 K 873 K 87 124 ------------------------------------------------------------------------------------------------------------------- * 0.37 32 M 87 M 3631 8859
OS: Ubuntu 14.04.4 LTS
Linux version: 4.4.5
Super Micro X10QBL-CT BIOS Version: 1.0b
I found BIOS updates(version 2.0), but I'm not sure that it is the BIOS problem so I need to upgrade the BIOS version.
Here is how linux finds the devices(ls result from /sys/bus/event_source/devices).
/sys/bus/event_source/devices $ ls breakpoint uncore_cbox_0 uncore_cbox_2 uncore_imc_0 uncore_qpi_1 cpu uncore_cbox_1 uncore_cbox_3 uncore_imc_1 uncore_r2pcie cstate_core uncore_cbox_10 uncore_cbox_4 uncore_imc_2 uncore_r3qpi_0 cstate_pkg uncore_cbox_11 uncore_cbox_5 uncore_imc_3 uncore_r3qpi_2 intel_bts uncore_cbox_12 uncore_cbox_6 uncore_imc_4 uncore_sbox_0 intel_cqm uncore_cbox_13 uncore_cbox_7 uncore_imc_5 uncore_sbox_1 msr uncore_cbox_14 uncore_cbox_8 uncore_imc_6 uncore_sbox_2 power uncore_cbox_15 uncore_cbox_9 uncore_imc_7 uncore_sbox_3 software uncore_cbox_16 uncore_ha_0 uncore_pcu uncore_ubox tracepoint uncore_cbox_17 uncore_ha_1 uncore_qpi_0
It shows only 'uncore_qpi_0' and 'uncore_qpi_1' detected.
Second,
We want to figure out how to monitor the performance counters which requires to set MATCH/MASK registers by using linux perf_event.
By using perf_even_open syscall, we can monitor the counters with setting perf_event_attr.
As I understand for the programming method to get the events with MATCH/MASK flags, config1 and config2 fields are required to be set.
For example, the derived event 'DRS_DataC_FROM_QPI_TO_NODEx' needs CTO_COUNT event with Q_Py_PCI_PMON_PKT_z_MATCH0 and Q_Py_PCI_PMON_PKT_z_MASK0 registers to specify the node id. (I checked Intel Xeon Processor E5 and E7 v3 Family Uncore Performance Monitoring Reference Manual)
So I checked the format for the event, mask0, and match0 as follows:
/sys/bus/event_source/devices/uncore_qpi_0/format $ cat event && cat mask0 && cat match0 config:0-7,21 config2:0-31 config1:0-31
Then I set the values to the corresponding bits of config, config1, and config2, but I got 0, that is, I cannot get the monitored results from the counter.
Does anyone know the reason why I cannot monitor only the events which requires additional registers?
My questions are:
- How can I make all the three QPI ports detected on our Intel Xeon E7-8870 v3 machine? Is it the problem of BIOS version? If it is, what is the BIOS setting option to enable the third QPI port?
- To monitor the performance counters with additional registers such as MATCH/MASK registers by using linux perf_event_open syscall, are there additional fields to be set in struct perf_event_attr? or Is there the restriction of the number of uncore performance counters which are monitored at the same time? or any other reason why I cannot get the monitored result even I set config, config2, and config1 fields to set the additional registers?
Thanks in advance.
Link Copied
0 Replies
Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page