Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
63 Views

What is PCICFG address in uncore of intel XEON?

Jump to solution

Hi community

I am reading XEON-E5-2600-UNCORE-GUIDE. and I got confused about the address column in the tables marked as PCICFG address.

What is PCICFG address and how can I transfer the address provided in the UNCORE-GUIDE to an address that I can use with by msr-tools?

 

0 Kudos

Accepted Solutions
Highlighted
Black Belt
63 Views

The addressing for the uncore performance counters in the Xeon E5-2600 processors can certainly be confusing!

1. The easiest approach is to use the "lspci" command to look for the PCI addresses of the units with performance counters in PCI configuration space.   According to the documentation in the "Intel Xeon Processor E5-1600/2400/2600 (E5-Product Family) Product Families: Datasheet - Volume 2"  (Intel document 326509), the PCI "bus number" of the uncore devices in the Xeon E5 processor in socket zero of the system is reported in the PCI configuration space of bus 0, device 5, function 0, offset 0x109.   You can read this with the following command:

setpci -s 00:5.0 109.b

The output will be one of the following: 1f, 3f, 7f, ff

Although the documentation does not make this clear, the PCI "bus number" of the uncore devices in the Xeon E5 processor in socket 1 of the system is the next number in the sequence above.   Most of my systems use 7f and ff, but some use 3f and 7f.

2. The next piece of confusion is that the PCI Device numbers used in the Xeon E5-2600 uncore configuration guide are *decimal* values, while the "lspci" and "setpci" commands use hexadecimal numbering.  

The iMC Performance Monitors provide a good example.  Table 2-59 of the Xeon E5-2600 Uncore Performance Monitoring Guide lists the "MC Channel 0 PMON Registers" as using D16:F0.     This needs to be translated to "device" 10 (hex), function 0 (hex).  

3.  Combining 1 and 2 is best shown by example.   First I look up the "bus" used by socket 0:

# setpci -s 00:5.0 109.b
7f

Next print out the label for Device 10 (hex), Function 0:

# lspci -s 7f:10.0
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)

Note that the text here might be completely different or even completely wrong on your system -- the important thing is that the device exists.

Next print out the contents of the first 256 Bytes --- this will include all of the "offset" addresses used by the performance monitor control and counting "registers"

$ lspci -xxx -s 7f:10.0
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
00: 86 80 b0 3c 00 00 10 00 07 00 80 08 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 18 05
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00
40: 10 00 91 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 11 23 10 00 00 00 00 00 02 7c 05 00 00 00 00 00
b0: d9 e5 0b 00 00 00 00 00 fb 81 02 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 92 fa 84 c7 36 2d 00 00 04 03 40 01 04 0c 40 01
e0: 01 00 40 01 02 01 40 01 00 00 00 00 00 00 00 00
f0: 00 00 40 00 00 00 01 00 00 00 00 00 00 00 00 00


The numbering of this "lspci" output is from left (offset 0) to right (offset 15), but remember that all multi-byte accesses are little-endian.   An easy example to see what this means is to look at the first two bytes:

 $ setpci -s 7f:10.0 0.w
8086


The 16-bit value "8086" (reversed from the first two values in the first line above) is a code indicating that this PCI device is an Intel product.

4. Now we can look at some specific memory controller performance counter control registers.  Table 2-59 says that MC_CH0_PCI_PMON_CTL0 (channel 0 performance counter control) is a 32-bit value at offset 0xD8.    These can be read from the output above ("04 03 40 01") or by

$ setpci -s 7f:10.0 d8.l
01400304

This clearly gives the same bits in the expected order for a little-endian 32-bit word.

Going to Table 2-61, this 32-bit field can be expanded to see the current programming of the counter:

thresh     31:24       0x01    <-- threshold used in counter comparison

invert         23           0x0     <-- invert flag is not set

en              22           0x1    <-- enable bit is set

(rsv)         21:19        0x0    <-- reserved bits are correctly set to zero

edge_det   18           0x0    <-- edge detect bit is not enabled

(rsv)         17:16        0x0   <-- reserved bits are correctly set to zero

umask      15:8        0x03   <-- umask = 0x03 --> RD sub-event of CAS_COUNT

ev_sel        7:0        0x04   <-- event = 0x04 --> CAS_COUNT

All of the other counters based in PCI configuration space are analogous.

 

To program the counters, you can use "setpci" (as root) or you can open the PCI device driver files.  Using the same examples as above, the device driver is at "/proc/bus/pci/7f/10.0".    This file can be opened (with root permissions) and the "pread()" and "pwrite" commands can be used to read and write binary data into the device driver with the correct offsets.

 

View solution in original post

0 Kudos
3 Replies
Highlighted
Black Belt
64 Views

The addressing for the uncore performance counters in the Xeon E5-2600 processors can certainly be confusing!

1. The easiest approach is to use the "lspci" command to look for the PCI addresses of the units with performance counters in PCI configuration space.   According to the documentation in the "Intel Xeon Processor E5-1600/2400/2600 (E5-Product Family) Product Families: Datasheet - Volume 2"  (Intel document 326509), the PCI "bus number" of the uncore devices in the Xeon E5 processor in socket zero of the system is reported in the PCI configuration space of bus 0, device 5, function 0, offset 0x109.   You can read this with the following command:

setpci -s 00:5.0 109.b

The output will be one of the following: 1f, 3f, 7f, ff

Although the documentation does not make this clear, the PCI "bus number" of the uncore devices in the Xeon E5 processor in socket 1 of the system is the next number in the sequence above.   Most of my systems use 7f and ff, but some use 3f and 7f.

2. The next piece of confusion is that the PCI Device numbers used in the Xeon E5-2600 uncore configuration guide are *decimal* values, while the "lspci" and "setpci" commands use hexadecimal numbering.  

The iMC Performance Monitors provide a good example.  Table 2-59 of the Xeon E5-2600 Uncore Performance Monitoring Guide lists the "MC Channel 0 PMON Registers" as using D16:F0.     This needs to be translated to "device" 10 (hex), function 0 (hex).  

3.  Combining 1 and 2 is best shown by example.   First I look up the "bus" used by socket 0:

# setpci -s 00:5.0 109.b
7f

Next print out the label for Device 10 (hex), Function 0:

# lspci -s 7f:10.0
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)

Note that the text here might be completely different or even completely wrong on your system -- the important thing is that the device exists.

Next print out the contents of the first 256 Bytes --- this will include all of the "offset" addresses used by the performance monitor control and counting "registers"

$ lspci -xxx -s 7f:10.0
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
00: 86 80 b0 3c 00 00 10 00 07 00 80 08 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 18 05
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00
40: 10 00 91 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 11 23 10 00 00 00 00 00 02 7c 05 00 00 00 00 00
b0: d9 e5 0b 00 00 00 00 00 fb 81 02 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 92 fa 84 c7 36 2d 00 00 04 03 40 01 04 0c 40 01
e0: 01 00 40 01 02 01 40 01 00 00 00 00 00 00 00 00
f0: 00 00 40 00 00 00 01 00 00 00 00 00 00 00 00 00


The numbering of this "lspci" output is from left (offset 0) to right (offset 15), but remember that all multi-byte accesses are little-endian.   An easy example to see what this means is to look at the first two bytes:

 $ setpci -s 7f:10.0 0.w
8086


The 16-bit value "8086" (reversed from the first two values in the first line above) is a code indicating that this PCI device is an Intel product.

4. Now we can look at some specific memory controller performance counter control registers.  Table 2-59 says that MC_CH0_PCI_PMON_CTL0 (channel 0 performance counter control) is a 32-bit value at offset 0xD8.    These can be read from the output above ("04 03 40 01") or by

$ setpci -s 7f:10.0 d8.l
01400304

This clearly gives the same bits in the expected order for a little-endian 32-bit word.

Going to Table 2-61, this 32-bit field can be expanded to see the current programming of the counter:

thresh     31:24       0x01    <-- threshold used in counter comparison

invert         23           0x0     <-- invert flag is not set

en              22           0x1    <-- enable bit is set

(rsv)         21:19        0x0    <-- reserved bits are correctly set to zero

edge_det   18           0x0    <-- edge detect bit is not enabled

(rsv)         17:16        0x0   <-- reserved bits are correctly set to zero

umask      15:8        0x03   <-- umask = 0x03 --> RD sub-event of CAS_COUNT

ev_sel        7:0        0x04   <-- event = 0x04 --> CAS_COUNT

All of the other counters based in PCI configuration space are analogous.

 

To program the counters, you can use "setpci" (as root) or you can open the PCI device driver files.  Using the same examples as above, the device driver is at "/proc/bus/pci/7f/10.0".    This file can be opened (with root permissions) and the "pread()" and "pwrite" commands can be used to read and write binary data into the device driver with the correct offsets.

 

View solution in original post

0 Kudos
Highlighted
Black Belt
63 Views

@Zhu

Please take a look at this document about the accessing of PCIE config registers on Intel chipsets.

https://www.element14.com/community/servlet/JiveServlet/previewBody/20589-102-1-62045/White%20Paper%...

0 Kudos
Highlighted
Beginner
63 Views

Thanks Dr. B

I stuck at step 3, when I typed the cmd lspci -xxx -s 7f:10.0, here is what I got from my machine:

7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
00: 86 80 b0 3c 00 00 10 00 07 00 80 08 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

Obviously I have something missing on my machine. I think it may be my motherboard.

I am considering buying new server board such as s2600CWT, or use whole system product such as DELL PowerEdge R730.

But I don't know if my job is available on these machine, I contacted the sales but they don't know about this. what should I do?

0 Kudos