Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Jaeyoung__Choi
Beginner
346 Views

Understanding PCICFG space information

Jump to solution

Hi,

I have some difficulty in understanding PCICFG space information.

I was trying to read available CHA count and as I am using Xeon Gold 6132 processor, my expectation is 14.

I referred 

https://software.intel.com/en-us/download/intel-xeon-processor-scalable-memory-family-uncore-perform...

and I realize that I need to read device 30 , function 3 , offset 0x9c but bus number was not informed.

So I wrote this code,

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <asm/io.h>


const u32 PCI_ENABLE_BIT = 0x80000000;
const u32 PCI_CONFIG_ADDRESS = 0xCF8;
const u32 PCI_CONFIG_DATA = 0xCFC;


u32 r_pci_32(u8 bus, u8 device , u8 func , u8 offset){
        u32 ret;
        outl(PCI_ENABLE_BIT | (bus <<16 ) | (device <<11) | (func << 8) | (offset & 0xff) , PCI_CONFIG_ADDRESS);
        ret = inl(PCI_CONFIG_DATA);

        return ret;
}

static __init int init_pcilist(void){
        u8 bus ;
        u32 data;

        for(bus = 0 ; bus != 0xff ; bus ++){
                        data = r_pci_32(bus,30,3,0x9c);
                        printk(KERN_INFO "bus %d, device %d, func %d : value= 0x%08x\n" ,bus,30,3,data);
        }

        return 0;
}


static __exit void exit_pcilist(void){
        return;
}

module_init(init_pcilist);
module_exit(exit_pcilist);

and when I executed this code, most of the bus value was 0xffffffff

But only bus 23 and bus 133 value reported 0x03da1725 , 0x04ab64f4 respectively.

This value match what I expected but I wonder this is right approach and I am getting value properly

Finally If the my approach is right, Where can I get exact bus number??

This datasheet informs exact bus number

https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v2-datasheet-vol-2.p...

 

0 Kudos

Accepted Solutions
McCalpinJohn
Black Belt
346 Views

Intel's documentation refers to PCI device numbers in decimal, while (approximately) everyone else in the universe uses hexadecimal values....

Device 30 (decimal) is 1E hex, so I just looked for devices with this number and a defined function 3.

$ lspci | grep :1e.3
17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07)
85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07)

The CAPID6 register is then available with

$ setpci -s 17:1e.3 0x9c.l
0fffffff

$ setpci -s 85:1e.3 0x9c.l
0fffffff

This is the correct value of the bitmap for this processor (Xeon Platinum 8280), since all 28 L3/CHA slices are enabled.

The other useful approach is to look for the PCI Device ID (DID) for the unit you are looking for.  These are listed in Table 1-13 of the reference above.  For example, the UPI Link Layer devices all have DID 0x2058, so they can easily be found by

$ lspci -d :2058
5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07)
5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07)
5d:10.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:10.0 Performance counters: Intel Corporation Device 2058 (rev 07)

 

 

View solution in original post

8 Replies
McCalpinJohn
Black Belt
347 Views

Intel's documentation refers to PCI device numbers in decimal, while (approximately) everyone else in the universe uses hexadecimal values....

Device 30 (decimal) is 1E hex, so I just looked for devices with this number and a defined function 3.

$ lspci | grep :1e.3
17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07)
85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07)

The CAPID6 register is then available with

$ setpci -s 17:1e.3 0x9c.l
0fffffff

$ setpci -s 85:1e.3 0x9c.l
0fffffff

This is the correct value of the bitmap for this processor (Xeon Platinum 8280), since all 28 L3/CHA slices are enabled.

The other useful approach is to look for the PCI Device ID (DID) for the unit you are looking for.  These are listed in Table 1-13 of the reference above.  For example, the UPI Link Layer devices all have DID 0x2058, so they can easily be found by

$ lspci -d :2058
5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07)
5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07)
5d:10.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07)
d7:10.0 Performance counters: Intel Corporation Device 2058 (rev 07)

 

 

View solution in original post

Jaeyoung__Choi
Beginner
346 Views

Dear John

Thank you..!! I didn't know this simple method exist,

If I knew that, I wouldn't study what is kernel module and how it work...

I have learned a lot from you by referencing your comment for other question.

Thank you again!!

 

Newman__Chuck
Novice
174 Views

How do I find out what parts of each CHA are "alive," i.e., not fused out?  What I really want to know is what tile is a particular core in.
In my HPE ProLiant DL380 Gen10 server with (2) 33 MiB/12 core Intel Xeon Gold 6256 processors I see the following:

# lscpu | grep -E -e ' per ' -e 'Model name:'
Thread(s) per core: 1
Core(s) per socket: 12
Model name: Intel(R) Xeon(R) Gold 6256 CPU @ 3.60GHz
L3 cache: 33792K
# for Value in $(lspci | awk '/^..:1e.3/{print $1}'); do echo 2 o 16 i $(setpci -s ${Value} 0x9c.l | tr [:lower:] [:upper:]) p | dc; done
111110111111111111101110111
111111111011111111111110110

Counting the bits in those fields I get 24 CHAs for each processor.  At 1.375 MiB each, that agrees with this processor having 33 MiB of cache.

My question, then, is how can I tell which of these 24 tiles contributes a core?

For bonus points, how does one tell if it's a 28core part, an 18core part, or a 10core part?  Finding the highest set-bit position in the mask will give a lower bound on the number of physical CHAs in a part (e.g., if the mask is E73 then we know its not a 10core part, but that doesn't say if its 28 or 18).

Newman__Chuck
Novice
168 Views

I gave a bad example, as it had only 8 bits set; it should have been something like:

(e.g., if the mask is EF73 then we know its not a 10core part, but that doesn't say if its 28 or 18).

We can probably make these assumptions, though:

if ((coreCount <= 10) AND (cacheMiB <= 13.75)) then siliconSize=10Core
else If ((coreCount <= 18) AND (cacheMiB <= 24.75)) then siliconSize=18Core
else If ((coreCount <= 28) AND (cacheMiB <= 35.75)) then siliconSize=28Core
else IHaveNoIdeaWhatThisIs
McCalpinJohn
Black Belt
161 Views

The CAPID6 register only indicates the active CHA/SF/LLC slices.  In Xeon E5 v4 (Broadwell) there was a corresponding CAPID5 register that showed the active cores, but I have not seen this documented for SKX/CLX processors.

The die size can be obtained from some other PCI configuration space registers.  This is documented in the section "Component Identification via Programming Interface" section of the "Specification Update" document.  Intel document 336065 for Xeon Scalable gen 1, document 338848 for Xeon Scalable gen 2.

I have recently published two technical reports that cover very closely related material.

 

Newman__Chuck
Novice
141 Views

So no way on SkyLake/Cascade Lake to discern which CHAs present cores and which present only LLC; that's unfortunate.

I note that you gave the same URL for both of the two technical reports you cited; it's the one for "Observations on Core Numbering and Core ID's in Intel Processors."  Finding the second document is easy by navigating to the page for the first document and then searching for the title of the second document.

 

McCalpinJohn
Black Belt
105 Views

Sorry about that -- I don't see any way to edit my previous post in this new infrastructure.....

As I note in the second report, every processor that I have tested that has the same number of CHAs and Cores enabled always always has the enabled/disabled cores and CHA/SF/LLC slices co-located.  So the locations of the disabled CHAs are the same as the locations of the disabled Cores.

This can't be the case for processors with a different number of enabled cores and CHAs, but the minimal testing I have done so far on the 24-Core/26-CHA Cascade Lake processors shows that the core is also disabled at the location of each of the two disabled CHAs, and then two additional cores are disabled.  The CHAs without co-located cores are easy to identify with my measurement methodology -- they are the two CHAs for which no core activates two inbound mesh links when reading from both memory controllers.  I don't have access to very many of these nodes, so I don't want to over-generalize....  (I don't think there are any processor models with more cores than CHAs enabled.)

Corrected link for the second report....

 

McCalpinJohn
Black Belt
128 Views

Stupid website won't let me fix the typo in my earlier note.....

Considering only SKX/CLX processor models that have the same number of enabled Cores and CHA/SF/LLC slices, all of my test results show that the disabled Cores are co-located with the disabled LLCs.   So if you know the location of the disabled LLCs, you also know the locations of the disabled cores.

Some processor models have more enabled CHAs than Cores.  I have tested some 24-Core/26-CHA Cascade Lake models and found that the cores at the disabled CHA locations are disabled.  Two additional cores are disabled.  These are relatively easy to find -- they are the CHAs for which no core generates active mesh data traffic links on two sides when reading from both IMCs.   I have not gone looking through PCI configuration space to see if there is a matching (undocumented) bit mask value.....