Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

Xeon E5 PCI Extended address space base address

shiv_a_
Beginner
2,152 Views

I am trying to read Bus 0 Device 0 Function 0 and do not get a DID of "8086" as I would have expected on my Xeon E5 system. Can you help me with the PCI extended address space base address (pciexbar) to read the correct value?
 

0 Kudos
19 Replies
Patrick_F_Intel1
Employee
2,152 Views

Is this windows or linux?

On linux you can use 'lspci -x' to list the devices.

There is also a windows version at http://mj.ucw.cz/sw/pciutils/ but I don't know anything about it.

You have to 'run as admin' on windows to access the PCI registers. On windows you need to be root.

Pat

0 Kudos
Bernard
Valued Contributor I
2,152 Views

shiv a. wrote:

I am trying to read Bus 0 Device 0 Function 0 and do not get a DID of "8086" as I would have expected on my Xeon E5 system. Can you help me with the PCI extended address space base address (pciexbar) to read the correct value?

 

Actually DID 8086 was assigned by PCIE group to Intel.

I assume that you are on Linux. IIRC there is an Intel document which explains how to access PCI extended register space.

That doc is named "Accessing PCI Express* Configuration Registers Using Intel® Chipsets "

http://www.intel.la/content/www/xl/es/intelligent-systems/chipsets-pcie-config-reg-paper.html

0 Kudos
Bernard
Valued Contributor I
2,152 Views

>>>On windows you need to be root.>>>

If PCIe registers are memory-mapped then only kernel mode driver can access them.

0 Kudos
McCalpinJohn
Honored Contributor III
2,152 Views

In default Linux configurations, accessing the extended PCI configuration space requires root access, but this can be reduced to a one-time permissions setup in many cases.

On a standard RHEL 6.4 system running as a normal user, the beginning 64 Bytes of PCI configuration space is easily accessible:

    $ /sbin/lspci -xx -s 0:0.0
    00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
    00: 86 80 00 3c 00 00 10 00 07 00 00 06 10 00 00 00
    10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 8c 04
    30: 00 00 00 00 90 00 00 00 00 00 00 00 00 01 00 00

Running as the root user allows the lspci "-xxx" option to read the first 256 Bytes and the lspci "-xxxx" option to read the full 4 KiB.

You can write a program to open /proc/bus/pci/00/00.0 and read or write whatever you want -- you just need to chown the program to root and then mark it as setuid.   This is especially useful for codes that only need to read from PCI space, since you can open the device read-only and the setuid root binary won't have the ability to kill your system by writing inappropriate bits into the configuration space of various devices.

If you need the lowest possible overhead, you can open /dev/mem and mmap() the relevant PCI configuration space.  This allows direct user-space reads of the PCI configuration space data -- no device driver or kernel crossing required.  There are three ways to make this work -- run the program as root, chown the program to root and mark as setuid, or modify the permissions on the /dev/mem pseudo-file so that a group that your account belongs to has read permission.   Of course you can do the same thing with write permissions, but the probability of killing your system increases significantly.

Perhaps the best compromise for read-only data would be to create a specialized loadable kernel module and associated device driver that provides an mmap() interface for the limited area or areas of PCI configuration space that you need to read.  This provides both protection and the ability to read the data directly from user-space using an ordinary pointer.  

I use these approaches all the time on Xeon E5 (to read the uncore counters that are in PCI configuration space) and on Xeon Phi (to read the memory controller counters that are in MMIO space).   To reduce the risk of killing my systems, I have one very short code that has read/write access (to program counters) that was very carefully reviewed and is seldom modified, and a separate code that has read-only access that I can fiddle with more freely.

0 Kudos
shiv_a_
Beginner
2,152 Views

Thank You for the responses. I am using a Xeon E5 server with a Windows 7 (64 bit) OS.

I read through the article that iliyapolak has pointed to. From the article, it seems that the PCI Express Configuration Base Address Register varies processor to processor, and also it is called "HECBASE register" on some systems. How do I find the address for the xeon E5 processor? Also, my system has 2 processors. Does that change formula?

0 Kudos
shiv_a_
Beginner
2,152 Views

Thank You John! I will try the experiment you have mentioned.

0 Kudos
Bernard
Valued Contributor I
2,152 Views

@shiva

Try to read corresponding chipset datasheet.

0 Kudos
shiv_a_
Beginner
2,152 Views

John, Do you know what the base address is that gives the location of the enhanced address space? I was reading through the document that iliyapolak pointed to, it says

"The memory address range containing the PCIe* configuration registers is
implemented differently between different Intel chipsets. In Intel’s older
chipsets, the PCI Express* Configuration Base Address Register is contained
inside the memory controller portion of the chipset (MCH and GMCH). The
location of this register varies between chipsets. For the Intel® Q45 and
PM965 Express chipsets, for example, this register is located in PCI space at
B0:D0:F0-60h. In the Intel® EP80579 Integrated Processor (“Silicon on
Chip”) however, this register is called the HECBASE register and is located in
at B0:D0:F0-CEh. Programmers must be aware that the memory range and
its control registers change between chipsets."

I also tried to read the intel chipset documentation, but could not locate this information. I guess, I dont understand what section of the documentation talks about this.

0 Kudos
Patrick_F_Intel1
Employee
2,152 Views

So...are you on linux or not?

If you are on linux, you really should be reading the pci registers through the /proc/bus interface...

Like /proc/bus/pci/00/00.0 for BDF=/0/0/0. I think that this will give access to the extended space.

If you are on linux... then you can get the base addr of the memory pci space through the acpi MCFG table. you can access this table through /sys/firmware/acpi/tables/MCFG. On windows you can use the EnumSystemFirmwareTables('ACPI', NULL, BufferSize) API to find the MCFG table. The pci mem addr is at offset 0x44 for 8 bytes in the mcfg table. This works for wndows or linux.

Pat

0 Kudos
Bernard
Valued Contributor I
2,152 Views

 

If you are on Windows than windbg command !pci or !pcitree will probably display the needed information(I did not check it).

Some partially related information can be found in this link.

http://www.osronline.com/showthread.cfm?link=249094

0 Kudos
Bernard
Valued Contributor I
2,152 Views

 

@shiva 

As I told you earlier you must to read your chipset datasheet.

 

0 Kudos
McCalpinJohn
Honored Contributor III
2,152 Views

I don't have any idea of how to find where Windows would put the memory mapped interface to PCI configuration space, but there is an interface to PCI configuration space described at http://msdn.microsoft.com/en-us/library/windows/hardware/ff536890%28v=vs.85%29.aspx

On Linux systems I have been able to find the location of the memory-mapped interface to PCI configuration space by executing "cat /proc/iomem" and looking for "PCI MMCONFIG 0".   On the configurations that I have played with, this is a 256 MiB region of memory-mapped IO space that provides access to the entire 4KiB extended PCI configuration space for all possible devices on a single bus.  With root access, you can open /dev/mem and mmap() at the offset provided by "/proc/iomem", then access the PCI configuration space directly.  You need to be careful to use loads and stores that are no larger than 32 bits, declaring the pointer as a "volatile uint32 *" is usually enough to prevent the compiler from messing with your code.

Since writing to PCI config space is potentially hazardous to your system, it is a good idea to spend a lot of time reading it first and comparing the results with the output of "lspci" (on Linux systems). 

0 Kudos
Bernard
Valued Contributor I
2,152 Views

 

Here is the partial output from !pci command executed on Win system.BARn registers contain memory addresses used by the device.

PCI Configuration Space (Segment:0000 Bus:24 Device:00 Function:00)

Common Header: 00:

VendorID 1657 Brocade Communications Systems 

 BAR0 c5fe0004 

 BAR1 00000000

 BAR2 c5fdc004

0 Kudos
shiv_a_
Beginner
2,152 Views

Thanks Everyone! I will read through the material and try with a linux HDD today

0 Kudos
McCalpinJohn
Honored Contributor III
2,152 Views

To reinforce Patrick Fay's point -- you should almost certainly not be trying to access PCI config space the way I do it -- you should use the library interfaces provided with Linux systems.

Linux systems starting with kernel 2.1 (??) support the /proc/bus/pci interfaces.

Linux systems starting with kernel 2.6 (??) support the /proc interfaces, but recommend using the newer "sysfs" interfaces (e.g., "/sys/bus/pci/devices/").

There seems to be relatively little documentation for the PCI access library, but the source code for the "lspci" utility and its supporting libraries does not look too tricky.  (Not completely transparent, but not too tricky.)    You can find the "lspci" source code and its libraries at ftp://ftp.kernel.org/pub/software/utils/pciutils/pciutils-3.2.1.tar.bz2

 

 

0 Kudos
shiv_a_
Beginner
2,152 Views

So, I finally got Redhat Linux booted on my system and was easily able to access the config space as John and Patrick mentioned. Thanks again!

0 Kudos
shiv_a_
Beginner
2,152 Views

I had one followup question.

From executing the ""cat /proc/iomem" I found that the memory configuration space on my processor is (8000000h - 8FFFFFFFh). 

So, using this base address, I tried to do a read to "B/D/F = 1/15/0" and Offset = 180h. This in the datasheet (xeon E5 vol2 datasheet) is described as SMB_STAT_0 (smbus status register). I notice that I read all Fs, which is seems wrong.

Sorry about all these simple questions, but, I am just learning and I really appreciate your help.

Can you tell me the correct method for address translation based on the information above?

0 Kudos
Patrick_F_Intel1
Employee
2,152 Views

You are headed into deep waters now.

I wouldn't pursue read/writing smbus directly unless I had a really, really, really, good reason. If you are trying to access the BMC then there are already utilities for that... the code you develop may only work on your system or motherboards exactly like yours... after you've spent a bunch of time developing the code and hopefully not permanently disabling your system.

Look at the code for lm-sensors... that utility reads (I think) the smbus.

Pat

0 Kudos
McCalpinJohn
Honored Contributor III
2,152 Views

You should really be using the PCI library interfaces instead of /dev/mem -- I spent a lot of years working in microprocessor design before I learned enough to be willing to risk fiddling with these interfaces directly.  

In this case the result you have obtained looks pretty easy -- the PCI spec says that an access to an address that is not "claimed" by any device results in a "master abort" -- showing up as all '1' bits set.   This suggests that you are not using the correct address calculation for the device that you are interested in.

0 Kudos
Reply