Failed to access reg by pci address

[Zhu_G_]

Hi community!

I am trying to access register by PCI address using setpci command.

But I came across the following error message.

pcilib: sysfs_write: tried to write 1 bytes at xxx, but only 0 succeeded

BTW: I am using intel e5-2600v3 series cpu with command setpci -s 7f:14.0 190.l=ffffffff

[McCalpinJohn]

I think that "setpci" requires that you precede hex numbers with "0x".

A script that I use to set up the IMC counters on a Xeon E5-2660v3 is appended:

#!/bin/bash

# IMC Performance Events
# Most of our nodes have 2 channels on each of 2 IMCs
#     Buses [7f,ff], Devices [0x14,0x17], Functions [0,1]
# Each of these has four programmable counters
# Counter   Offset    Value       Description
#    0      0xD8   0x00400B01     ACT.(READ+WRITE+BYPASS) -- Umasks are new with Haswell
#    1      0xDC   0x00400304     CAS_COUNT.READS
#    2      0xE0   0x00400C04	  CAS_COUNT.WRITES
#    3      0xE4   0x00400102	  PRE_COUNT.MISS -- page closes due to page conflicts

echo "Setting up IMC Performance Counters"
for BUS in 7f ff
do
	for DEVICE in 14 17
	do
		for FUNCTION in 0 1
		do
			lspci -s ${BUS}:${DEVICE}.$FUNCTION
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xD8.L=0x00400B01
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xDC.L=0x00400304
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xE0.L=0x00400C04
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xE4.L=0x00400102
		done
	done
done

[Zhu_G_]

John D. McCalpin wrote:

I think that "setpci" requires that you precede hex numbers with "0x".

A script that I use to set up the IMC counters on a Xeon E5-2660v3 is appended:

#!/bin/bash

# IMC Performance Events
# Most of our nodes have 2 channels on each of 2 IMCs
#     Buses [7f,ff], Devices [0x14,0x17], Functions [0,1]
# Each of these has four programmable counters
# Counter   Offset    Value       Description
#    0      0xD8   0x00400B01     ACT.(READ+WRITE+BYPASS) -- Umasks are new with Haswell
#    1      0xDC   0x00400304     CAS_COUNT.READS
#    2      0xE0   0x00400C04	  CAS_COUNT.WRITES
#    3      0xE4   0x00400102	  PRE_COUNT.MISS -- page closes due to page conflicts

echo "Setting up IMC Performance Counters"
for BUS in 7f ff
do
	for DEVICE in 14 17
	do
		for FUNCTION in 0 1
		do
			lspci -s ${BUS}:${DEVICE}.$FUNCTION
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xD8.L=0x00400B01
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xDC.L=0x00400304
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xE0.L=0x00400C04
			setpci -s ${BUS}:${DEVICE}.${FUNCTION} 0xE4.L=0x00400102
		done
	done
done

Hi Dr.B!

I have already tried setpci with uncore iMC counter and it seems to be working correctly.

I used both pread and setpci to access 7f:14.0 with offset of pmu controller and it works.But when I am trying thrt_pwr_dimm_[0:2] it failed.

the command I used is 

setpci -s 7f:14.0 0x190.l=0xffffffff

I think it might be the way I am trying to access thrt_pwr_dimm is wrong or what?

BTW: I am referencing xeon e5-1600/2400/2600/4600 v3 datasheet volume 2

[Zhu_G_]

A lot of regs accessed by PCI address are all set to 0xffffffff on my CPU .

1. chn_temp_cfg

2. chn_temp_stat

3. dimm_temp_oem

4. dimm_temp_th

....

I think it might be some regs are not available on my CPU?

My CPU is  Intel(R) Xeon(R) CPU E5-2603 v3

[McCalpinJohn]

I don't remember where I read it, but I am pretty sure that some of the registers described in Volume 2 of the processor datasheet are only accessible by the BIOS.   The response of 0xffffffff corresponds to a "master abort" (as described in Section 1.2.1.4), which means that the device/function is not supported (though it may be accessible to the BIOS -- see Section 1.2.1.5). 

Many of the functions related to the DRAM are locked by the BIOS, since it makes no sense to attempt to change them at runtime -- for example the address mapping can only be changed if you are using the cache as main memory during the early steps of the boot process.  On the other hand the registers that you are trying to access don't seem to be in this category -- they seem safe enough to read/write at runtime.

In this case it looks like the BIOS blocks access to the "extended PCI configuration space" (offsets from 256B to 4095B) for this device.  A quick test on one of my systems shows that the BIOS blocks access to "extended PCI configuration space" for all PCI configuration space devices/functions for the uncore.

[Zhu_G_]

Thanks for your answer Dr. Bandwidth.

I did a little digging and I found that intel E7 may support this feature. However I don't have an E7 CPU at hand to confirm it.

I wonder if I can enable access to these registers by tuning my BIOS? Or is there any lead that I can follow? 

[Zhu_G_]

Hi Dr.Bandwidth.

I wonder how you have tuned your BIOS to enable these registers? I have been working on it on my dell t430 and still no clue.

[McCalpinJohn]

The BIOS(es) on my systems do not allow access to the extended PCI configuration space for any of the Uncore devices.   The "master abort" response (0xffffffff) shows that these are being blocked on purpose, so the only way to change the behavior would be to rewrite the BIOS.   Many years ago there was an attempt to build an "open BIOS" for x86_64 systems, but it did not get very far.  Since then the systems have gotten dramatically more complex and the BIOS is required to handle a lot more complexity than it used to, so I don't expect an open source BIOS for Intel-based systems any time soon.