Contradiction in manual for PCIe read combining

[Kalia__Anuj]

I am reading "Intel Xeon Processor E5 v2 Product Family. Datasheet - Volume Two: Registers. Available here: http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v2-datasheet-vol-2.pdf. I am trying to find out the maximum PCIe read combining size on Ivy Bridge processors. In Section 8.12.3.50 (MISCCTRLSTS), bit #18 is defined as: "Max Read Completion Combine Size. This bit when set, will enable completion combining to a maximum of 256B (values less than or equal to 256B allowed). When clear, the maximum read completion combining size is 128B (values less than or equal to 256B allowed)." In Section 8.2.84 (miscctrlsts_0), however, bit #18 is defined as: "max_read_completion_combine_size: When set, all completions are returned without combining. Completions are naturally broken on cacheline boundaries, so all completions will be 64B or less." Which one is correct? Does someone know what the read combining size is for IvyBridge / SandyBridge / Haswell? Thanks!

[McCalpinJohn]

These two entries refer to different modes of operation for Port 3.

Section 8.2.84 is the more general reference, as suggested by the very long section title:

"Device 0 Function 0 DMI, Device 0 Function 0 PCIe, Device 1 Function 0-1, Device 2 Function 0-3 PCIe, Device 3 Function 0-3 PCIe"

The register descriptions in section 8.2 include all of the Device+Function combinations listed in the title.

Section 8.12.3.50 refers only to a specific "Non-Transparent Bridge" configuration, as suggested by that section's title:

"NTB Port 3A Configured as Primary Endpoint Device"

The register descriptions in section 8.3 refer only to Bus 0, Device 3, Function 0. 

Section 8.12 describes the three modes of operation available for Port 3.

[Kalia__Anuj]

Thank you for the answer. From lspci, I see 00:03:0 is a PCIe root port, so the definition from Section 8.12 applies. I get the value of MISCCTRLSTS as follows:

sudo cat /proc/bus/pci/00/03.0 | xxd -b -c 8 | grep 188                             
0000188: 00010000 00010000 00010000 01100100 00001110 00000000 00000000 00000000  ...d....
                                _                                                   

The underlined bit is the 18th bit of the 64-bit register, and it is 0. It is also 0 on a Haswell-based server that I tested, although the manual for Xeon E5-v3 says that the bit is 1 by default.

Now I want to set the bit to 1. The initial hex value of the longword at offset 0x188 is 0x64101010. When I execute setpci, the register does not change. How can I fix this? Is setpci the correct tool for this job?

sudo setpci -s 0:3:0 0x188.L=0x64141010

[McCalpinJohn]

This is getting outside of my experience, but I would guess that these bits have to be set by the BIOS at the initial setup of the PCI device.