PCIe NUMA Affinity on Haswell & Ivy Bridge

[John_S_2]

On Ivy Bridge and Haswell dual socket servers, does each NUMA node have access to its own PCIe slot?  Or does all PCIe traffic flow through one node?

And what is the easiest way of telling which NUMA node is closest to the PCIe slot I'm using on Windows Server 2008 R2?

[Bernard]

Check following thread: https://software.intel.com/en-us/forums/topic/379378

[McCalpinJohn]

Xeon E5-2xxx processors (Sandy Bridge/v1, Ivy Bridge/v2, and Haswell/v3) have PCIe interfaces on each chip.  Whether these are all exposed on the motherboard depends on the vendor.  Typically systems that are physically small expose fewer slots than systems that are physically large, but the motherboard vendor documentation is a good place to look for information on which slot is attached to which processor socket.

On Linux systems the "hardware topology" infrastructure queries the hardware to determine the physical layout of the various cores and PCIe devices.  On a dual-socket Xeon E5-2680 (Sandy Bridge) node, the "lstopo" command returns something like:

Machine (64GB)
  NUMANode L#0 (P#0 32GB) + Socket L#0 + L3 L#0 (20MB)
    L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
    L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2)
    L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#4)
    L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#6)
    L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#8)
    L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#10)
    L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#12)
    L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#14)
  NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (20MB)
    L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#1)
    L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#3)
    L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#5)
    L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#7)
    L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#9)
    L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#11)
    L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#13)
    L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#15)
  HostBridge L#0
    PCIBridge
      PCI 14e4:165f
        Net L#0 "eth2"
      PCI 14e4:165f
        Net L#1 "eth3"
    PCIBridge
      PCI 14e4:165f
        Net L#2 "eth0"
      PCI 14e4:165f
        Net L#3 "eth1"
    PCIBridge
      PCI 1000:005b
        Block L#4 "sda"
        Block L#5 "sdb"
    PCIBridge
      PCIBridge
        PCIBridge
          PCIBridge
            PCI 102b:0534
    PCI 8086:1d02
      Block L#6 "sr0"
  HostBridge L#8
    PCIBridge
      PCI 15b3:1003
        Net L#7 "eth4"
        Net L#8 "ib0"
        OpenFabrics L#9 "mlx4_0"

The first 19 lines tell which cores are in the two sockets, while lines starting with "HostBridge L#0" list the PCIe devices attached to socket 0 (including four Ethernet devices, two disks, etc), and the lines starting with "HostBridge L#8" list the PCIe devices attached to socket 1 (including an Infiniband + Ethernet card).

Windows Server 2008 might have similar information in the system management screens, but I don't have administrative access to any Windows systems to check.