Information about PCM PCIe counters

[Kalia__Anuj]

Hi everyone.

I have been working on measuring the PCIe activity of network cards and I wanted to understand PCM counters better. I'm running the pcm-pcie.x executable on a Haswell server which displays the following counters (full event description here: http://pastebin.com/pnuj1eKu):

• PCIeRdCur (PCIe read current transfer (full cache line)
• RFO (Demand Data RFO)
• CRd (Demand Code Read)
• DRd (Demand Data Read)
• ItoM (PCIe write full cache line)
• PRd (MMIO Read)
• WiL (MMIO Write)

I had these questions:

• What is the difference between PCIeRdCur and DRd? PCIeRdCur measures the number of partial and full cache line reads. Does it miss any PCIe reads that are captured by DRd, or does PCIeRdCur include DRd? I'm seeing non-zero values for both these counters.
• The description printed by pcm-pcie.x says that WiL measures traffic for "PCI devices writing to memory - application reads from disk/network/PCIe device", but it also describes it as "MMIO Writes (Full/Partial)". Aren't these two descriptions contradictory, since MMIO writes involve the CPU writing to PCIe devices?
• In the counter description, RFO appears twice:
  • RFO * - Demand Data RFO
  • RFO - PCIe partial write
  • If think that this is OK because a PCIe partial write will perform a read-for-ownership, making these counters equivalent. Can someone verify this.

Thank you for your help!

Anuj

[Patrick_L_Intel]

Anuj K. wrote:

I had these questions:

• What is the difference between PCIeRdCur and DRd? PCIeRdCur measures the number of partial and full cache line reads. Does it miss any PCIe reads that are captured by DRd, or does PCIeRdCur include DRd? I'm seeing non-zero values for both these counters.

If you have the default I/O coherency setting in the BIOS, the default interpretations are:

PCIeRdCur means IO reading from memory

DRd means CPU doing data read to last level cache

• The description printed by pcm-pcie.x says that WiL measures traffic for "PCI devices writing to memory - application reads from disk/network/PCIe device", but it also describes it as "MMIO Writes (Full/Partial)". Aren't these two descriptions contradictory, since MMIO writes involve the CPU writing to PCIe devices?

Thanks for pointing it out. WiL is wrongly categorized. It is not part of PCIe writing to memory. It's just MMIO write from CPU. We'll fix it in the next release.

• In the counter description, RFO appears twice:
  • RFO * - Demand Data RFO
  • RFO - PCIe partial write
  • If think that this is OK because a PCIe partial write will perform a read-for-ownership, making these counters equivalent. Can someone verify this.

When you have PCIe partial write, yes, it will generate RFO. There is an optional settings in the BIOS to make the PCIe read flow to use RFO as well, that's why it's in the read category as well (but as far as I know, no one is using RFO read flow)

[Kalia__Anuj]

Thanks for the information, Patrick. This is very helpful.

Are the DRd and CRd counters related to PCIe activity in some way? I think that knowing the number of LLC reads is useful while studying the other PCIe counters, but is there a bigger reason why these counters are displayed by pcm-pcie.x?

[Patrick_L_Intel · Accepted solution]

Anuj K. wrote:

Thanks for the information, Patrick. This is very helpful.

Are the DRd and CRd counters related to PCIe activity in some way?

• E5-2600v3 family support several type of PCIe read flow, PCIeRdCur (default), DRd (Data Read), CRd (Code Read), RFO are all possible depending on the I/O coherency setting in the BIOS. You can change it to see if it has any impact to your specific application performance.

I think that knowing the number of LLC reads is useful while studying the other PCIe counters, but is there a bigger reason why these counters are displayed by pcm-pcie.x?

• I knew this question must come up :) Besides the above reason, the reason DRd, CRd, and RFO are included in the pcm-pcie because it uses CBo performance monitoring facility to track L3 cache activity. Since both CPU and I/O can make use of L3 cache, the utility track both sources. For more information, please check below document. https://www-ssl.intel.com/content/www/us/en/processors/xeon/xeon-e5-v3-uncore-performance-monitoring.html