Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

Information about PCM PCIe counters

Kalia__Anuj
Beginner
2,583 Views

Hi everyone.

I have been working on measuring the PCIe activity of network cards and I wanted to understand PCM counters better. I'm running the pcm-pcie.x executable on a Haswell server which displays the following counters (full event description here: http://pastebin.com/pnuj1eKu):

  • PCIeRdCur (PCIe read current transfer (full cache line)
  • RFO (Demand Data RFO)
  • CRd (Demand Code Read)
  • DRd (Demand Data Read)
  • ItoM (PCIe write full cache line)
  • PRd (MMIO Read)
  • WiL (MMIO Write)

I had these questions:

  • What is the difference between PCIeRdCur and DRd? PCIeRdCur measures the number of partial and full cache line reads. Does it miss any PCIe reads that are captured by DRd, or does PCIeRdCur include DRd? I'm seeing non-zero values for both these counters.
  • The description printed by pcm-pcie.x says that WiL measures traffic for "PCI devices writing to memory - application reads from disk/network/PCIe device", but it also describes it as "MMIO Writes (Full/Partial)". Aren't these two descriptions contradictory, since MMIO writes involve the CPU writing to PCIe devices?
  • In the counter description, RFO appears twice:
    • RFO * - Demand Data RFO
    • RFO - PCIe partial write
    • If think that this is OK because a PCIe partial write will perform a read-for-ownership, making these counters equivalent. Can someone verify this.

Thank you for your help!

Anuj

0 Kudos
1 Solution
Patrick_L_Intel
Employee
2,583 Views

Anuj K. wrote:

Thanks for the information, Patrick. This is very helpful.

Are the DRd and CRd counters related to PCIe activity in some way?

  • E5-2600v3 family support several type of PCIe read flow, PCIeRdCur (default), DRd (Data Read), CRd (Code Read), RFO are all possible depending on the I/O coherency setting in the BIOS. You can change it to see if it has any impact to your specific application performance.

I think that knowing the number of LLC reads is useful while studying the other PCIe counters, but is there a bigger reason why these counters are displayed by pcm-pcie.x?

View solution in original post

0 Kudos
3 Replies
Patrick_L_Intel
Employee
2,583 Views

Anuj K. wrote:

I had these questions:

  • What is the difference between PCIeRdCur and DRd? PCIeRdCur measures the number of partial and full cache line reads. Does it miss any PCIe reads that are captured by DRd, or does PCIeRdCur include DRd? I'm seeing non-zero values for both these counters.

If you have the default I/O coherency setting in the BIOS, the default interpretations are:

PCIeRdCur means IO reading from memory

DRd means CPU doing data read to last level cache

  • The description printed by pcm-pcie.x says that WiL measures traffic for "PCI devices writing to memory - application reads from disk/network/PCIe device", but it also describes it as "MMIO Writes (Full/Partial)". Aren't these two descriptions contradictory, since MMIO writes involve the CPU writing to PCIe devices?

Thanks for pointing it out. WiL is wrongly categorized. It is not part of PCIe writing to memory. It's just MMIO write from CPU. We'll fix it in the next release.

  • In the counter description, RFO appears twice:
    • RFO * - Demand Data RFO
    • RFO - PCIe partial write
    • If think that this is OK because a PCIe partial write will perform a read-for-ownership, making these counters equivalent. Can someone verify this.

When you have PCIe partial write, yes, it will generate RFO. There is an optional settings in the BIOS to make the PCIe read flow to use RFO as well, that's why it's in the read category as well (but as far as I know, no one is using RFO read flow)

0 Kudos
Kalia__Anuj
Beginner
2,583 Views

Thanks for the information, Patrick. This is very helpful.

Are the DRd and CRd counters related to PCIe activity in some way? I think that knowing the number of LLC reads is useful while studying the other PCIe counters, but is there a bigger reason why these counters are displayed by pcm-pcie.x?

 

0 Kudos
Patrick_L_Intel
Employee
2,584 Views

Anuj K. wrote:

Thanks for the information, Patrick. This is very helpful.

Are the DRd and CRd counters related to PCIe activity in some way?

  • E5-2600v3 family support several type of PCIe read flow, PCIeRdCur (default), DRd (Data Read), CRd (Code Read), RFO are all possible depending on the I/O coherency setting in the BIOS. You can change it to see if it has any impact to your specific application performance.

I think that knowing the number of LLC reads is useful while studying the other PCIe counters, but is there a bigger reason why these counters are displayed by pcm-pcie.x?

0 Kudos
Reply