How to get CXL PMU event sources in Linux(perf)?

[Shunyu]

Hi, I'm trying to profiling performance of cxl  type3 devices using CXL PMON mentioned in Xeon manual, but i can't see any cxl related event sources under /sys/bus/event_sources, so how to get performance data in perf?

Linux kernel is 6.5.0, ubuntu 22.04.

[NormanS_Intel]

Hello Shunyu,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer

[Shunyu]

Update: I've tried these raw events in https://github.com/torvalds/linux/blob/66ebbdfdeb093e097399b1883390079cd4c3022b/tools/perf/pmu-events/arch/x86/sapphirerapids/uncore-cxl.json , but the result i got is really weird, it looks like these are totally unrelated events.

[pudding_art]

Hello,

I am seeking guidance on utilizing perf to monitor events as defined in the uncore_cxl.json file located within tools/perf/pmu-events/arch/x86/sapphirerapids/uncore_cxl.json. Despite the presence of these events in the JSON file, they do not appear when I execute the perf list command. Consequently, I attempted to employ this command below to monitor these events, only to encounter an issue where certain count units are labeled as "unsupported" in the output.

 perf stat -e $EVENT_NAME:$UMASK my_command

hwt@cxl2-2288H-V7:~/workspace/mlc_v3.11a/Linux$ perf stat -I 500 -e 0x02:0x10 ./test_seq_write_bw.sh
#           time             counts unit events
     0.500505514    <not supported>      0x02:0x10                                                             
     1.001040375    <not supported>      0x02:0x10                                                             
     1.501550639    <not supported>      0x02:0x10                                                             
     2.002069833    <not supported>      0x02:0x10                                                             
     2.003776122    <not supported>      0x02:0x10

Could you provide any insights or solutions to address this problem? How do you specifically use perf for monitoring the events mentioned in the uncore_cxl.json file?

Thank you for your time and assistance.

[Shunyu]

事实上perf确实测不到，但是可以通过intel-pcm中的pcm-raw访问，比如 `pcm-raw -e cxlcm/config=4098` ， 这个编号可以看 `pcm/src/cpucounter.cpp` 中的 `PCM::programCXLCM()` 也即第10180行，具体寄存器在XCC (Codename Sapphire Rapids) Uncore Performance .

BTW，这些数值基本只代表带宽，事实上pcm测得的cxl带宽就是其中两个寄存器换算的，没有太大意义

[pudding_art]

十分感谢您的回复！

我还有几个问题想向您请教一下。

1. perf list | grep cxl 输出的事件中有一部分Kernel PMU events, 请问这部分events和uncore_cxl.json中的寄存器有关系吗？
https://www.kernel.org/doc/html/latest/admin-guide/perf/cxl.html#cxl-performance-monitoring-unit-cpmu

2. 我在pcm的源码中找到了两部分和CXL相关的事件统计，一部分在您说的pcm/src/cpucounter.cpp中，如下图所示：

另一部分在pcm/src/pcm-memory.cpp中，如下图所示：

请问我在使用pcm-raw指令的时候在cxlcm/config=xxx后面输入编号的时候是输入UNC_PMON_CTL_EVENT+UNC_PMON_CTL_UMASK的组合即可吗？比如`pcm-raw -e cxlcm/config=0210`对应CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA。输出结果如下：

这个输出结果正确吗？应该如何解读这个结果呢？

pcm-memory.cpp这里的UNC_PMON_CTL_UMASK_EXT是UMASK的扩展？我在uncore-cache.json找到了这里pcm-memory.cpp中的这些event, 最终的UMASK=UNC_PMON_CTL_UMASK_EXT+UNC_PMON_CTL_UMASK，这样理解对吗？但是Event Code看起来对不上。

谢谢！十分期待您的回复！

[Shunyu]

1. 没有关系。你列的是CXL Revision 3.0 Specification中定义的PMU，首先FPGA不支持，其次CPU目前不支持，我提到的是Intel在自家CPU(甚至自家FPGA)上定制的PMU

2. cpucounter.cpp 中的就是我提到的文档中的pmu，而 pcm-memory.cpp 中你搜索到的是被分类在UNC_CHA下面的，这些事件用perf list就可以查询到

3. 你的用法是错误的，你只要看一下这两个宏定义就会发现UNC_PMON_CTL_UMASK其实是左移8位，因此应该是0x1002，但是！pcm-raw工具的这个参数是十进制，因此应该是cxlcm/config=4098

[pudding_art]

谢谢！

不过实在不好意思，这里的左移八位的意思我没太搞懂？是最终cxlcm/config的值是UNC_PMON_CTL_UMASK左移八位再加上UNC_PMON_CTL_EVENT，也就是<umask><eventcode>的格式？

如果按照uncore-cxl.json中给的数据使用pcm-raw的话，是用cxlcm/config的值是x1041再转换成10进制吗？

[Shunyu]

是的，umask_ext << 32 + umask << 8 + event ，如果用左图的Rx就是0x1041(4161)

在`pcm-raw -e aaa -e bbb -e ccc -e ddd` 中 `aaa`会出现在CXLCMEvents的第0位，`bbb`出现在数组的第1位，以此类推。需要注意这些寄存器允许出现的位置(Counters)在手册上是有限制的。

另外，这些个寄存器除了带宽之外没有什么用

[pudding_art]

谢谢！谢谢！太感谢了！毕设要用到这个看带宽，终于解决了！🥹

[Shunyu]

你要看带宽直接用pcm-memory就行了，不用这个

nudt也有cxl吗

[pudding_art]

因为想统计每个task的带宽，所以想找一下有没有相关的事件。pcm-memory是不是只能看整个流经CXL端口的流量。

嗯呢，nudt也有

[pudding_art]

而且在我使用perf stat去监控被分类在UNC_CHA下面的事件时，就会报错Bad event name.

[Shunyu]

perf的event请使用全大写。。

[pudding_art]

全大写看起来也不行

[Shunyu]

你用的是什么版本的kernel？

加vx吧，太麻烦了

[pudding_art]

ID: Pudding_Art

[ManoranjanDas]

Hi Shunyu,

Greetings for the day!

Regarding your query, we recommend consulting your Linux team or IT support regarding the Compute Express Link (CXL) Performance Monitoring Unit (CPMU) and the Linux Kernel documentation.

Should you have any further questions or need assistance with other issues, please feel free to reach out.

Thank you for choosing Intel products and services.

Best regards,

Manoranjan.

[ManoranjanDas]

Hello Shunyu,

Greetings for the day!

I hope this message finds you well.

We are following up to find out if you were able to find the information we provided. Please reply to confirm, so we can continue helping on a resolution. Looking forward to receiving your reply.

Regards,

Manoranjan.

[Shunyu]

A bunch of useless answers.

[ManoranjanDas]

Hello Shunyu,

Greetings of the day!

Apologies for the inconvenience caused.

Could you please confirm which Xeon manual you are referring to? If possible, could you also provide the link to the manual and specify the processor model?

Regards,

Manoranjan