topic Re: How to get CXL PMU event sources in Linux(perf)? in Intel® Xeon® Processor and Server Products

How to get CXL PMU event sources in Linux(perf)?

Shunyu — Mon, 22 Jul 2024 15:23:28 GMT

Hi, I'm trying to profiling performance of cxl type3 devices using CXL PMON mentioned in Xeon manual, but i can't see any cxl related event sources under /sys/bus/event_sources, so how to get performance data in perf?

Linux kernel is 6.5.0, ubuntu 22.04.

Re:How to get CXL PMU event sources in Linux(perf)?

NormanS_Intel — Wed, 24 Jul 2024 05:57:19 GMT

Hello Shunyu,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer

Re: How to get CXL PMU event sources in Linux(perf)?

Shunyu — Wed, 24 Jul 2024 08:23:27 GMT

Update: I've tried these raw events in https://github.com/torvalds/linux/blob/66ebbdfdeb093e097399b1883390079cd4c3022b/tools/perf/pmu-events/arch/x86/sapphirerapids/uncore-cxl.json , but the result i got is really weird, it looks like these are totally unrelated events.

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Wed, 24 Jul 2024 10:53:09 GMT

Hi Shunyu,

Greetings for the day!

Regarding your query, we recommend consulting your Linux team or IT support regarding the Compute Express Link (CXL) Performance Monitoring Unit (CPMU) and the Linux Kernel documentation.

Should you have any further questions or need assistance with other issues, please feel free to reach out.

Thank you for choosing Intel products and services.

Best regards,

Manoranjan.

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Fri, 26 Jul 2024 18:33:02 GMT

Hello Shunyu,

Greetings for the day!

I hope this message finds you well.

We are following up to find out if you were able to find the information we provided. Please reply to confirm, so we can continue helping on a resolution. Looking forward to receiving your reply.

Regards,

Manoranjan.

Re: Re:How to get CXL PMU event sources in Linux(perf)?

Shunyu — Sat, 27 Jul 2024 08:57:47 GMT

A bunch of useless answers.

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Sat, 27 Jul 2024 10:18:36 GMT

Hello Shunyu,

Greetings of the day!

Apologies for the inconvenience caused.

Could you please confirm which Xeon manual you are referring to? If possible, could you also provide the link to the manual and specify the processor model?

Regards,

Manoranjan

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Tue, 30 Jul 2024 14:46:13 GMT

Hello Shunyu,

Greetings for the day!

I hope this message finds you well.

We are following up to find out if you were able to find the information we provided. Please reply to confirm, so we can continue helping on a resolution. Looking forward to receiving your reply.

Regards,

Manoranjan.

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Tue, 06 Aug 2024 12:50:27 GMT

Hello Shunyu,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan Das.

Re:How to get CXL PMU event sources in Linux(perf)?

ManoranjanDas — Fri, 09 Aug 2024 12:18:58 GMT

Hello Shunyu,

Greetings for the day!

We would like to inform you that we are closing this request as no response has been received from our previous follow-ups. Please don't hesitate to ask any further questions in the future. Feel free to start a new conversation, as this thread will no longer be monitored.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Mon, 19 Aug 2024 14:34:32 GMT

https://www.kernel.org/doc/html/latest/admin-guide/perf/cxl.html#cxl-performance-monitoring-unit-cpmu
Maybe help.

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Mon, 19 Aug 2024 16:41:12 GMT

Hello,

I am seeking guidance on utilizing perf to monitor events as defined in the uncore_cxl.json file located within tools/perf/pmu-events/arch/x86/sapphirerapids/uncore_cxl.json. Despite the presence of these events in the JSON file, they do not appear when I execute the perf list command. Consequently, I attempted to employ this command below to monitor these events, only to encounter an issue where certain count units are labeled as "unsupported" in the output.

perf stat -e $EVENT_NAME:$UMASK my_command hwt@cxl2-2288H-V7:~/workspace/mlc_v3.11a/Linux$ perf stat -I 500 -e 0x02:0x10 ./test_seq_write_bw.sh # time counts unit events 0.500505514 <not supported> 0x02:0x10 1.001040375 <not supported> 0x02:0x10 1.501550639 <not supported> 0x02:0x10 2.002069833 <not supported> 0x02:0x10 2.003776122 <not supported> 0x02:0x10

Could you provide any insights or solutions to address this problem? How do you specifically use perf for monitoring the events mentioned in the uncore_cxl.json file?

Thank you for your time and assistance.

Re: How to get CXL PMU event sources in Linux(perf)?

Shunyu — Tue, 20 Aug 2024 03:09:57 GMT

事实上perf确实测不到，但是可以通过intel-pcm中的pcm-raw访问，比如 `pcm-raw -e cxlcm/config=4098` ，这个编号可以看 `pcm/src/cpucounter.cpp` 中的 `PCM::programCXLCM()` 也即第10180行，具体寄存器在XCC (Codename Sapphire Rapids) Uncore Performance .

BTW，这些数值基本只代表带宽，事实上pcm测得的cxl带宽就是其中两个寄存器换算的，没有太大意义

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Tue, 20 Aug 2024 08:15:10 GMT

十分感谢您的回复！

我还有几个问题想向您请教一下。

1. perf list | grep cxl 输出的事件中有一部分Kernel PMU events, 请问这部分events和uncore_cxl.json中的寄存器有关系吗？
https://www.kernel.org/doc/html/latest/admin-guide/perf/cxl.html#cxl-performance-monitoring-unit-cpmu

2. 我在pcm的源码中找到了两部分和CXL相关的事件统计，一部分在您说的pcm/src/cpucounter.cpp中，如下图所示：

另一部分在pcm/src/pcm-memory.cpp中，如下图所示：

请问我在使用pcm-raw指令的时候在cxlcm/config=xxx后面输入编号的时候是输入UNC_PMON_CTL_EVENT+UNC_PMON_CTL_UMASK的组合即可吗？比如`pcm-raw -e cxlcm/config=0210`对应CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA。输出结果如下：

这个输出结果正确吗？应该如何解读这个结果呢？

pcm-memory.cpp这里的UNC_PMON_CTL_UMASK_EXT是UMASK的扩展？我在uncore-cache.json找到了这里pcm-memory.cpp中的这些event, 最终的UMASK=UNC_PMON_CTL_UMASK_EXT+UNC_PMON_CTL_UMASK，这样理解对吗？但是Event Code看起来对不上。

谢谢！十分期待您的回复！

Re: How to get CXL PMU event sources in Linux(perf)?

Shunyu — Tue, 20 Aug 2024 08:28:18 GMT

1. 没有关系。你列的是CXL Revision 3.0 Specification中定义的PMU，首先FPGA不支持，其次CPU目前不支持，我提到的是Intel在自家CPU(甚至自家FPGA)上定制的PMU

2. cpucounter.cpp 中的就是我提到的文档中的pmu，而 pcm-memory.cpp 中你搜索到的是被分类在UNC_CHA下面的，这些事件用perf list就可以查询到

3. 你的用法是错误的，你只要看一下这两个宏定义就会发现UNC_PMON_CTL_UMASK其实是左移8位，因此应该是0x1002，但是！pcm-raw工具的这个参数是十进制，因此应该是cxlcm/config=4098

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Tue, 20 Aug 2024 09:08:31 GMT

谢谢！

不过实在不好意思，这里的左移八位的意思我没太搞懂？是最终cxlcm/config的值是UNC_PMON_CTL_UMASK左移八位再加上UNC_PMON_CTL_EVENT，也就是<umask><eventcode>的格式？

如果按照uncore-cxl.json中给的数据使用pcm-raw的话，是用cxlcm/config的值是x1041再转换成10进制吗？

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Tue, 20 Aug 2024 09:16:31 GMT

而且在我使用perf stat去监控被分类在UNC_CHA下面的事件时，就会报错Bad event name.

Re: How to get CXL PMU event sources in Linux(perf)?

Shunyu — Tue, 20 Aug 2024 09:17:58 GMT

是的，umask_ext << 32 + umask << 8 + event ，如果用左图的Rx就是0x1041(4161)

在`pcm-raw -e aaa -e bbb -e ccc -e ddd` 中 `aaa`会出现在CXLCMEvents的第0位，`bbb`出现在数组的第1位，以此类推。需要注意这些寄存器允许出现的位置(Counters)在手册上是有限制的。

另外，这些个寄存器除了带宽之外没有什么用

Re: How to get CXL PMU event sources in Linux(perf)?

Shunyu — Tue, 20 Aug 2024 09:21:47 GMT

perf的event请使用全大写。。

Re: How to get CXL PMU event sources in Linux(perf)?

pudding_art — Tue, 20 Aug 2024 09:24:07 GMT

谢谢！谢谢！太感谢了！毕设要用到这个看带宽，终于解决了！🥹