Server Products
Data Center Products including boards, integrated systems, Intel® Xeon® Processors, RAID Storage, and Intel® Xeon® Processors
4917 Discussions

How to get CXL PMU event sources in Linux(perf)?

Shunyu
Novice
1,867 Views

Hi, I'm trying to profiling performance of cxl  type3 devices using CXL PMON mentioned in Xeon manual, but i can't see any cxl related event sources under /sys/bus/event_sources, so how to get performance data in perf?

 

Linux kernel is 6.5.0, ubuntu 22.04.

 

0 Kudos
24 Replies
NormanS_Intel
Moderator
1,714 Views

Hello Shunyu,

 

Thank you for posting in the community!

 

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

 

Best regards,

Norman S.

Intel Customer Support Engineer


0 Kudos
Shunyu
Novice
1,688 Views

Update: I've tried these raw events in https://github.com/torvalds/linux/blob/66ebbdfdeb093e097399b1883390079cd4c3022b/tools/perf/pmu-events/arch/x86/sapphirerapids/uncore-cxl.json , but the result i got is really weird, it looks like these are totally unrelated events.

0 Kudos
pudding_art
Novice
1,192 Views

Hello,

I am seeking guidance on utilizing perf to monitor events as defined in the uncore_cxl.json file located within tools/perf/pmu-events/arch/x86/sapphirerapids/uncore_cxl.json. Despite the presence of these events in the JSON file, they do not appear when I execute the perf list command. Consequently, I attempted to employ this command below to monitor these events, only to encounter an issue where certain count units are labeled as "unsupported" in the output.

 perf stat -e $EVENT_NAME:$UMASK my_command

hwt@cxl2-2288H-V7:~/workspace/mlc_v3.11a/Linux$ perf stat -I 500 -e 0x02:0x10 ./test_seq_write_bw.sh
#           time             counts unit events
     0.500505514    <not supported>      0x02:0x10                                                             
     1.001040375    <not supported>      0x02:0x10                                                             
     1.501550639    <not supported>      0x02:0x10                                                             
     2.002069833    <not supported>      0x02:0x10                                                             
     2.003776122    <not supported>      0x02:0x10

pudding_art_0-1724085644802.png

 

Could you provide any insights or solutions to address this problem? How do you specifically use perf for monitoring the events mentioned in the uncore_cxl.json file?

Thank you for your time and assistance.

0 Kudos
Shunyu
Novice
1,152 Views

事实上perf确实测不到,但是可以通过intel-pcm中的pcm-raw访问,比如 `pcm-raw -e cxlcm/config=4098` , 这个编号可以看 `pcm/src/cpucounter.cpp` 中的 `PCM::programCXLCM()` 也即第10180行,具体寄存器在XCC (Codename Sapphire Rapids) Uncore Performance .

 

BTW,这些数值基本只代表带宽,事实上pcm测得的cxl带宽就是其中两个寄存器换算的,没有太大意义

pudding_art
Novice
1,136 Views

十分感谢您的回复!

我还有几个问题想向您请教一下。

1. perf list | grep cxl 输出的事件中有一部分Kernel PMU events, 请问这部分events和uncore_cxl.json中的寄存器有关系吗?
https://www.kernel.org/doc/html/latest/admin-guide/perf/cxl.html#cxl-performance-monitoring-unit-cpmu

pudding_art_2-1724140241006.png

2. 我在pcm的源码中找到了两部分和CXL相关的事件统计,一部分在您说的pcm/src/cpucounter.cpp中,如下图所示:

pudding_art_3-1724140381025.png

另一部分在pcm/src/pcm-memory.cpp中,如下图所示:

pudding_art_4-1724140430603.png

请问我在使用pcm-raw指令的时候在cxlcm/config=xxx后面输入编号的时候是输入UNC_PMON_CTL_EVENT+UNC_PMON_CTL_UMASK的组合即可吗?比如`pcm-raw -e cxlcm/config=0210`对应CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA。输出结果如下:

pudding_art_5-1724140625237.png

这个输出结果正确吗?应该如何解读这个结果呢?

pcm-memory.cpp这里的UNC_PMON_CTL_UMASK_EXT是UMASK的扩展?我在uncore-cache.json找到了这里pcm-memory.cpp中的这些event, 最终的UMASK=UNC_PMON_CTL_UMASK_EXT+UNC_PMON_CTL_UMASK,这样理解对吗?但是Event Code看起来对不上。

pudding_art_0-1724141061475.png

 

谢谢!十分期待您的回复!

 

 

0 Kudos
Shunyu
Novice
1,128 Views

1. 没有关系。你列的是CXL Revision 3.0 Specification中定义的PMU,首先FPGA不支持,其次CPU目前不支持,我提到的是Intel在自家CPU(甚至自家FPGA)上定制的PMU

 

2. cpucounter.cpp 中的就是我提到的文档中的pmu,而 pcm-memory.cpp 中你搜索到的是被分类在UNC_CHA下面的,这些事件用perf list就可以查询到

 

3. 你的用法是错误的,你只要看一下这两个宏定义就会发现UNC_PMON_CTL_UMASK其实是左移8位,因此应该是0x1002,但是!pcm-raw工具的这个参数是十进制,因此应该是cxlcm/config=4098

 

 

pudding_art
Novice
1,116 Views

谢谢!

 

不过实在不好意思,这里的左移八位的意思我没太搞懂?是最终cxlcm/config的值是UNC_PMON_CTL_UMASK左移八位再加上UNC_PMON_CTL_EVENT,也就是<umask><eventcode>的格式?

如果按照uncore-cxl.json中给的数据使用pcm-raw的话,是用cxlcm/config的值是x1041再转换成10进制吗?

pudding_art_0-1724144507173.png

 

 

0 Kudos
Shunyu
Novice
1,102 Views

是的,umask_ext << 32 + umask << 8 + event ,如果用左图的Rx就是0x1041(4161)

在`pcm-raw -e aaa -e bbb -e ccc -e ddd` 中 `aaa`会出现在CXLCMEvents的第0位,`bbb`出现在数组的第1位,以此类推。需要注意这些寄存器允许出现的位置(Counters)在手册上是有限制的。

 

另外,这些个寄存器除了带宽之外没有什么用

0 Kudos
pudding_art
Novice
1,094 Views

谢谢!谢谢!太感谢了!毕设要用到这个看带宽,终于解决了!🥹

0 Kudos
Shunyu
Novice
1,092 Views

你要看带宽直接用pcm-memory就行了,不用这个

 

nudt也有cxl吗

0 Kudos
pudding_art
Novice
1,088 Views

因为想统计每个task的带宽,所以想找一下有没有相关的事件。pcm-memory是不是只能看整个流经CXL端口的流量。

 

嗯呢,nudt也有

0 Kudos
pudding_art
Novice
1,106 Views

pudding_art_0-1724145315665.png

而且在我使用perf stat去监控被分类在UNC_CHA下面的事件时,就会报错Bad event name.

0 Kudos
Shunyu
Novice
1,096 Views

perf的event请使用全大写。。

0 Kudos
pudding_art
Novice
1,075 Views

pudding_art_0-1724146357968.png

全大写看起来也不行

0 Kudos
Shunyu
Novice
1,074 Views

你用的是什么版本的kernel?

 

加vx吧,太麻烦了

0 Kudos
pudding_art
Novice
1,066 Views
0 Kudos
ManoranjanDas
Employee
1,678 Views

Hi Shunyu,


Greetings for the day!


Regarding your query, we recommend consulting your Linux team or IT support regarding the Compute Express Link (CXL) Performance Monitoring Unit (CPMU) and the Linux Kernel documentation.


Should you have any further questions or need assistance with other issues, please feel free to reach out.


Thank you for choosing Intel products and services.


Best regards,

Manoranjan.


0 Kudos
ManoranjanDas
Employee
1,595 Views

Hello Shunyu,

 

Greetings for the day!

 

I hope this message finds you well.

 

We are following up to find out if you were able to find the information we provided. Please reply to confirm, so we can continue helping on a resolution. Looking forward to receiving your reply.

 

Regards,

Manoranjan.


0 Kudos
Shunyu
Novice
1,561 Views
0 Kudos
ManoranjanDas
Employee
1,555 Views

Hello Shunyu,


Greetings of the day!


Apologies for the inconvenience caused.


Could you please confirm which Xeon manual you are referring to? If possible, could you also provide the link to the manual and specify the processor model?


Regards,

Manoranjan


0 Kudos
Reply