Processors
Intel® Processors, Tools, and Utilities
15863 Discussions

How to obtain bandwidth information of CXL memory

Iriscsa
Beginner
151 Views

hardware and system:

host: Intel Xeon Gold 6438Y+ Processor

system: ubuntu23.10, linux kernal 6.5.0

CXL device: Intel Agilex 7 FPGA I-Series Development Kit

CXL devive follow CXL 1.1 work in Type 3 mode for memory expand

node 4 cpus:                

node 4 size: 16123 MB

node 4 free: 16051 MB

my problem:

1. Does my host and CXL device currently follow CXL 1.1 have the ability to monitor CXL memory access bandwidth?

2. If have, how should I monitor it?

My attempt, but it didn't work:

1.  I can see 16GB CXL memory work as numa node4, and I can see /dev/dax0.0 work in "system-ram" mode. But i can not see /dev/cxl/mem0 device, and demsg report :

"cxl_pci 0000:bd:00.0: Device DVSEC not present, skip CXL.mem init"

So I turn for CXL driver code, follow https://lore.kernel.org/lkml/ZaGVm5tHXoTZd4Ad@debian/T/ modified my cxl driver from linux-6.5.0, then I get /dev/cxl/mem0 at address 0000:bd:00.0. However, dmesg reports an error :CXL port topology not found, and /dev/cxl/mem0 is in a disabled state and cannot be activated.

2. I found some CXL related counters in perf:

CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA, CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA from https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426 

UNC_CHA_TOR_INSERTS.IA_MISS_RFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFRFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_PREF_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFDATA_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFCODE_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_RFO_PREF_CXL_ACC from https://github.com/intel/pcm/blob/master/src/pcm-memory.cpp#L1257 

but when i monitor them, i only get 0 value, such as 

UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC: pcm-raw -e cha/config=0x10c8178200000135,name=UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC

or

CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA : pcm-raw -e cxlcm/config=4161

I see that in the pcm-memory code, it first check if the CXL unit exists before  monitor the CXL memory bandwidth. Is it because I do not have an active /dev/cxl/mem0 device that causes the results of monitoring the above counters to be 0, or there is an error when i use pcm-raw?

Looking forward to your help.

 

0 Kudos
0 Replies
Reply