How to obtain bandwidth information of CXL memory

[Iriscsa]

hardware and system:

host: Intel Xeon Gold 6438Y+ Processor

system: ubuntu23.10, linux kernal 6.5.0

CXL device: Intel Agilex 7 FPGA I-Series Development Kit

CXL devive follow CXL 1.1 work in Type 3 mode for memory expand

node 4 cpus:                

node 4 size: 16123 MB

node 4 free: 16051 MB

my problem:

1. Does my host and CXL device currently follow CXL 1.1 have the ability to monitor CXL memory access bandwidth?

2. If have, how should I monitor it?

My attempt, but it didn't work：

1.  I can see 16GB CXL memory work as numa node4, and I can see /dev/dax0.0 work in "system-ram" mode. But i can not see /dev/cxl/mem0 device, and demsg report :

"cxl_pci 0000:bd:00.0: Device DVSEC not present, skip CXL.mem init"

So I turn for CXL driver code, follow https://lore.kernel.org/lkml/ZaGVm5tHXoTZd4Ad@debian/T/ modified my cxl driver from linux-6.5.0, then I get /dev/cxl/mem0 at address 0000:bd:00.0. However, dmesg reports an error :CXL port topology not found, and /dev/cxl/mem0 is in a disabled state and cannot be activated.

2. I found some CXL related counters in perf:

CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA, CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA from https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426 

UNC_CHA_TOR_INSERTS.IA_MISS_RFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFRFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_PREF_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFDATA_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFCODE_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_RFO_PREF_CXL_ACC from https://github.com/intel/pcm/blob/master/src/pcm-memory.cpp#L1257 

but when i monitor them, i only get 0 value, such as 

UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC: pcm-raw -e cha/config=0x10c8178200000135,name=UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC

or

CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA : pcm-raw -e cxlcm/config=4161

I see that in the pcm-memory code, it first check if the CXL unit exists before  monitor the CXL memory bandwidth. Is it because I do not have an active /dev/cxl/mem0 device that causes the results of monitoring the above counters to be 0, or there is an error when i use pcm-raw?

Looking forward to your help.

[NormanS_Intel]

Hello Iriscsa,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer