How to obtain bandwidth information of CXL memory on Intel Agilex 7 FPGA I-Series Development Kit

[Iriscsa]

How to obtain bandwidth information of CXL memory ?
hardware and system:
host: Intel Xeon Gold 6438Y+ Processor
system: ubuntu23.10, linux kernal 6.5.0
CXL device: Intel Agilex 7 FPGA I-Series Development Kit
CXL devive follow CXL 1.1 work in Type 3 mode for memory expand
node 4 cpus:
node 4 size: 16123 MB
node 4 free: 16051 MB
my problem:
1. Does my host and CXL device currently follow CXL 1.1 have the ability to monitor CXL memory access bandwidth?
1.1 In https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html , it say 4th Gen Intel® Xeon® Scalable Processors support CXL 1.1,
but on lspci, my CXL device show "CXL: Intel Corporation Device 0ddb (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])"
I am very confused whether my device is CXL 1.1 or CXL2.0 ?
2. If have, how should I monitor it?
My attempt, but it didn't work：
1. I can see 16GB CXL memory work as numa node4, and I can see /dev/dax0.0 work in "system-ram" mode.
But i can not see /dev/cxl/mem0 device, and demsg report : "cxl_pci 0000:bd:00.0: Device DVSEC not present, skip CXL.mem init"
So I turn for CXL driver code, follow https://lore.kernel.org/lkml/ZaGVm5tHXoTZd4Ad@debian/T/ modified my cxl driver from linux-6.5.0, then I get /dev/cxl/mem0 at address 0000:bd:00.0.
However, dmesg reports an error :CXL port topology not found, and /dev/cxl/mem0 is in a disabled state and cannot be activated.
2. I found some CXL related counters in perf:
CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA, CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA from https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426 UNC_CHA_TOR_INSERTS.IA_MISS_RFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFRFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_PREF_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFDATA_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFCODE_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_RFO_PREF_CXL_ACC from https://github.com/intel/pcm/blob/master/src/pcm-memory.cpp#L1257
but when i monitor them, i only get 0 value, such as
UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC: pcm-raw -e cha/config=0x10c8178200000135,name=UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC
or
CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA : pcm-raw -e cxlcm/config=4161
I see that in the pcm-memory code, it first check if the CXL unit exists before monitor the CXL memory bandwidth. Is it because I do not have an active /dev/cxl/mem0 device that causes the results of monitoring the above counters to be 0, or there is an error when i use pcm-raw?
Looking forward to your help.

[WZ2]

Hi there,

1. Does my host and CXL device currently follow CXL 1.1 have the ability to monitor CXL memory access bandwidth?

[Altera] you can try the “MLC” tool from intel, please refer to the Design example guide

1.1 In https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html , it say 4th Gen Intel® Xeon® Scalable Processors support CXL 1.1,

but on lspci, my CXL device show "CXL: Intel Corporation Device 0ddb (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])"

I am very confused whether my device is CXL 1.1 or CXL2.0 ?

[Altera] For both CXL1.1 and CXL2.0 , the link speed is 32GT/s, but there is new added filed in the Training Sequence(TS) to indicate the CXL2.0 link capability (R-Tile doesn’t support this filed as it the silicon limitation) thought the finial link speed is still 32GT/s , but from the Transaction layer perspective, R-Tile supports most of the CXL 2.0 feature, that is the reason we claim R-Tile as the CXL1.1 link but CXL2.0 device, regarding the "CXL: Intel Corporation Device 0ddb (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])" during the kernel info display, this depends on the Kernel behavior, so it might discover CXL2.0 feature on the R-Tile (32GT/s with CXL1.1 link)

2. If have, how should I monitor it?

My attempt, but it didn't work：

1. I can see 16GB CXL memory work as numa node4, and I can see /dev/dax0.0 work in "system-ram" mode.

But i can not see /dev/cxl/mem0 device, and demsg report : "cxl_pci 0000:bd:00.0: Device DVSEC not present, skip CXL.mem init"

So I turn for CXL driver code, follow https://lore.kernel.org/lkml/ZaGVm5tHXoTZd4Ad@debian/T/ modified my cxl driver from linux-6.5.0, then I get /dev/cxl/mem0 at address 0000:bd:00.0.

However, dmesg reports an error :CXL port topology not found, and /dev/cxl/mem0 is in a disabled state and cannot be activated.

[Altera] this also depends on the Kernel version , because there is some update from the Kernel to identify the CXL device, regarding the “/dev/cxl/mem0” , might this is the new feature of the Kernel OS, in our 5.16 Kernel version we only monitor the “/dev/dax0.0 work in "system-ram" mode” , we don’t have the platform to support up to 6.5 kernel , and actually you can access “/dev/dax0.0 as "system-ram" mode, regarding the CXL driver you mentioned, not sure if this is the mature driver since we never use it

1. 
   
2. 2. I found some CXL related counters in perf:
3. CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA, CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA from https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426 UNC_CHA_TOR_INSERTS.IA_MISS_RFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFRFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_PREF_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFDATA_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFCODE_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_RFO_PREF_CXL_ACC from https://github.com/intel/pcm/blob/master/src/pcm-memory.cpp#L1257
4. but when i monitor them, i only get 0 value, such as
5. UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC: pcm-raw -e cha/config=0x10c8178200000135,name=UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC
6. or
7. CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA : pcm-raw -e cxlcm/config=4161
8. I see that in the pcm-memory code, it first check if the CXL unit exists before monitor the CXL memory bandwidth. Is it because I do not have an active /dev/cxl/mem0 device that causes the results of monitoring the above counters to be 0, or there is an error when i use pcm-raw?
9. Looking forward to your help.

4th Gen Intel Xeon Processor Scalable Family, sapphire rapids

A technical overview of the 4th Gen Intel® Xeon® Processor Scalable Family based on the formerly codenamed Sapphire Rapids architecture.

[Altera] we don’t know what is the code you mentioned “https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426” , might this is for some specific usage, and seems it for the CPU only because CHA is mentioned, but no such Agent in the FPGA

So , most of the questions are about whatever Platform/Kernel/CPU related code, and could you share us your final purpose regarding FPGA CXL solution ? might we can share something we once verified from the FPGA CXL example design perspective, thanks

Best regards,

WZ

[WZ2]

Hi there,

If there's no further questions, would you mind me set this case to close pending first? The close-pending status will last ~20 days before completely closed, so let me know if further assistance is needed, I'll re-open it, thank you for your understanding.

Have a nice day!

Best regards,

WZ