How to find if iommu memory range is cached on a Xeon system with DDIO

[jflarin]

Hi,

I have memory regions begin allocated by a Linux driver on a system using Xeon Silver 4xxx processor.  The system uses IOMMU and the address given to the PCIe device is under 4GB, starting at 4GB and going down.  I understand that those "virtual PCIe addresses" correspond to some physical pages which could be anywhere in the physical ram.

I am trying to find if those regions are in fact in uncachable or Writeback memory.  My test tend to show that those are in uncachable memory, but the fact that this processor supports DDIO could be another explanation.

I mention DDIO because when the PCIe device uses the No-Snoop bit when writing in host memory, it tells the system to skip the cache snooping and push data directly in physical memory. If that memory region was in fact cached, then the processor doesn't see the data written in host memory. That is the observed behavior on non-Xeon system.

However, on Xeon Processor with DDIO, the No-Snoop bit is reused to indicate allocating write  (see https://www.dpdk.org/wp-content/uploads/sites/35/2018/09/Roman-Sudarikov-DPDK_PRC_Summit_Sudarikov.pptx for details).  So when  the PCIe device  is writing in cachable region with no-snoop bit, the data can go directly in cache which is seen by the processor as it should. If the PCIe is writing in a non-cachable region with no-snoop bit, data will be written in host memory and the processor will see the data, as it should.

I tried to get the cache attribute under Linux with the following command:

sudo cat /sys/kernel/debug/x86/pat_memtype_list

However the result are not conclusive, as I suspect the memory address given to the PCIe device cannot be linked to the attributes in the par_memttype_list.

Is there a way to find if that memory range is UC or WB?

Thank your for any help or pointers,

jflarin