Xeon CPU how to bypass LLC during IOAT

[nskim]

I am trying to use IOAT but during the memory copy from a source address region to a destination address region, I don't want the data from the source address region to be cached at LLC to prevent cache pollution. Is there any knob or option that prevents the data transferred by IOAT from being cached at LLC in Xeon Scalable processors?

[JoseH_Intel]

Hello nskim,

Thank you for joining the Intel community

According with the general description of the functioning of Intel® I/O Acceleration Technology, the data is placed directly into the CPU cache (doesn't specify if LLC or MLC). The fact that it does it directly to LLC is hardcoded by design. Let me check if there is any possible way to change this on BIOS. I will update you soon.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

What operating system are you using?

Which IOAT driver are you using, e.g. (Linux kernel, DPDK, or SPDK)??

Which Intel(R) Xeon(R) processors are you asking about?

DPDK is Data Plane Development Kit (Home - DPDK).

SPDK is Storage Performance Development Kit (spdk.io , Introduction to the Storage Performance Development Kit (SPDK) (intel.com).

We will look forward to your updates

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[nskim]

Hello --

Xeon 2nd Gen Scalable Processor (a.k.a. Skylake) and Linux kernel (but open to know how to do it with user space IO stack such as DPDK and SPDK)

[JoseH_Intel]

Hello nskim,

Thank you for the updates. Let me research further on this. I will let you know soon.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

Our Intel Engineering team suggests to DMA from memory region that is allocated uncached by the kernel like through dma_alloc_coherent(). Typically customers are concerned with cache pollution with the destination. Unless using uncached memory, source data would always be cached because the processor has to access it to put data there. DMA could only cause cache pollution to the destination. By default, it does not push into cache when operating.

If you have further questions just let us know.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

I am just following up to double-check if you found the provided information useful. If you have further questions please don't hesitate to ask. If you consider the issue to be completed please let us know so we can proceed to mark this ticket as closed. I will try to reach you back on next Monday 3rd. After that the thread will be automatically archived. 

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[nskim]

Thank you for your follow up.

"Our Intel Engineering team suggests to DMA from memory region that is allocated uncached by the kernel like through dma_alloc_coherent()."

In my case, I am trying to do mem copy from (cacheable) source regions that were already allocated by other applications.

"Typically customers are concerned with cache pollution with the destination."

In my case, I make the destination region uncacheable so it is not an issue but more concerned about brining source data polluting LLC. That said, reading "Unless using uncached memory, source data would always be cached because the processor has to access it to put data there." and "DMA could only cause cache pollution to the destination. By default, it does not push into cache when operating." I am confused because these two sentences sound contradicting. What does "it" point to in "By default, it does not push into cache when operating."?

Although the source region is cacheable, there are two possibilities assuming some of data in the source region are already in LLC. To explain the two possibilities, let me give you an example based on traditional DMA between IO devices and CPU/DRAM; it's a bit different case but  fundamentally it's the same. For DMA from a MMIO IO device to main memory, depending on whether DDIO is enabled or not, LLC is updated differently.

(1) DDIO disabled -- only the data that were already in the LLC are updated based on coherence mechanism, but no additional cache lines are not allocated for the data  transferred from the IO device. In other words, all the data from the IO device are directly sent to the DRAM memory w/o polluting LLC (except for the data that were already in LLC).

(2) DDIO enabled -- all the data from the IO device are allocated to the LLC (if some were already in the LLC, they are updated).

I believe IOAT DMA should work similarly, as it is fundamentally designed the same way as DMA in IO devices. So, my question is how I can get the same effect as (1) above during IOAT memory copy. I hope my original question is clearer now.

Also, there are research papers saying that mem copy using IOAT can prevent cache pollution compared to traditional way of doing mem copy with load/store instructions although these papers didn't say how: 

"Designing Efficient Asynchronous Memory Operations Using Hardware Copy Engine: A Case Study with I/OAT"

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4228207

and

"Efficient asynchronous memory copy operations on multi-core systems and I/OAT"

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4629228

And interestingly I found out that the first author of these two papers (Karthikeyan Vaidyanathan) is currently at Intel. Can you check with him?

Thank you.

[JoseH_Intel]

Hello nskim,

Thank you very much for the comprehensive explanation. Let me pass this over to our engineering team and I will let you know as soon as we have updates.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

We want tp inform you that our engineering team is reviewing the information provided.

And we still need clarification on this:

Could you please tell the exact model number of the Xeon processor you are using,? (we need to confirm the processor generation)

Also, which Linux distribution and kernel version are you using?

We will look forward to your updates.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[nskim]

Hello -- 

We tried two different machines -- 

Intel Xeon 6138P, Ubuntu 18.04.6 LTS with kernel 5.9.9

Xeon E5-2640 v3, Ubuntu 18.04.6 LTS with kernel 5.9.9

And we are willing to use any recent Xeon CPUs and Linux version as long as the combination works for what I asked.

[JoseH_Intel]

Hello nskim,

Thank you for the updates. Let me pass this over to our engineering team. We will get back to you soon.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

We have the following update:

The processor's DMA engine works by default when DDIO is disabled.

By any chance are you not seeing this to be the case?

We will look forward to your updates.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

Do you have any updates, questions or comments in regards to this issue? Please do not hesitate to contact us back. I will try to reach you on next Thursday 20th. After that the thread will be archived automatically.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello nskim,

We will proceed to mark this thread as closed. If you have further issues or questions just go ahead and submit a new topic.

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios