Where is the directory placed? In CPU or in memory

[huangwentao]

Hi,

I have a two-socket machine with Intel Xeon Sapphire Rapids CPUs, and I am doing some cache-sensitive workload analysis across NUMA. I have read some documents regarding the cache-coherence protocol, some says the directory for the cache-coherence is placed across the LLC slices or integrated memory controllers, while others state that the directory is actually stored in memory. Can anyone tell me where exactly is the directory placed for Sapphire Rapids architecture? 

Additionally, I would like to measure the overhead for maintaining cache coherence in my platform, especially over NUMA. If anyone can help share some valuable ideas of how to measure this coherence overhead, I would be very much appreciated.

Thanks in advance.

[NormanS_Intel]

Hello huangwentao,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer

[huangwentao]

Hi Norman,

Thank you, please go ahead moving this to server forum.

Best regards,

Wentao

[Fikri_Intel]

Hi huangwentao,

Thank you for reaching Intel Community.

Kindly find the link below for Technical Overview Of The 4th Gen Intel® Xeon® Scalable processor family.

https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html

Regards,

Fikri O.

[huangwentao]

Hi Fikri,

Thank you for showing me this.

But is there any specific description regarding directory placement? I found limited information in this link.

Regards,

Wentao 

[RyanFeeko]

Hey @huangwentao , just wondering the directory you mention is it the data cache or instruction?

i found in the link provided where is shows data cache in CPU.

or there is something else you mean.

[huangwentao]

Hi RyanFeeko,

Thanks for pointing out this.

However, I am actually looking for the information of the directory, which regarding the directory-based cache coherence protocol of the Xeon processors. Specifically, I would like to know the very exact location that this so-called directory structure placed on.

Thank you.

Wentao

[Fikri_Intel]

Hi huangwentao,

Thank you for your response.

Kindly allow us some time to check and get back to you once the info is available.

Regards,

Fikri O.

[Subhashish]

Hello huangwentao,

Greetings!

Thank you for writing to us. After reviewing your query, we would like to share with you this manual. Please see this document:-

https://www.intel.com/content/www/us/en/content-details/819723/intel-64-and-ia-32-architectures-software-developer-s-manual-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html

The processor’s cache coherency protocols ensure that other processors that are caching the same memory locations are managed properly while atomic operations are performed on cached memory locations. (Page 3282) so is stored at the memory level.

A-32 Architecture Compatibility:

Beginning with the P6 family processors, when the LOCK prefix is prefixed to an instruction and the memory area being accessed is cached internally in the processor, the LOCK# signal is generally not asserted. Instead, only the processor’s cache is locked. Here, the processor’s cache coherency mechanism ensures that the operation is carried out atomically with regards to memory. See “Effects of a Locked Operation on Internal Processor Caches” in Chapter 9 of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, for more information on locking of caches. Page 1207

Regarding the measurement of coherence overhead, we do not have any suggestion about tools, as this measurement would involve the calculation of bits necessary for the cache, divided by the cache size, something in this direction.

We hope that this information is helpful to you. If you have any other query/doubts, please feel free to write to us. Thank you for choosing Intel.

Regards,

Subhashish_Intel.

[huangwentao]

Hi  Subhashish,

Thank you for pointing out this description.

I would like to know if there are some Intel server-level processors that keep the directory within CPUs. As I just came across a paper, 

SecDir: A Secure Directory to Defeat Directory Side-Channel Attacks, which states that,

"Typically, the directory is partitioned and physically distributed into as many slices as cores. Each directory slice can store directory entries for a fixed set of physical addresses. A proprietary hash function maps each requested address to its corresponding slice. In an inclusive cache hierarchy, each directory slice is closely associated with a Last-Level Cache (LLC) slice." (section 2.1 Directories and Non-Inclusive Caches).

I am not sure if the "slices" mentioned above are referred to as the LLC slices. if so, which Intel CPU family maintains the directory within the LLC (perhaps the CPUs with an inclusive LLC )? 

Thank you.

Regards,

Wentao

[Sazirah]

Hi huangwentao,

Thank you for your reply.

We are currently checking on this matter at our end. Please give us some time and we will revert back to you.

Regards,

Sazzy_Intel

[Subhashish]

Hi huangwentao,

Greetings!

Intel doesn't provide much documentation for memory directories. This issue should be addressed by accessing the Software Developer Forums. Specifically, I recommend you to check the following forum: https://community.intel.com/t5/Software-Tuning-Performance/Directory-Structure-in-Skylake-Server-CPUs/m-p/1185376.

I also suggest you to check the external link: https://sites.utexas.edu/jdm4372/2023/08/28/memory-directories-in-intel-processors/..

Hope that this information helps. Please let us know if there is any other issue/concerns.

Regards,

Subhashish_Intel.

[Fikri_Intel]

Hi huangwentao,

Good day to you.

Just wanted to follow up with you, kindly let us know if you have further concerns.

If no, we will proceed to close this thread.

Regards,

Fikri O.

[Fikri_Intel]

Hi huangwentao,

Hope you are doing well. 

As we have not heard a response in the past few days, we will proceed in closing this thread as of now.

If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards,

Fikri O.