Server Products
Data Center Products including boards, integrated systems, Intel® Xeon® Processors, RAID Storage, and Intel® Xeon® Processors
4851 Discussions

Where is the directory placed? In CPU or in memory

huangwentao
New Contributor I
847 Views

Hi,

 

I have a two-socket machine with Intel Xeon Sapphire Rapids CPUs, and I am doing some cache-sensitive workload analysis across NUMA. I have read some documents regarding the cache-coherence protocol, some says the directory for the cache-coherence is placed across the LLC slices or integrated memory controllers, while others state that the directory is actually stored in memory. Can anyone tell me where exactly is the directory placed for Sapphire Rapids architecture? 

 

Additionally, I would like to measure the overhead for maintaining cache coherence in my platform, especially over NUMA. If anyone can help share some valuable ideas of how to measure this coherence overhead, I would be very much appreciated.

 

Thanks in advance.

Labels (1)
0 Kudos
13 Replies
NormanS_Intel
Moderator
795 Views

Hello huangwentao,

 

Thank you for posting in the community!

 

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

 

Best regards,

Norman S.

Intel Customer Support Engineer


0 Kudos
huangwentao
New Contributor I
790 Views

Hi Norman,

 

Thank you, please go ahead moving this to server forum.

 

Best regards,

Wentao

0 Kudos
Fikri_Intel
Employee
747 Views

Hi huangwentao,


Thank you for reaching Intel Community.


Kindly find the link below for Technical Overview Of The 4th Gen Intel® Xeon® Scalable processor family.


https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html



Regards,

Fikri O.


0 Kudos
huangwentao
New Contributor I
744 Views

Hi Fikri,

 

Thank you for showing me this.

 

But is there any specific description regarding directory placement? I found limited information in this link.

 

Regards,

Wentao 

0 Kudos
RyanFeeko
New Contributor II
738 Views

Hey @huangwentao , just wondering the directory you mention is it the data cache or instruction?

 

i found in the link provided where is shows data cache in CPU.

 

RyanFeeko_0-1717650764362.png

 

or there is something else you mean.

0 Kudos
huangwentao
New Contributor I
736 Views

Hi RyanFeeko,

 

Thanks for pointing out this.

 

However, I am actually looking for the information of the directory, which regarding the directory-based cache coherence protocol of the Xeon processors. Specifically, I would like to know the very exact location that this so-called directory structure placed on.

 

Thank you.

 

Wentao

0 Kudos
Fikri_Intel
Employee
631 Views

Hi huangwentao,


Thank you for your response.


Kindly allow us some time to check and get back to you once the info is available.



Regards,

Fikri O.


0 Kudos
Subhashish
Employee
604 Views

Hello huangwentao,

Greetings!


Thank you for writing to us. After reviewing your query, we would like to share with you this manual. Please see this document:-


https://www.intel.com/content/www/us/en/content-details/819723/intel-64-and-ia-32-architectures-software-developer-s-manual-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html


The processor’s cache coherency protocols ensure that other processors that are caching the same memory locations are managed properly while atomic operations are performed on cached memory locations. (Page 3282) so is stored at the memory level.


A-32 Architecture Compatibility:


Beginning with the P6 family processors, when the LOCK prefix is prefixed to an instruction and the memory area being accessed is cached internally in the processor, the LOCK# signal is generally not asserted. Instead, only the processor’s cache is locked. Here, the processor’s cache coherency mechanism ensures that the operation is carried out atomically with regards to memory. See “Effects of a Locked Operation on Internal Processor Caches” in Chapter 9 of Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, for more information on locking of caches. Page 1207


Regarding the measurement of coherence overhead, we do not have any suggestion about tools, as this measurement would involve the calculation of bits necessary for the cache, divided by the cache size, something in this direction.


We hope that this information is helpful to you. If you have any other query/doubts, please feel free to write to us. Thank you for choosing Intel.



Regards,

Subhashish_Intel.


0 Kudos
huangwentao
New Contributor I
568 Views

Hi  Subhashish,

 

Thank you for pointing out this description.

 

I would like to know if there are some Intel server-level processors that keep the directory within CPUs. As I just came across a paper, 

SecDir: A Secure Directory to Defeat Directory Side-Channel Attacks, which states that,

"Typically, the directory is partitioned and physically distributed into as many slices as cores. Each directory slice can store directory entries for a fixed set of physical addresses. A proprietary hash function maps each requested address to its corresponding slice. In an inclusive cache hierarchy, each directory slice is closely associated with a Last-Level Cache (LLC) slice." (section 2.1 Directories and Non-Inclusive Caches).

I am not sure if the "slices" mentioned above are referred to as the LLC slices. if so, which Intel CPU family maintains the directory within the LLC (perhaps the CPUs with an inclusive LLC )? 

 

Thank you.

 

Regards,

Wentao

0 Kudos
Sazirah
Employee
550 Views

Hi huangwentao,


Thank you for your reply.


We are currently checking on this matter at our end. Please give us some time and we will revert back to you.


Regards,

Sazzy_Intel



0 Kudos
Subhashish
Employee
465 Views

Hi huangwentao,

Greetings!


Intel doesn't provide much documentation for memory directories. This issue should be addressed by accessing the Software Developer Forums. Specifically, I recommend you to check the following forum: https://community.intel.com/t5/Software-Tuning-Performance/Directory-Structure-in-Skylake-Server-CPUs/m-p/1185376.


I also suggest you to check the external link: https://sites.utexas.edu/jdm4372/2023/08/28/memory-directories-in-intel-processors/..


Hope that this information helps. Please let us know if there is any other issue/concerns.



Regards,

Subhashish_Intel.




0 Kudos
Fikri_Intel
Employee
348 Views

Hi huangwentao,


Good day to you.


Just wanted to follow up with you, kindly let us know if you have further concerns.


If no, we will proceed to close this thread.



Regards,

Fikri O.


0 Kudos
Fikri_Intel
Employee
284 Views

Hi huangwentao,


Hope you are doing well. 


As we have not heard a response in the past few days, we will proceed in closing this thread as of now.


If you need any additional information, please submit a new question as this thread will no longer be monitored.



Regards,

Fikri O.


0 Kudos
Reply