Server Products
Data Center Products including boards, integrated systems, Intel® Xeon® Processors, RAID Storage, and Intel® Xeon® Processors
4761 Discussions

sub-NUMA activation in R1208WFTYS is buggy

STurc
New Contributor I
3,876 Views

I have a 2-socket R1208WFTYS system with the latest BIOS (00.01.0014) and it seems that there is a bug/inconsistency in regards to sub-NUMA activation in BIOS.

If I set "sub_NUMA Cluster" to "Enabled" and leave "IMC Interleaving" to "Auto", which suggests that BIOS will use the most appropriate settings (which has to be "1-way interleave"), then sub-NUMA is not activated - I see only two NUMA nodes (one per socket), whereas if I set "IMC Interleaving" to "1-way" explicitly, everything works as expected - I see 4 numa nodes (two for each socket).

So, since "IMC Interleaving" setting is tighly coupled to "sub_NUMA Cluster", "IMC Interleaving" should be removed altogether or at least "Auto" mode of it must be coordinated with "sub-NUMA" settings.

0 Kudos
9 Replies
idata
Employee
2,000 Views

Hello Sergei,

Thank you so much for contacting us.

I was reviewing the information of this case and I would like to provide you the following details:

 

 

Sub NUMA Cluster (SNC) breaks up the last-level cache (LLC) into two disjoint clusters based on address range, with each cluster bound to one memory controller.

 

SNC improves average latency to the LLC/memory and is a replacement for the "Cluster On Die" (COD) feature found in previous processor families.

 

For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains. IMC Interleaving must be set to the correct value to correspond with SNC enable/disable. If SNC and IMC Interleave are both set to Auto, the result will be SNC disabled (only one cluster per socket) with 2-way IMC interleave. If SNC is set to Enable, IMC Interleave should be set to 1-way, which will result in two clusters per socket. The BIOS switch "Sub NUMA Clustering" allows 3 options: "auto", "enabled" and "disabled". The default setting is "enabled" (PRIMERGY servers), "auto" (PRIMEQUEST servers).

 

 

 

IMC Interleaving, this feature in BIOS option controls the interleaving between the Integrated Memory Controllers (IMCs). There are two IMCs per socket in Skylake Server. If IMC Interleaving is set to 2-way, addresses will be interleaved between the two IMCs. If IMC Interleaving is set to 1-way, there will be no interleaving. If SNC is disabled, IMC Interleaving should be set to 2-way. If SNC is enabled, IMC Interleaving should be set to 1-way. Default setting is "Auto".

 

 

 

 

On the other hand, I would like to share with you the properly information about the settings of each feature, in that way you will know the the specific role of configuration.

 

 

 

IMC (Integrated memory controller) Interleaving

 

This BIOS option controls the interleaving between the Integrated Memory Controllers (IMCs), Memory could be interleaved across sockets, memory controllers, DDR channels, Ranks. Memory is interleaved for performance and thermal distribution.
  • If IMC Interleaving is set to 2-way, addresses will be interleaved between the two IMCs.
  • If IMC Interleaving is set to 1-way, there will be no interleaving.
  • If IMC Interleaving is set to auto, it depends on the SNC (Sub NUMA Clustering) setting, when SNC is set to enbaled, the IMC Interleaving will be 1-way interleave, SNC is set to disabled, the IMC Interleaving will be 2-way interleave.
  • If SNC is disabled, IMC Interleaving should be set to 2-way. If SNC is enabled, IMC Interleaving should be set to 1-way.

 

Sub NUMA Cluster(SNC)

 

 

-SNC breaks up the last level cache (LLC) into disjoint clusters based on address range, with each cluster bound to a subset of the memory controllers in the system. SNC improves average latency to the LLC and memory. SNC is a replacement for the cluster on die (COD) feature found in previous processor families. For a multi-socketed system, all SNC clusters are mapped to unique NUMA (Non Uniform Memory Access) domains.
  • SNC AUTO supports 1-cluster or 2-clusters depending on IMC interleave. SNC and IMC interleave both AUTO will support 1-cluster 2-way IMC interleave.
  • SNC Enable supports Full SNC (2 clusters) and 1-way IMC interleave. Utilizes LLC capacity more efficiently and reduces latency due to core/IMC proximity. This may provide performance improvement on NUMA-aware operating systems.
  • SNC disable supports 1-cluster and 2-way IMC interleave, the LLC is treated as one cluster.

If you have any other question please do not hesitate and reply to this and I will be more than happy to assist you.

 

 

Best Regards,

Emeth X

0 Kudos
STurc
New Contributor I
2,000 Views

I quote from your reply:

"

If IMC Interleaving is set to auto, it depends on the SNC (Sub NUMA Clustering) setting, when SNC is set to enbaled, the IMC Interleaving will be 1-way interleave, SNC is set to disabled, the IMC Interleaving will be 2-way interleave.

"

The actual behaviour of my system contradicts what you are saying. As I have already said in my first message I have "sub_NUMA Cluster" = "Enabled" and "IMC Interleaving" = "Auto". With this configuration sub-NUMA is not activated (that is numactl --hardware shows only one NUMA domain per socket). If I explicitly set "IMC Interleaving" = "1-way", everything works as expected - I see two NUMA domains per socket, that is sub-NUMA is activated.

0 Kudos
idata
Employee
2,000 Views

Hello Sergei,

Thank you so much for the information provided.

I would like to analyze more information about the configuration of your server system.

 

Could you please so kind and provide us the Sysinfo logs of your system in order to make sure some details and provide you the most accurate information about it?

Best Regards,

Emeth X

0 Kudos
STurc
New Contributor I
2,000 Views

Here are SysInfo logs

0 Kudos
idata
Employee
2,000 Views

Hello,

Thank you so much for the information provided.

Let me analyze the information and as soon as I have an outcome of this I will let you know.

Best Regards,

Emeth X.

0 Kudos
idata
Employee
2,000 Views

Hello,

Please provide us more time, we are still investigating this information in order to provide you the best solution of this scenario.

 

 

Best regards,

Emeth X

0 Kudos
idata
Employee
2,000 Views

Hello,

As per an upper level engineer department, the recommendation is "1-way interleaving" to use "Sub-NUMA clustering feature" on this case.

Best regards,

Emeth X

0 Kudos
STurc
New Contributor I
2,000 Views

I do know that "1-way interleave" is a requirement for "Sub-NUMA clustering feature"!

You missed the point of everything I just said.

I said that when I set "sub_NUMA Cluster" to "Enabled" and leave "IMC Interleaving" to "Auto" I expected the BIOS to use "1-way interleave" for IMC interleaving automatically (that's why it is called "Auto", right?!). But sub-NUMA is not activated until I manually set "IMC Interleaving" to "1-way interleave".

As you previosly mentioned, "Auto" mode for "IMC Interleaving" will use "1-way" or "2-way" interleaving depending on the value of "sub_NUMA Cluster" option. In reality this doesn't happen.

So "Auto" mode of "IMC Interleaving" doesn't work as specified. It is a BUG.

0 Kudos
idata
Employee
2,000 Views

Hello,

Please accept my apologizes for this misunderstanding of everything.

We are going to analyze this issue and evaluate every detail in order to recreate and find a possible solution of this matter. Nonetheless I would like to highly appreciate your feedback provided and your time invested.

If you have any other question, please do not hesitate and let us know.

Best Regards,

Emeth X

0 Kudos
Reply