tuning 0xc8b register by wrmsr command

[sc3289]

Hi.

I am currently working with an Intel Xeon Gold 6354 processor.

(ice lake)

I'm attempting to increase the DDIO portion of the L3 cache through MSR settings.

Specifically, I have been using the wrmsr 0xc8b command, aiming to allocate a larger portion of the LLC to DDIO.

ex) wrmsr 0xc8b 0xF00

ex) wrmsr 0xc8b 0xFC0

However, upon testing with iperf3 to generate network flows and measuring L3 cache miss rates via perf, it seems that changing the MSR settings with wrmsr 0xc8b is not increasing the IIO_LLC_WAYS allocation as expected.

Could you provide guidance on whether adjusting 0xc8b for this model is indeed effective for increasing DDIO allocation within the LLC?

If there is an alternative method or a specific MSR configuration applicable to the Intel Xeon Gold 6354, I would appreciate any information or insights on how to properly increase the DDIO portion of the LLC cache.

Thank you very much for your assistance.

[NormanS_Intel]

Hello sc3289,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer

[pujeeth]

Hello sc3289,

Greetings!

Kindly let us know if the processor came with the Intel L9 system or if you purchased it separately. If yes, please share the server details and serial number.

Regards,

Pujeeth

Intel Customer Support Technician

intel.com/VROC

[sc3289]

I did not purchase the server from Intel.
I bought a Dell PowerEdge R750.

The CPU used in this server is an Intel Xeon Gold 6354 @ 3.0GHz model, so I posted a question regarding DDIO settings.

If there’s any additional information needed to assist me, let me know, and I’ll provide it.

Thank you in advance for your assistance.

[Devi_Intel]

Hi sc3289,

Greetings ~

Thanks for the update, we understand your inquiry to increase the DDIO portion of the L3 cache through MSR settings.

Please refer to the following link for addtional information:

https://www.intel.com/content/www/us/en/content-details/831418/intel-virtualization-technology-for-directed-i-o-architecture-specification.html

Also, for supported Processor Software Instructions – Refer to Intel Software Developer Manual (public) at https://software.intel.com/en-us/articles/intel-sdm

Additionally, we recommend you to check our Intel Developer Zone if you still have further inquiry. https://www.intel.com/content/www/in/en/developer/overview.html

Thank you & Best Regards

Devi

[sc3289]

I am currently not utilizing IOMMU, so this issue appears unrelated to VT-d. Additionally, I have reviewed documentation related to MSR registers, but I have not been able to find the information I need.

In previous research on the Skylake model, it was possible to modify the DDIO portion using the wrmsr command. However, on the Ice Lake model I am currently using, it seems that changes made to the DDIO portion via wrmsr do not take effect.

Is there a manual or documentation that details the cache hierarchy of Ice Lake, or an alternative method to modify the DDIO portion?

Thank you.

[Devi_Intel]

Hi sc3289,

Greetings ~

As suggested earlier, you may check supported Processor Software Instructions – Refer to Intel Software Developer Manual (public) at https://software.intel.com/en-us/articles/intel-sdm

Besides, we recommend you to check our Intel Developer Zone if you still have further inquiry. https://www.intel.com/content/www/in/en/developer/overview.html

Thank you & Best Regards

Devi

[sc3289]

I've already seen these.

But I couldn't find a way that I find.

Could you give me some specific or idea about this problem?

I'm finding a way to change IIO_LLC_WAYS for DDIO in ICE-lake or if it is possible or not?

[Meghak]

Hi sc3289,

Greetings for the day!

Please note we understand your query, however the requested details can be found on Intel Developer Zone.

In order to access the Developer zone, we request to please follow the steps mentioned on the below link:

https://www.intel.com/content/www/us/en/support/articles/000058073/programs/resource-and-documentation-center.html

Please let us know if the above information was helpful.

Regards,

Megha K

[sc3289]

Dear Megha K,

Greetings!

Thank you for your prompt response and for providing the relevant information. I will review the details on the Intel Developer Zone as suggested.

Should I need further assistance or have additional queries, I will not hesitate to reach out.

I appreciate your support and guidance.

Best regards,
sc3289.

[sc3289]

Dear Megha K,

Greetings, and thank you for your response and the provided guidance.

Following your suggestion, I visited the Intel Developer Zone and attempted to locate the details regarding DDIO portion configurations and adjustments, as well as information about the L3 cache hierarchy in Ice Lake processors. Unfortunately, I was unable to find the specific information I was seeking.

Could you kindly clarify what type of information or documents I should expect to find through the recommended path? I would like to ensure that I have not overlooked anything relevant. Your additional guidance would be greatly appreciated.

Thank you in advance for your support.

Best regards,
sc3289

[Devi_Intel]

HI

Greetings ~

Thank you for reaching us. Could you please specify the exact configurations or adjustments you are looking into? For example, are you looking for information on adjusting on any specific setting, optimizing DDIO performance, or any other specific aspect.

Which Intel Processors Support Intel® Data Direct I/O Technology...

Thank you & Best regards

Devi

[sc3289]

Hi

As mentioned earlier, I am currently using a DELL server equipped with an Intel Xeon Gold 6354 processor. I have been conducting research on Intel DDIO technology, with a particular interest in understanding how to increase DDIO capacity.

During my search, I discovered that some studies in the Intel Community involved modifying the MSR register values to increase or decrease the DDIO portion. For example:

• wrmsr 0xc8b 0xF00
• wrmsr 0xc8b 0xFC0

It seems that this method was applicable only up to the Skylake generation, but I have not been able to find any information about subsequent generations. Additionally, when I attempted to use this method in my environment, the performance did not reflect an increase in LLC_WAY, as might have been expected.

I would like to know whether Ice Lake has undergone changes in terms of cache hierarchy compared to previous Scalable generations, and whether it is possible to adjust the DDIO portion in Ice Lake.

Best regards,
sc3289

[IntelSupport]

Hi sc3289,

Thank you for providing the details. I’ll review the information and get back to you with an update at the earliest.

If you have any further questions in the meantime, feel free to reach out.

Regards,

Akshaya 

Intel Customer Support Technician

Intel.com/vroc  

[Sazirah]

Hi sc3289,

Greetings.

Regarding this case, we would like to verify if you was able to read the register with the rdmsr instruction? If so, what value did it provide and did this change after trying to modify the MSR?

We look forward for your response.

Regards,

Sazzy_Intel

[sc3289]

Hi Sazzy_Intel,

Greetings, and thank you for your follow-up.

Regarding your query, I performed the following steps to read and modify the 0xC8B register using rdmsr and wrmsr commands:

>> rdmsr 0xc8b
c00
>> wrmsr 0xc8b 0xF00
>> rdmsr 0xc8b
f00
>> wrmsr 0xc8b 0xC00
>> rdmsr 0xc8b
c00

I observed that the value of the register did change as expected after using wrmsr. However, during the execution of the wrmsr command, I noticed the following logs were printed in the kernel log:

>> msr: Write to unrecognized MSR 0xc8b by wrmsr (pid: 15245).

>> msr: See https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/about for details.

Despite these logs, I was able to modify the 0xC8B register value using the wrmsr command and attempted to adjust the DDIO portion through this approach.

I hope this provides the details you were looking for. Please let me know if further clarification or information is required.

Best regards,
sc3289

[IntelSupport]

Hi sc3289,

Thank you for providing the details. We will review the information you shared and get back to you with an update as soon as possible.

Regards,

Akshaya 

Intel Customer Support Technician

Intel.com/vroc  

[MACM]

Hi sc3289,

Greetings of the day.

Hope you are doing great.

This change is related to the OS Linux and we only can do our best effort. For detailed information about errors on the kernel, customers need to check with the OS vendor.

To increase the DDIO (Data Direct I/O) portion of the L3 cache on an Ice Lake processor, you will need to adjust the configuration of cache allocation.

However, it is important to note that Intel DDIO is typically configured to use a subset of the L3 cache, and the degree of control over this allocation can be limited by the processor's firmware and BIOS settings.

Here are the steps you can follow to adjust the DDIO portion of the L3 cache:

1 - Ensure DDIO is Enabled: Make sure that DDIO is enabled in the BIOS settings. This feature should be enabled by default, but it's good to verify.

2 - Understanding Cache Allocation Technology (CAT): Intel's CAT can be used to allocate portions of the L3 cache to specific uses, including DDIO. CAT allows you to define Class of Service (CoS) and assign cache ways to these classes.

3 - Configure CAT for DDIO: You can use the Intel Resource Director Technology (RDT) tools to configure CAT. The tool pqos from the Intel RDT library allows you to adjust the cache allocation.

• Install RDT Tools: If you don't have the Intel RDT tools installed, you can download and install them from Intel's website or your system's package manager.
• Determine Current Configuration: Use the pqos tool to check the current cache allocation:
  • pqos -s
• Adjust Cache Allocation: To allocate more cache ways to the DDIO (assuming DDIO uses a specific CoS), you need to adjust the allocation using pqos.
• For example, if CoS 0 is used for DDIO and you want to allocate more cache ways to it:

1. pqos -e "llc:0=0xffff" # Example: allocate all cache ways to CoS 0

• Replace "0xffff" with the appropriate bitmask representing the cache ways you want to allocate. Each bit in the mask represents a cache way.

1. pqos -s

Example Commands: Here is a practical example assuming you want to allocate 8 out of 12 ways of the L3 cache to DDIO:

1. # Install Intel RDT tools if not already installed
2. sudo apt-get install intel-cmt-cat
3.  
4. # Check current cache allocation
5. sudo pqos -s
6.  
7. # Allocate cache ways to DDIO (assuming DDIO uses CoS 0)
8. sudo pqos -e "llc:0=0x0ff0" # Allocating 8 ways (ways 4-11) to CoS 0
9.  

Note: Adjusting the L3 cache allocation can impact the performance of other workloads running on the system. Ensure you test the changes in a controlled environment before applying them to production systems. Additionally, some settings may be restricted or overridden by the system firmware or BIOS, so consult your system documentation for more details. This feature should be enabled by default in BIOS, and it is recommended that this not be changed.

More details can be found: Intel® Xeon® Scalable Family [Whitley] Platform Performance and Power Optimization Guide (CNDA required).

Please let us know if you have any further questions.

Best regards,

Ali

Intel Customer Support

[sc3289]

Thank you for the detailed explanation. I am familiar with the method you described.

However, as far as I know, the LLC ways allocated to DDIO are shared among all CPUs within the same NUMA node. In other words, when using a command like pqos -e "llc:0=0xffff", increasing the cache ways allocated to CoS 0 should increase the cache ways available to the applications running on the CPUs associated with CoS 0, rather than specifically impacting DDIO.

That said, your explanation seems to suggest that in Ice Lake, DDIO portions are managed independently for each CoS. Have I understood this correctly? Looking forward to your clarification.

Thank you in advance for your response.

Best regards,
sc3289

[IntelSupport]

Hi sc3289,

Greetings!

Yes, your understanding regarding the DDIO portions managing independently for each CoS is correct.

Please feel free to let us know if you have any further questions or need additional clarification.

Regards,

Akshaya 

Intel Customer Support Technician

Intel.com/vroc 

[pujeeth]

Hi sc3289,

Greetings!

This is the first follow-up regarding the issue you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided.

Feel free to reply to this email, and we'll be more than happy to assist you further.

Regards,

Pujeeth

Intel Customer Support Technician

intel.com/vroc