Broadwell CPU (E5 2650) with two memory controllers

[EBoug]

Hello,

I have a server with one socket (E5 2650 cpu) and 128GB DRAM (8 DIMMS of 16 GB). As it can be seen from the specs the cpu has 12 cores and two integrated memory controllers. The first memory controller uses two channels(channel 0 and 1) and the second one also uses two channels(channels 3 and 4). Although the channel interleaving is disabled through bios (my setting is 1 -way channel interleaving) there is an interleaving between two channel of different controllers. The interleaving appears between channels 0 and 2 or 1 and 3. Can i disable this type of interleaving?

Thanks,

diama13

[idata]

Hello Diama13,

Thank you for contacting Intel Technical Support.I understand that you are looking to disable the interleaving between two channel of different controllers. In an effort to better assist, can you please provide the model of the server board you are working with? regards,Jeremiah A.Intel Technical Support

[EBoug]

Hello Jeremiah,

thank you for your prompt response.

The server we use in our lab is a X10SRi-F Supermicro Server (http://www.supermicro.com/products/motherboard/xeon/c600/x10sri-f.cfm Supermicro | Products | Motherboards | Xeon® Boards | X10SRi-F ). Just to mention that BIOS setting is set to one way channel interleaving.

Thanks,

diama13

[idata]

Hello diama13,

Thank you for your quick response.Based on the information provided, have you consulted this issue with SuperMicro? Since the interleaving feature is part of the motherboard BIOS? It could be that the board does not support 128 GB and they disabled that feature for the same.Please let me know your results.regards,Jeremiah A.Intel(R) Technical Support

[EBoug]

Hello Jeremiah,

Supermicro provided us with a BIOS version with channel interleaving set to 1 way. Our motherboard can support up to 1 TB memory. As it seems the interleaving exists between the two Home Agents (memory controllers). Which unit is responsible for this interleaving?

Thanks a lot.

Regards,

diama13

[idata]

Hi diama13,

You may go ahead and enable the Two -Way interleaving from the BIOS for this purpose.Please let me know your results.regards,Jeremiah A.Intel(R) Technical Support

[EBoug]

Hello Jeremiah ,

before contacting with Supermicro in order to provide us with a new BIOS with this option enabled(channel interleaving set to two way) could you explain me the reason for setting this option?

Thanks,

diama13

[idata]

Hello diama13,

I hope you are doing well today.The Xeon processor optimizes memory accesses by creating interleave sets across the memory controllers and/or memory channels. For example, if identical DIMMs are populated on both memory channels attached to a memory controller, the memory controller creates a 2-way interleave set across both DIMMs.If DIMMS with different memory capacities are populated on the memory channels attached to a memory controller or if different numbers of identical capacity DIMMs are populated on the memory channels, the memory controller has to create multiple interleave sets. Managing multiple interleave sets creates overhead for the memory controller which can reduce memory bandwidth. Please see attachment for example. I hope this clarifies your inquiry. Please let me know if you have more questions, more than glad to assist.regards,Jeremiah A.Intel(R) Technical Support

[EBoug]

Hello Jeremiah ,

As you mentioned Xeon processor optimizes memory accesses by creating interleave sets across the memory controllers and/or memory channels. I have disabled only the interleaving between channels. So, can I disable all types of interleaving (across both memory controllers and channels)? If yes, which is the process (steps) for doing this?

Thank you in advance.

Regards,

diama13

[idata]

Hello diama13,

In an effort to better assist you, can you please explain the purpose of you disabling this feature?regards,Jeremiah A.

[EBoug]

Hello Jeremiah

We use an FPGA board that accesses data on the host's DRAM over PCIe. In our case the latency for accessing the host memory is the most critical parameter. Although we have 128GB memory, at boot time we restrict the memory used by the OS to some GigaBytes. So, the PCIe transactions are performed to the remaining GigaBytes. We believe that all kind of interleaving must be disabled in order to achieve our target.

Regards,

diama13

[idata]

Hello diama13,

Thank you for the explanation.In this case, because the feature is the BIOS, I suggest contacting SuperMicro in order to get assistance on how to disable the feature.regards,Jeremiah A.

[idata]

Hello diama13,

I hope you are doing well today.I'm following up with you to see if the information provided helps and if there is something else I can help you with or we can proceed in closing this case.regards,Intel(R) Technical Support

[idata]

Hello diama13,

I hope you are doing well today.I'm following up with you to see if the information provided helps and if there is something else I can help you with or we can proceed in closing this case.regards,Intel(R) Technical Support

[EBoug]

Hello,

as already mentioned Supermicro has informed us about the settings in BIOS in order to deactivate the channel interleaving. Unfortunately interleaving between channels of different controllers remains. Eventually, I can't find a solution. You can close our discussion.

Regards,

diama13

[idata]

Hello diama13,

I will proceed in closing this case.Thank you for contacting Intel Technical Support.regards,Jeremiah A.Intel Technical Support