How to use NTB (Non-transparent-bridge) in R1000GZ/GL server

[MLee57]

Hi,

I got two R10000GZ/GL servers couple days ago. I need them to try the PCIe Non-transparent-bridge (NTB) function. After installing Fedora 17 OS, there is no NTB function available. I think I should set something in BIOS to enable the NTB function but I can not find the place to do it. Could someone can teach me how to enable the NTB function ? Or there is any document about this? Thanks very much.

[MLee57]

I found BIOS setting information on NTB_s2600jf_r1_4.pdf. There are many NTB setting in BIOS Adavance->PCI configuration menu. But in my BIOS there is nothing about NTB. My BIOS version is SE5C600, 01.08.0003, build date 02/26/2013 15:21:51

[Daniel_O_Intel]

This option is very specific to the hardware configuration of the S1600JP/S2600JF/S2600CO/S2600WP board, and appears only for those boards.

[MLee57]

Yes, I understand the NTB is a very specific function. So I confirmed with the sales that the R1000GZ/GL must support the NTB function. The answer is positive so I buy the two server. I read many documents and all of them say the R1000GZ/GL supports NTB, doesn't it?

[MLee57]

I check the document of R1000GZ/GL, the board is S2600GZ/S2600GL. So the R1000GZ/GL has no change to support NTB function? The only way is to change board ? Thanks for your help!

[Daniel_O_Intel]

I checked the TPS, and NTB is mentioned once. I'll dig around and find out more information.

[Daniel_O_Intel]

Applefly -

I verified, NTB is *not* supported on the Intel Server Board S2600GZ/S2600GL Family, which is used inside the Intel Server System R1000GZ/GL Family

[MLee57]

Thanks! I will change to the following board, it supports NTB ?

http://www.intel.com/content/www/us/en/server-systems/server-system-r1300jp.html Intel® Server System R1304JP4GS

Intel® Server Board S1600JP4

8 DDR3 ECC UDIMM 1333, RDIMM 1600, LRDIMM 1333

Supports up to 4 hot-swap 2.5" or 3.5" drives

750W AC, Platinum Efficient

[Daniel_O_Intel]

Applefly -

Yes, that list of boards I mentioned above, S1600JP/S2600JF/S2600CO/S2600WP, the one you are looking at is the S1600JP.

[SBlitz]

Pay attention it's a full system, not a board, so you need S1600JP based system (like R1208JP...)

[ICust32]

Hi Dan_0 and Saar Blitz,

Does S2600CP family also support NTB? Thank you.

Regards,

William

[Daniel_O_Intel]

u9012063 -

No, not the S2600CP. Only the:

S1600JP

S2600JF

S2600CO

S2600WP

Those two letters at the end, tell you which product family. If your product doesn't have JP, JF, CO, or WP in it, then it does not support NTB.

[MLee57]

Hi Dan_0 and Saar Blitz,

I want to purchase a INTEL server with the following capability,

1. There are two PCIe adapters pluged in the server

2. The PCIe adapters will read/write the memory in the other adapter through DMA

3. The most important is that the IOMMU is enabled.

As my understanding,

When the IOMMU is enabled, all the PCIe TLP transaction will go to the IOMMU to check the authority.

case 1.

The adapters access the DRAM is OK, the TLP path is OK from adapter to IOMMU then go to DRAM.

case 2.

But when adapter A tries to write a data into adapter B's memory, the TLP should go to IOMMU and then go to adapter B.

In this case, there should be one PCIe bridge behind IOMMU to handle the TLP, let it resend to adapter B.

I can not find any information about the relation of IOMMU and PCIe root complex.

I don't know which INTEL server can support the case 2.

Please give me the server model name or where I can get more information about my requirement.

Thank you very much.

[Daniel_O_Intel]

http://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware List of IOMMU-supporting hardware - Wikipedia, the free encyclopedia