Creating a direct ethernet path to Xeon Phi

[Volker_G_]

Hi, 

We have a requirement to be able to pass traffic from an ethernet network directly on a Xeon Phi card installed on our host, bypassing the CPU and main memory of the host for performance reasons. That is, we want to have the Xeon Phi and the interface exchanging network packets directly. What are the options to achieve this? From what I understand, Xeon Phi is able to communicate directly with IB HCAs via CCL. Supposing that there is such an HCA that supports ethernet connectivity and can perform all the necessary eth-to-IB/IB-to-eth  transformations (encaps/decaps, assembly/disassembly, whatever they might be...), would it suffice for Xeon Phi to achieve the desired result? 

Thanks in advance

[Volker_G_]

What I am sketching in my initial post I guess that is some kind of RDMA over Ethernet, which seems to be provided by the RoCE technology. Some adapters like Mellanox ConnectX-3 provide RoCE capabilities. There is a blog post in the developer zone demonstrating remote MIC-to-MIC connectivity using ConnectX-3 adapters, but in back-to-back mode using Infiniband cable (https://software.intel.com/en-us/blogs/2014/09/23/working-with-mellanox-infiniband-adapter-on-system-with-intel-xeon-phi-coprocessors). My guess is that the adapters should be able to connect in RoCE mode on an ethernet switch, as well, by now the question is whether the Xeon Phi-compatible OFED distributions would work properly in this mode. Do you have any clear view on that?  

[Frances_R_Intel]

That is a good question. I will have to pass it on to see what the developers have to say. It seems to me that there are two cases:

1) Using something like a Mellanox adapter that has both IB and Ethernet ports.

2) Using a straight Ethernet adapter with RoCE. The IB connection to the coprocessor is done using the iWARP protocol over the PCI bus. How far is it to go from that to RoCE?

I'm not a networking person, so I couldn't even hazard a guess, but as I say, it is a good question and I will pass it on.

[Frances_R_Intel]

From the developers: RoCE on the Intel Xeon Phi coprocessor is something that has not been investigated and the amount of effort to do this is unknown. There are no plans to look into it.

As far as going directly to an Ethernet adapter in general, you might want to read https://software.intel.com/en-us/forums/topic/394321. The original question asked about directly controlling the Ethernet adapter from the coprocessor, not just using RDMA. The host still needs to get involved in the set up and interrupt handling and getting around this is not reasonable. The post then goes on to talk about writing something along the lines of CCL-Direct for an Ethernet adapter. As it says in that thread:  "one of the reasons why CCL-direct works well is because Infiniband hardware is intentionally designed in such a way that only a few simple control register manipulations are necessary in the fast path."

As far as doing something like using a coprocessor InfiniBand connection to the Mellanox adapter to Ethernet network, I got one opinion that said maybe. You would want to talk to Mellanox to see if this is actually doable.

[Volker_G_]

Hi Frances, 

Many thanks for your feedback. My feeling is that the split driver might be more suitable for our case. A thought that passed from my mind after your last input is the possibility to use some kind of poll-mode drivers (PMD) for the ethernet adapter, like those offered by the DPDK framework (http://dpdk.org/doc/nics), instead of the regular, interrupt-based driver. PMDs typically poll device's RX queues and receive packets in bursts. I am not sure what mechanisms are being used to achieve this, if host involvement is required or not, but I thought that it might be easier compared to the interrupt-based drivers. 

[Frances_R_Intel]

Based on what Evan says in https://software.intel.com/en-us/forums/topic/394321, you might be able to build a solution using your polling idea, but there are a lot of dragons lurking in the shadows. I'm afraid this is not something the MPSS developers are working on and you have reached the limit of my knowledge. Perhaps someone else who is  lurking in the shadows along with the dragons will have ideas you can bounce off each other. We can but hope.