how do I use direct SMB in MPI?

[ALaza1]

I have dual x550-t2s running with Windows 10 direct SMB. Direct SMB is able to use both of these NICs for file transfers between systems. Because teaming for Windows 11 won't be done, it seems to me that direct SMB would be an excellent feature for MPI to use as a transport layer. Or has Intel already done this, and I missed this in the documentation?

I can configure both NICs for use by an MPI application, and the performance is decent but not nearly close to what the Linux bonding option round-robin delivers on the same equipment that I’m using for my Windows/MPI work.

Also, I'm using a pair of 10 gig-e switches, one for each NIC's subnet. These NICs support large frames. This is a private network with effectively direct connections between the systems... a one hop maximum.

TIA.

Art

[ShivaniK_Intel]

Hi,

Thanks for posting in the Intel forums.

We are working on it and will get back to you soon.

Thanks & Regards

Shivani

[ALaza1]

https://www.intel.com/content/www/us/en/developer/articles/release-notes/mpi-library-release-notes-windows.html

includes this remark:

"Server Message Block (SMB) shares may be unavailable for MPI processes in case of powershell bootstrap (out-of-box behavior). Consider Network File System (NFS) usage or domain-based authorization for this case."

So maybe this version of MPI does support SMB?

art

[ShivaniK_Intel]

Hi,

Thanks for patience.

We are regret to inform you that we do not support multi-rail in any way on Windows at the moment. Direct SMB is not able to help here.

If you need support can raise a feature request for multi-rail support through priority support.

If you want to get access and support through priority support please refer to the below link

https://www.intel.com/content/www/us/en/developer/get-help/priority-support.html

Thanks & Regards

Shivani

[ALaza1]

Thanks for your attention to my question.

I've got working familiarity with Linux bonding and Windows teaming (an Intel feature included with its LAN drivers). Some Linux platforms support non-kernel based teams. Neither of these require multi-rail since bonding/teaming configure a host's NICs as a single virtual NIC. I think the major advantage to Linux bonding is that the kernel supports bonding whereas teaming is not supported by the Linux and probably not by the Windows either.

Direct SMB is supported by an undercover round-robin for whatever NICs might be in the same subnet. I only specify one hostname and direct SMB makes use of more than one NIC if more are available. It simply appears to me that multirail support wouldn't be necessary for MPI to make use of direct SMB, but this might just be wishful thinking on my part.

My current pseudo multirail includes two NICs per host. Each NIC has its own hostname. A two host run using both NICs like so:
mpiexec -host node1 -env FI_SOCKETS_IFACE=eth0 -n 16 executable : -host node2 -env FI_SOCKETS_IFACE=eth1 -n 16 executable

There's a noticeable improvement in the executable's clock time and the Windows task manager shows both NICs in use often approaching their maximum transfer rates. Other than MPI there are no additional apps or configuration tweaks. These NICs are part of a private subnet with the first NIC of each host is connected to one switch and the second NIC of each host is connected to a second switch. The switches themselves are not using teaming support per se.

Regards,

art

[TobiasK]

@ALaza1 Intel MPI does not make use of any OS provided bonding, instead we are using the NICs directly. MPI related workloads are usually quite latency sensitive, so the message striping has to happen at the MPI layer not at some other layer, as such, DirectSMB is not really an option to go forward with.
As of today, there was no request to support multi-rail on a Windows platform. I am happy that your workaround works, but keep in mind it's not officially supported and it may or may not work in future releases. 

Best
Tobias

[ALaza1]

Thanks for your explanation. One reason I've become interested in SMB is that it bypasses TCP/IP and delivers the best performance between my Windows 10 systems. At the moment, I get my best MPI performance with Ubuntu v22.4 and round-robin bonding. I don't yet have a working configuration for w_HPCKit_p_2024.0.1.35.
For example, doing simple Windows functions like drag 'n drop between mounted systems makes use of multiple NICs even when I use only a single domain name. I only 'mount' the remote system on a single physical NIC. I do not have Windows teaming/bonding installed/configured. Windows task manager shows both subnets are used for these copies, and clock times reflect the higher transfer rates. Even if my multiple subnet use is not actually simultaneous use to/from a host, other hosts can make use of a second subnet instead of loading up on a single subnet.