MPI ports

[Danilom67]

Hi All,

I need to run an application using MPI on Windows over a network where:

- firewall must be active

- I can only add firewall exeptions for a limited range of ports (I can't add exception for all ports and a given exe)

I have set I_MPI_PORT_RANGE, FI_TCP_PORT_LOW_RANGE and FI_TCP_PORT_HIGH_RANGE to the ports opened in the firewall. I have also set I_MPI_HYDRA_SERVICE_PORT to a port inside the range and run hydra_service.exe with the -port option on the remote machine.

mpiexec hangs or returns a connection error when I try to run an exe  (cpuinfo.exe) on the remote machine.

Checking with TCPView the port it is trying to use I see that they are outside the specified range.

mpiexec version is 2021.6.0.0.

What I need to do to ensure that mpi will use only a given port range?

Thanks,

Danilo

[ShivaniK_Intel]

Hi,

Thanks for reaching out to us.

Could you please share the debug output by setting I_MPI_DEBUG?

set I_MPI_DEBUG=20

mpiexec -n <# of processes> test.exe

Thanks & Regards

Shivani

[Danilom67]

Thanks Shivani.

Just to clarify, in my test case I have two Windows PCs (PC1 and PC2):

•  Hydra_service.exe has been installed and is running on PC2.
• On PC1 I run mpiexec -host PC2 -n 1 <mpiPath>\cpuinfo.exe; cpuinfo.exe is the tool provided in the intel mpi package and it is not an MPI program. I would like to run it in the remote PC2 machine and visualize its output in the local PC1.

Windows firewall is active in both PCs and I am trying to create firewall rules to allow mpiexec (on PC1) to hydra_service (on PC2) communications.

The test runs successfully if I create (see fig1.png) :

• two inbound rules on PC1 allowing the connection to mpiexec.exe through TCP and UDP without limiting the port range and
• two inbound  rules on PC2 allowing the connection to hydra_service.exe through TCP and UDP without limiting the port range 

I then tried to limit to the range 50000:50500 the ports to open in the firewall . To do so:

• I have set on PC2 I_MPI_PORT_RANGE=50000:50500
• I started hydra_service on PC2  running hydra_service.exe -p 50000 -install
•  I have set on PC1 I_MPI_HYDRA_SERVICE_PORT=50000 and I_MPI_PORT_RANGE=50000:50500
•  I changed firewall rules specifying the given port range (see fig2.png) for both Local and Remote ports

In this situation, mpiexec executed on PC1 hangs without showing any error message.

With further investigations I noticed that it works again if I change all the firewall rules (both on PC1 and PC2) limiting the port range only for local ports as shown in fig.3.png.

It sounds like mpi selects input ports inside the range specified by the I_MPI_PORT_RANGE while output ports can be outside the range. That's why limiting in the firewall the remote ports to the range make it stop working.

Thanks,

Danilo

[ShivaniK_Intel]

Hi,

Could you please share the output of the below command?

netstat -ab

Could you also please share the logs generated with I_MPI_HYDRA_DEBUG=1 ?

Thanks & Regards

Shivani

[Danilom67]

Hi,

Hydra service is running on 192.168.230.40:50000.

mpiexec is executed on the PC called PCH638 (192.168.220.158).

The following environment variables are defined on PCH638:

I_MPI_HYDRA_DEBUG=1
I_MPI_HYDRA_SERVICE_PORT=50000
I_MPI_PORT_RANGE=50000:50500

While mpiexec is running, I executed netstat -ab on PCH638 ad it returned the following lines related to mpiexec.exe:

TCP 0.0.0.0:50000 PCH638:0 LISTENING
[mpiexec.exe]
TCP 127.0.0.1:1917 PCH638:1918 TIME_WAIT
TCP 127.0.0.1:1928 PCH638:1929 ESTABLISHED
[mpiexec.exe]
TCP 127.0.0.1:1929 PCH638:1928 ESTABLISHED
[mpiexec.exe]
TCP 192.168.220.158:1930 192.168.230.40:50000 ESTABLISHED
[mpiexec.exe]

As you can see in the last line, it opened a connection to hydra service on 192.168.230.40:50000 opening on PCH638 the TCP port number 1930 that is outside the range specified in I_MPI_PORT_RANGE.

No logging is generated by mpiexec.

Thanks.

Best regards,

Danilo

[ShivaniK_Intel]

Hi,

Could you please let us know whether you can set the environment variable by using the "-genv I_MPI_PORT_RANGE 50000:50500" as an option to mpiexec?

Thanks & Regards

Shivani

[Danilom67]

Hi Shivani,

No I set it in the dos shell before running mpiexec.

Thanks,

Danilo

[ShivaniK_Intel]

Hi,

Could you please try setting the environment variable by using the "-genv I_MPI_PORT_RANGE 50000:50500" as an option to mpiexec, and let us

 know if you further face any issues?

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

As we didn't hear back from you, Could you please let us know if you face any issues?

Thanks & Regards

Shivani

[Danilom67]

Hi Shivani,

Sorry for the delay in my response.

Unfortunately even if I pass "-genv I_MPI_PORT_RANGE 50000:50500" to mpiexec I have the same behaviour.

Just to help you reproduce it in your test environment I inform you that I tried in a couple of PC With Windows 10 pro  (one with version1909 and the other with version 21H2) using mpiexec version 2021.6.0.0. 

In both PC it happens sistematically.

One example of netstat output showing the issue is:

TCP 192.168.201.166:65085 192.168.230.40:50000 ESTABLISHED
[mpiexec.exe]

where port 65085 is outside the specified range.

Thanks,

Danilo

[ShivaniK_Intel]

Hi,

>>>"TCP 192.168.201.166:65085 192.168.230.40:50000 ESTABLISHED"

 Could you please confirm if the IP address 192.168.201.166 correspond to the remote machine?

Thanks & Regards

Shivani

[Danilom67]

Hi,

No, 192.168.201.166 is the machine runnig mpiexec.

192.168.230.40 is the remote machine where hydra_service is listening at port 50000.

Regards,

Danilo

[ShivaniK_Intel]

Hi,

We are working on it and will get back to you.

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

Thanks for reporting this issue. We are able to reproduce it and we have informed the development team about it.

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

Could you please try the latest version of Intel oneAPI 2023.0.0 and let us know if you face any issues?

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

As we did not hear back from you could you please us with an update on my previous post?

Thanks & Regards

Shivani

[ShivaniK_Intel]

Hi,

I have not heard back from you. This thread will be no longer monitored by Intel. If you need further assistance, please post a new question.

Thanks & Regards

Shivani

[Danilom67]

Hi Shivani,

sorry for my late reply.

I can't do the test now, I will open a new thread in case of problems.

Thank you very much for your support.

Danilo