Intel MPI 2021.6 on 64 core Windows Server 2022

[Frank_R_1]

Dear support team,

A customer have the following setup:
Microsoft Windows Server 2022 Standard Version 10.0.20348 Build 20348
Dell Inc. PowerEdge R840 quad socket 4 x Intel Xeon Gold 6254 with 128GB RAM (8x16GB)
Hyperthreading enabled, so 72 cores and 144 logical cores

We run our product on this machine with Intel MPI 2021.6 and the following call command:
mpiexec.exe -np 32 -delegate -genvall -print-all-exitcodes -genv I_MPI_OFI_PROVIDER sockets -genv I_MPI_HYDRA_BSTRAP_KEEP_ALIVE 1 -genv I_MPI_CBWR 2 -genv I_MPI_ADJUST_GATHERV 3 -envall OUR_PRODUCT PARAMETERS

The performance is very good on 32 cores, but with
mpiexec.exe -np 64 -delegate -genvall -print-all-exitcodes -genv I_MPI_OFI_PROVIDER sockets -genv I_MPI_HYDRA_BSTRAP_KEEP_ALIVE 1 -genv I_MPI_CBWR 2 -genv I_MPI_ADJUST_GATHERV 3 -envall OUR_PRODUCT PARAMETERS

performance totally breaks down and we don't have a clue why.

In the taskmanager we see four NUMA domains which all are utilized homogeneously also kernel time is decently low.
What we didn't check so far are the BIOS settings, but we think these are standard settings by vendor (Dell).

Question:
What could be the problem here:
Windows config?
BIOS config?
Intel MPI config (libfabric etc)

The Machine has four sockets with Xeons which have each 6 memory channels, so it should definitly give speed up from 32 to 64 cores.
What we see from MPI Debug output is the following rank to core pinning:

rank 0 0
rank 1 1
rank 2 2
rank 3 3
rank 4 4
rank 5 5
rank 6 6
rank 7 7
rank 8 8
rank 9 9
rank 10 10
rank 11 11
rank 12 12
rank 13 13
rank 14 14
rank 15 15
rank 16 16
rank 17 17
rank 18 18
rank 19 19
rank 20 20
rank 21 21
rank 22 22
rank 23 23
rank 24 24
rank 25 25
rank 26 26
rank 27 27
rank 28 28
rank 29 29
rank 30 30
rank 31 31
rank 32 32
rank 33 33
rank 34 34
rank 35 35
rank 36 0
rank 37 1
rank 38 2
rank 39 3
rank 40 4
rank 41 5
rank 42 6
rank 43 7
rank 44 8
rank 45 9
rank 46 10
rank 47 11
rank 48 12
rank 49 13
rank 50 14
rank 51 15
rank 52 16
rank 53 17
rank 54 18
rank 55 19
rank 56 20
rank 57 21
rank 58 22
rank 59 23
rank 60 24
rank 61 25
rank 62 26
rank 63 27

This looks a little bit strange.

Has anyone similar experience on quad socket machines?

Best regards
Frank

[SantoshY_Intel]

Hi,

Thanks for posting in the Intel communities.

According to the Intel® MPI Library System Requirements, given below are the supported Windows Os:

• Microsoft* Windows* 7 (deprecated), 10
• Microsoft* Windows* Server 2012 (deprecated), 2012 R2 (deprecated), 2016, 2019

So could you please try with any one of the supported Windows OS and get back to us if the issue exists?

Thanks & Regards,

Santosh

[Frank_R_1]

Hi,

Thank you for hint with the operating system.

Is Windows 10/11 professional for workstations supported? That would be my suggestion to use, since it supports quad socket systems.

Best regards

Frank

[SantoshY_Intel]

Hi,

As stated in the Intel® MPI Library System Requirements, you can use Windows 10 Professional for workstations.

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

We haven't heard back from you. Could you please provide us any update on your issue?

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.

Thanks & Regards,

Santosh