topic Hi Eric, in Software Archive

Improving MPI performance between 2 Xeon Phi cards

luckynew — Mon, 11 Feb 2013 13:35:43 GMT

Hello,

I am trying to improve performance running native MPI version between 2 Xeon Phi cards.

I wanted to test if I_MPI_SSHM_SCIF=1 may help, but I encountered the error below:

Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(647)........................:
MPID_Init(192)...............................: channel initialization failed
MPIDI_CH3_Init(100)..........................:
MPID_nem_impi_shm_scif_connections_init(1864):
MPID_nem_impi_shm_scif_accept(1956)..........: Internal MPI error! cannot receive peer's rank
Fatal error in MPI_Init: Other MPI error, error stack:

If I don't set this environment variable, my application is working correctly.

Is it worth investigating why this setting is not working? Or is there other setting to optimize MPI communcation between 2 cards or host and one card?

Thanks for your advice,

Eric.

Hi Eric,

Loc_N_Intel — Mon, 11 Feb 2013 23:20:08 GMT

Hi Eric,

Before you launched the application, did you enable the environment variable I_MPI_MIC ?

# export I_MPI_MIC=enable

The article at http://software.intel.com/en-us/articles/how-to-run-intel-mpi-on-xeon-phi gives a good overview on how to run an MPI program on Intel(R) Xeon Phi(TM).

Thanks,

Hi Loc-Nguyen,

luckynew — Tue, 12 Feb 2013 10:11:35 GMT

Hi Loc-Nguyen,

Thank you for your reply!

Yes, I use I_MPI_MIC=enable though I never notice bad behaviour without it or performance gain to use it.

And, yes, I already know the documentation you referred to.

Any other idea about my error, the importance of using I_MPI_SSHM_SCIF, or any other flag that may help to speedup communication between 2 cards or host and Xeon Phi?

Hello,

Loc_N_Intel — Thu, 21 Feb 2013 00:50:09 GMT

Hello,

Can you enter the following command and try to run again:

sudo /sbin/sysctl -w net.ipv4.ip_forward=1

Thanks.

Hello,

luckynew — Thu, 21 Feb 2013 08:23:16 GMT

Hello,

This was already done (without, I don't think it is even possible to run MPI application between 2 cards or host+Phi)...

Thanks.

Hi Eric,

Loc_N_Intel — Wed, 27 Feb 2013 23:33:20 GMT

Hi Eric,

I ran the following command:

mpiexec.hydra -genv I_MPI_SSHM_SCIF 1 -n 2 -host mic0 /tmp/montecarlo.mic : -n 3 -host mic1 /tmp/montecarlo.mic

Where montecarlo.mic is the binary for Phi. And it worked as expected.

If it still doesn't work for you, could you provide information about your Linux distribution version, the MPSS version (use the command "micinfo"), the Intel compiler version and the Intel MPL library version please.

Regards.

Hello Loc-Nguyen,

luckynew — Wed, 13 Mar 2013 10:53:35 GMT

Hello Loc-Nguyen,

Sorry about my late answer. I tried to sort out why it fails on my system but still have no clue. Instead of my industrial application, I tried with a simple example comparable to yours (montecarlo based computation of PI) and I reproduced the same problem. So, I think it is related to my local installation. Please find below the answer to the questions you asked:

[radioss@fradf-shady neon]$ mpiexec.hydra -genv I_MPI_SSHM_SCIF 0 -n 2 -host mic0 a.out.mic : -n 3 -host mic1 a.out.mic
MPI program results with 5 processes:
number of samples = 10000, seed = 10
estimated pi = 3.1000000000
difference between estimated pi and math.h M_PI = 0.0415926536
time to compute = 0 seconds
[radioss@fradf-shady neon]$ mpiexec.hydra -genv I_MPI_SSHM_SCIF 1 -n 2 -host mic0 a.out.mic : -n 3 -host mic1 a.out.mic
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(647)........................:
MPID_Init(192)...............................: channel initialization failed
MPIDI_CH3_Init(100)..........................:
MPID_nem_impi_shm_scif_connections_init(1877):
MPID_nem_impi_shm_scif_connect(2067).........: Internal MPI error! cannot perform connect
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(647)........................:
MPID_Init(192)...............................: channel initialization failed
MPIDI_CH3_Init(100)..........................:
MPID_nem_impi_shm_scif_connections_init(1864):
MPID_nem_impi_shm_scif_accept(1956)..........: Internal MPI error! cannot receive peer's rank
[radioss@fradf-shady neon]$ uname -a
Linux fradf-shady.europe.altair.com 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux[radioss@fradf-shady neon]$ ifort -V

Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 13.0.1.117 Build 20121010

[radioss@fradf-shady neon]$ which mpiexec.hydra
/opt/intel/impi/4.1.0.018/intel64/bin/mpiexec.hydra

[radioss@fradf-shady neon]$ sudo micinfo
[sudo] password for radioss:
sudo: micinfo: command not found
[radioss@fradf-shady neon]$ which micinfo
/opt/intel/mic/bin/micinfo
[radioss@fradf-shady neon]$ sudo /opt/intel/mic/bin/micinfo
MicInfo Utility Log

Created Wed Mar 13 11:45:27 2013

System Info
Host OS : Linux
OS Version : 2.6.32-71.el6.x86_64
Driver Version : 4982-15
MPSS Version : 2.1.4982-15
Host Physical Memory : 24541 MB
CPU Family : GenuineIntel Family 6 Model 44 Stepping 2
CPU Speed : 1596.000
Threads per Core : 1

Device No: 0, Device Name: Intel(R) Xeon Phi(TM) Coprocessor

Version
Flash Version : 2.1.05.0375
UOS Version : 2.6.38.8-g32944d0
Device Serial Number : ADKC22000357

Board
Vendor ID : 8086
Device ID : 225c
SubSystem ID : 2500
Coprocessor Stepping ID : 1
PCIe Width : x16
PCIe Speed : 5 GT/s
PCIe Max payload size : 256 bytes
PCIe Max read req size : 512 bytes
Coprocessor Model : 0x01
Coprocessor Model Ext : 0x00
Coprocessor Type : 0x00
Coprocessor Family : 0x0b
Coprocessor Family Ext : 0x00
Coprocessor Stepping : B0
Board SKU : ES2-P1750
ECC Mode : Enabled
SMC HW Revision : Product 300W Active CS

Core
Total No of Active Cores : 61
Voltage : 1021000 uV
Frequency : 1090909 kHz

Thermal
Fan Speed Control : On
SMC Firmware Version : 1.7.4172
FSC Strap : 14 MHz
Fan RPM : 2700
Fan PWM : 50
Die Temp : 62 C

GDDR
GDDR Vendor : Elpida
GDDR Version : 0x1
GDDR Density : 2048 Mb
GDDR Size : 7936 MB
GDDR Technology : GDDR5
GDDR Speed : 5.500000 GT/s
GDDR Frequency : 2750000 kHz
GDDR Voltage : 1000000 uV

Device No: 1, Device Name: Intel(R) Xeon Phi(TM) Coprocessor

Version
Flash Version : 2.1.05.0375
UOS Version : 2.6.38.8-g32944d0
Device Serial Number : ADKC22000339

Core
Total No of Active Cores : 61
Voltage : 1047000 uV
Frequency : 1090909 kHz

Thermal
Fan Speed Control : On
SMC Firmware Version : 1.7.4172
FSC Strap : 14 MHz
Fan RPM : 2400
Fan PWM : 50
Die Temp : 58 C

GDDR
GDDR Vendor : Elpida
GDDR Version : 0x1
GDDR Density : 2048 Mb
GDDR Size : 7936 MB
GDDR Technology : GDDR5
GDDR Speed : 5.500000 GT/s
GDDR Frequency : 2750000 kHz
GDDR Voltage : 1000000 uV

Hello Eric,

Loc_N_Intel — Thu, 14 Mar 2013 00:47:47 GMT

Hello Eric,

I just remember one more thing, would you like to verify the file /etc/modprobe.d/mic.conf to see if p2p enabled? If not, please set p2p=1 in that file and then reboot the system.

For your reference,here is what I set in the file /etc/modprobe.d/mic.conf:

options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1

Regards.

Hi,

luckynew — Thu, 14 Mar 2013 08:53:45 GMT

Hi,

This is was I already have:

options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 pm_qos_cpu_dma_lat=75

So, I think it is ok.

I still not understand why everything is working except this setting of I_MPI_SSHM_SCIF 1

And I even do not know how it is important to enable it or not regarding performance...

Thanks for all your help,

Eric.

Hi Eric,

Loc_N_Intel — Thu, 14 Mar 2013 21:37:09 GMT

Hi Eric,

Would you like to set the environment variable I_MPI_FABRICS to "shm:tcp" and run your program again please? Thank you.

> mpiexec.hydra -genv I_MPI_FABRICS shm:tcp -genv I_MPI_SSHM_SCIF 1 -n 2 -host mic a.out : -n 3 -host mic1 a.out

Hi Loc-Nguyen,

luckynew — Fri, 15 Mar 2013 12:48:25 GMT

Hi Loc-Nguyen,

I tried your modification, but I still encounter the same error.

Eric.

Hi Eric,

Dmitry_S_Intel — Sat, 16 Mar 2013 05:18:21 GMT

Hi Eric,

You are using quite old Beta Intel(R) MPI Library 4.1.0.018. Now we have released Intel MPI Library.

Please consider to upgrade to release version. It can be found at https://registrationcenter.intel.com/RegCenter/Download.aspx?productid=1626

---

Dmitry

Hi Dmitry, Loc,

luckynew — Tue, 26 Mar 2013 16:49:58 GMT

Hi Dmitry, Loc,

You were right! I installed latest MPI 4.1.0.30 and now the issue with I_MPI_SSHM_SCIF=1 has disapeared.

Now, it works the same way with I_MPI_SSHM_SCIF = 0 or 1.

Well, I haven't noticed any significant effect on performance to use 1 or 0.

The good news is that MPI 4.1.0.30 performs better than the old one in term of performance.

Thank you very much for your help !

Eric.

Hi Eric,

Loc_N_Intel — Tue, 26 Mar 2013 17:11:54 GMT

Hi Eric,

I just learned that I_MPI_SSHM_SCIF is deprecated in Release version of IMPI 4.1.

Thanks,