topic Re: impi/mkl_scalapack causing kernel panic in Intel® MPI Library

impi/mkl_scalapack causing kernel panic

poulson_jack — Mon, 21 Jan 2008 19:26:21 GMT

I'm testing out the cluster toolkit for Linux on em64t and the scalapack Cholesky factorization routine (pdpotrf_) causes a Machine Check Exception and kernel panic very predictably. No other scalapack/pblas routine so far has caused any problems.

Additionally, each time I start an mpi job using 'mpiexec', the mpd daemon outputs the following:
unable to parse pmi message from the process :cmd=put kvsname=kvs_nerf_4268_1_0 key=DAPL_PROVIDER value=

It may not matter, but the only way I've found to get the mpd ring running is:
[host]$ mpd --ifhn=10.0.0.1 -l 4268 &
[node]$ mpd -h nerf -p 4268 &

Any help clearing these up would be greatly appreciated, as I won't be purchasing the software otherwise. MCEs are unacceptable.

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Mon, 21 Jan 2008 19:28:04 GMT

The forum pulled out the last part of the error message: "value="

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Mon, 21 Jan 2008 19:28:38 GMT

In brackets, "NULL string"

Re: impi/mkl_scalapack causing kernel panic

Andrey_D_Intel — Thu, 24 Jan 2008 12:21:29 GMT

Hi,

Could you clarify the Intel MPI Library version you use? Please check package ID information in the mpisupport.txt file.

By the way, you can fill a bug report at https://primer.intel.com to get a technical assistance.

Best regards,

Andrey

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Thu, 24 Jan 2008 19:54:36 GMT

Package ID: l_mpi_p_3.1.026

If I had to guess, I would say it's in pdsyrk. I heavily performance tested pdtrsm and dpotrf before trying pdpotrf.

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Fri, 25 Jan 2008 15:19:54 GMT

That link doesn't work.

Re: impi/mkl_scalapack causing kernel panic

Andrey_D_Intel — Fri, 25 Jan 2008 15:27:45 GMT

Could you give more details on cluster configuration? I'd like to understand why you was not able to use mpdboot to launch MPD ring.

Re: impi/mkl_scalapack causing kernel panic

Andrey_D_Intel — Fri, 25 Jan 2008 15:29:53 GMT

I did a misprint. Sorry. The right link is https://premier.intel.com

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Fri, 25 Jan 2008 16:23:40 GMT

The test cluster consists of 2 4 processor machines behind a firewall. The headnode, nerf, has two ethernet ports, one connected to the firewall, one to the node, ball. All IPs are in the 10.0.0.0 network.

When I try:
mpdboot --totalnum=2 --file=./mpd.hosts --rsh=ssh

the output is:
mpdboot_nerf (handle_mpd_output 681): failed to ping mpd on ball; received output={}

Also, the premier support link won't let me in, as I'm only evaluating the software right now.

Re: impi/mkl_scalapack causing kernel panic

Andrey_D_Intel — Wed, 30 Jan 2008 12:39:42 GMT

Is it allowed to establish connection from compute nodes to the head node?mpdboot alwaysstart mpd daemon on local node first .After that remote mpd daemonst attemt to perfrom connection to it?
Do you able to start mpd manually?

Run the mpd -e -d command on the head node. The port number will be printed on stdout.
Run the mpd -h head_node -p -d command to establish MPD ring. Use the port number printed at pervious step.
Check if ring was established succesfully. Run the mpdtrace command for that.

Re: impi/mkl_scalapack causing kernel panic

Andrey_D_Intel — Wed, 30 Jan 2008 12:46:32 GMT

Ops! I see that you can start ring manually. Could you share the content of your mpd.hosts file? Could you share the output from mpdboot -d -v... command? Is there any useful information in /tmp/mpd2.logfile_

Re: impi/mkl_scalapack causing kernel panic

poulson_jack — Sat, 02 Feb 2008 01:33:00 GMT

After reconfiguring the network settings several times, and reorganizing all of my environment variables (I had several MPI implementations installed), the problem went away, and I could boot up the MPD daemons via:
mpdboot --file= --rsh=ssh

I wish I could explain more specifically, but I changed far too many things in the process of compiling ScaLAPACK from scratch for several MPI implementations.