topic I was thinking that there was in Software Archive

Troubleshooting HOWTO: Bad hardware? MPSS? Configuration?

BelindaLiviero — Tue, 18 Jun 2013 23:30:12 GMT

Are you having problems with your hardware (Cannot see your Intel(R) Xeon Phi(tm) coprocessor? Sporadic accessibility?) or with the Intel(R) Manycore Platform Software Stack (Intel(R) MPSS) running reliably?

Attached to this post are PDF "flowcharts" that explain how you can troubleshoot the problem (note: Both Linux and Windows flowcharts are available), and shows what information you will want to collect if you need to escalate your issue to your OEM provider or Intel.

We hope this is is useful to you! Please let us know if you have found a boundary condition not comprehended properly by this "flow".

I have experienced a problem

Paul_C_7 — Mon, 26 Aug 2013 10:42:27 GMT

I have experienced a problem that looks like a bug in the the 64 bit memory to stack push instruction.

I am porting the Glasgow Pascal compiler to the MIC and have run into an error that looks very like a bad implementation of PUSH

It appears that a push instruction of the form:

push QWORD[ r8* 8+ label]

actually pushes the quadword at

push QWORD[ r8* 8+ label140ba08d9aadf+8]

Here are the relevant source lines along with the relevant assembler lines that they translate into

First we have a call on a run time library function written in C using C parameter passing.

;writeln( shiftindex[d,0]);

; note that shiftindex is declared as array[0..4,0..1] of integer

mov rcx, 5 ; field width info

mov rdx, 12 ; field width info

mov bl,BYTE ptr [ rbp+ -49]

movsx r8, bl

imul r8, 8

movsx rsi, dword ptr [ r8+ label140ba08d9aadf] ; the parameter for the value to be printed

movsx rdi, dword ptr [ unit$system$base+ -24] ; the file it will be sent to

.ifndef definedprintint

definedprintint=1

.extern printint

.endif

call printint;#imported

;--------

; this correctly prints out the 0th element of the row of the array shiftindex

Now we call a pascal function passing a row of the array by value on the stack using a push instruction to place the row on the stack

;compareImagePair (shiftindex)

; d is a byte

; #297

mov bl,BYTE ptr [ rbp+ -49]

movsx r8, bl

push QWORD[ r8* 8+ label140ba08d9aadf]

call label140ba08d9abe3

; this passes to the function the d+1 th element of the array shiftindex ; in other words the push function fetches the wrong element from the array ; as compared to the mov instruction used earlier

Printout from programme

First the contents of the shiftindex array

0 0

0 -1

0 1

-1 0

1 0

d shiftindex[d,0]

3 -1

what we get inside the function compareImagePair when we print the parameter

dirvec = 1 0

I have now concluded that

Paul_C_7 — Mon, 26 Aug 2013 15:53:09 GMT

I have now concluded that this is a bug in the assembler distributed with the MIC, if you replace the line

push QWORD[ r8* 8+ label140ba08d9aadf]

with

push QWORD ptr [ r8* 8+ label140ba08d9aadf]

it fetches the correct value not a value 8 on from the correct address.

Thanks for isolating this bug

James_C_Intel2 — Tue, 27 Aug 2013 15:17:03 GMT

Thanks for isolating this bug. It has been reported to the team that owns the assembler.

It appears that the x86_64

BelindaLiviero — Thu, 05 Sep 2013 16:52:55 GMT

It appears that the x86_64 assembler does the same thing.

The error is that "QWORD ptr" must be used here, as Paul realized. The fact that QWORD alone is allowed may be a bug, which we need to discuss internally; If AT&T syntax is used, what happens?

I am trying to install Xeon

Yue_H_ — Tue, 24 Sep 2013 19:12:02 GMT

I am trying to install Xeon Phi card on a Supermicro server (http://www.supermicro.com/products/superblade/module/sbi-7127rg.cfm). According to the flow chart, I need to "Enable support for mapping >4GB MMIO in the host BIOS" . However, I cannot see the MMIO setup option in BIOS even after upgrading to the latest version. Could anyone please give me some suggestions?

Hi Yue,

Matthias_H_Intel — Fri, 27 Sep 2013 09:49:16 GMT

Hi Yue,

from a quick view on your link it looks like it's an old server which might not support Xeon Phi at all. You might check with Supermicro whether this server could host Xeon Phi.

otherwise it might also be

Matthias_H_Intel — Fri, 27 Sep 2013 09:56:50 GMT

otherwise it might also be the case that your BIOS has this option by default - is your card detected at all?

... see also http://www

Matthias_H_Intel — Fri, 27 Sep 2013 10:01:42 GMT

... see also http://www.supermicro.com/support/resources/gpu/, and http://www.supermicro.com/products/superblade/module/SBI-7127RG-E.cfm. From that it looks like you'd need the "-E" version

Belinda,

jimdempseyatthecove — Wed, 06 Nov 2013 02:05:28 GMT

Belinda,

I've installed MPSS for Windows on a Windows 7 Pro x64 system. I can get the two Xeon Phi 5510P cards up and running, firmware updates, cards boot, micinfo shows both cards, cards ready, I can ping both cards. MicSmc-gui.exe shows both cards ideling along, ...

I can compile my first project selected in the tutorials coi folder "hello_world". The project compiles and runs fine up to the point where it wants to launch the native side app hello_world_sink_mic, which is not built by the solution (as separate project).

Launching an Intel Parallel Studio XE 2013 command prompt for use with Visual Studio 2012, and navigating to the demo folder (under C:\Program Files\..." and issuing

icl -Qmic hello_world_sink.cpp -o hello_world_sink_mic

I receive an error stating stdio.h cannot be found, check MPSS environment variables.

If I remove the -Qmic (not what I want as this compiles as host app) I get an error writing the .obj file (due to folder off C:\Program Files\..."

If I copy the MPSS folder elsewhere (not under protected folder)...

compile with -Qmic fails with stdio.h not found

compile without -Qmic succeeds.

IOW -Qmic expects a different set of environment variables (with respect to INCLUDE)

How do I properly set the environment variable(s) for compiling the coprocessor side (-Qmic) of the demo programs under Windows?

Jim Dempsey

We installed the 3120A card

Alex_R_ — Wed, 12 Feb 2014 15:21:16 GMT

We installed the 3120A card in Windows7 box, the card is blinking blue. Installed MPSS 3.1.2. The card is not displayed in Device Manager (?). "micctrl -s" command results in error:

Error manipulating coprocessor: Intel(R) Xeon Phi(TM) coprocessor driver is not loaded or you have insufficient access

Hi Alex, can you check the

BelindaLiviero — Wed, 12 Feb 2014 17:30:53 GMT

Hi Alex, can you check the following (this is based on similar forum posts reported earlier this month)

1. physically inspect the card installation - is the card inserted properly, and are all power connectors on the card plugged in properly

2. If you are working with a numa machine, where some of the PCI slots are enabled or disabled, you need to make sure the coprocessor is installed on an enabled PCI slot.

Thank you, BELINDA! Switching

Alex_R_ — Wed, 12 Feb 2014 22:05:05 GMT

Thank you, BELINDA! Switching to another PCI slot worked!

Thanks for this. Really

Jess — Wed, 19 Mar 2014 14:08:58 GMT

Thanks for this. Really helpful. Unfortunately the problem we are seeing is the driver crashing when the machine boots. I've attached the stacktrace from the logs. This is from the latest MPSS on RHEL 6.

The other problem that we are

Jess — Wed, 19 Mar 2014 17:01:56 GMT

The other problem that we are having is actually with NFS exporting GPFS shares. Since the GPFS drivers and client software does not support MIC, we NFS export the drives from each host to its MICs. It is very unreliable though, and so we find that the MICs will not mount the drives sometimes, citing "stale NFS filehandle" as the cause, which is untrue. It seems related to the order of the mounts in /etc/fstab, as the first one will mount and the second won't.

Ideally we'd like GPFS binaries for MIC, as this is a kludge anyway. In the current state we can't really say to users that the systems are ready to use.

(We'd also really like MPSS to support OFED 2.x, since that is what the rest of the machine is using. Only the nodes with MICs in are on 1.5.x, and that's entirely due to needing it to support the IPoIB software provided with MPSS.)

Hi Zaniyah,

BelindaLiviero — Tue, 25 Mar 2014 20:53:39 GMT

Hi Zaniyah,

is your mic stracktrace from a consistently failing coprocessor (and can you send the tarball that gets created by the micdebug.sh script?)

As for GPFS -- are you in a position where you can ask IBM for their plans to support GPFS with Intel Xeon Phi Coprocessors ? You can even tell them that there is now a Lustre client (was recently released, we'll provide a writeup on that soon).

I will make sure to pass on your comments about wanting OFED 2.x support.

Hi Belinda,

Virginie_Favrat — Thu, 27 Mar 2014 07:30:40 GMT

Hi Belinda,

I have installed mpss-3.2 for first use of Xeon Phi, but I can not know which version of Flash is installed and I can not update it.

Neither can I start mpss service.

Here are several results of commands :

sudo micinfo
MicInfo Utility Log
Copyright 2011-2013 Intel Corporation All Rights Reserved.

Created Thu Mar 27 08:18:47 2014


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-431.5.1.el6.x86_64
		Driver Version		: 3.2-1
		MPSS Version		: 3.2
		Host Physical Memory	: 65918 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : NotAvailable
		SMC Firmware Version	 : NotAvailable
		SMC Boot Loader Version	 : NotAvailable
		uOS Version 		 : NotAvailable
		Device Serial Number 	 : NotAvailable
...

sudo micflash -update -device all -smcbootloader
No image path specified - Searching: /usr/share/mpss/flash
mic0: No valid image found

micsmc
DEBUG: ***** MicSettings(parent)::fileName():  "/home/vivi/.config/Intel Corp/MicSmcGUI.ini" 
DEBUG: ***** SessionSettings(parent)::fileName():  "/home/vivi/.config/Intel Corp/MicSmcGUI.ini" 
Avertissement�: mic0 : Connexion avec le p�riph�rique perdue !
Infos Web mic0 : Connexion avec le p�riph�rique r�tablie.
Avertissement�: mic0 : Connexion avec le p�riph�rique perdue !
Infos Web mic0 : Connexion avec le p�riph�rique r�tablie.
Avertissement�: mic0 : Connexion avec le p�riph�rique perdue !

sudo service mpss start
Starting Intel(R) MPSS:                                    [ÉCHOUÉ]

May you help me to find what is the trouble ?

Thanks in advance.

Virginie

I was thinking that there was

Virginie_Favrat — Thu, 27 Mar 2014 10:32:48 GMT

I was thinking that there was a probleme beacause of the kenrnel version.

So I restart with the original kernel version and reinstall MPSS. But here is the result of micctrl --initdefaults :

micctrl --initdefaults
micctrl(segv_handler+0x18) [0x4070c8]
/lib64/libpthread.so.0() [0x34cb40f710]
/usr/lib64/libmpssconfig.so.0.0.1(_add_miclist_not_present+0xb8) [0x7f5c84a35b98]
/usr/lib64/libmpssconfig.so.0.0.1(mpss_get_miclist+0x4d) [0x7f5c84a35e7d]
micctrl(create_miclist+0x1cd) [0x42123d]
micctrl(parse_config_args+0x370) [0x40db60]
micctrl(main+0x236) [0x40df56]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x34cb01ed1d]
micctrl() [0x406d29]

Hi Virginie, can you send us

BelindaLiviero — Thu, 27 Mar 2014 21:18:25 GMT

Hi Virginie, can you send us the output of /usr/bin/micdebug.sh (just attach the tarball to this thread). That would be most helpful.

Hi Belinda !

Virginie_Favrat — Fri, 28 Mar 2014 10:01:00 GMT

Hi Belinda !

I send you the last one, but if you want I have 2 other ones (made on Tuesday and Wednesday).

The usual commands I use to try as root :

lspci | grep proc
setenforce 0
modprobe mic
service mpss start

Thanks in advance.