Xeon PHI MIA after flash update 3.8.2

[AndrewC]

I have been trying to get the Xeon Phi in my Microway Windows 7 SP1 Workstation to work with MKL Automatic OffLoad

As part of that process I upgraded to MPSS 3.8 and re-flashed - the process went without error. Of course I rebooted.

However the PHI card seems now non-functional , and crashes MKL when I call mkl_mic_enable()

Before upgrade

>micinfo
MicInfo Utility Log
Copyright 2011-2013 Intel Corporation All Rights Reserved.

Created Wed Jun 14 11:34:46 2017

        System Info
                HOST OS                 : Windows
                OS Version              : Microsoft Windows 7 Professi
                Driver Version          : 3.3.30726.0
                MPSS Version            : 3.3.30726.0
                Host Physical Memory    : 32709 MB

Device No: 0, Device Name: mic0

        Version
                Flash Version            : 2.1.02.0390
                SMC Firmware Version     : 1.16.5078
                SMC Boot Loader Version  : 1.8.4326
                uOS Version              : 2.6.38.8+mpss3.3
                Device Serial Number     : ADKC32800563

        Board
                Vendor ID                : 0x8086
                Device ID                : 0x225d
                Subsystem ID             : 0x3608
                Coprocessor Stepping ID  : 2
                PCIe Width               : x16
                PCIe Speed               : 5 GT/s
                PCIe Max payload size    : 256 bytes
                PCIe Max read req size   : 512 bytes
                Coprocessor Model        : 0x01
                Coprocessor Model Ext    : 0x00
                Coprocessor Type         : 0x00
                Coprocessor Family       : 0x0b
                Coprocessor Family Ext   : 0x00
                Coprocessor Stepping     : C0
                Board SKU                : C0PRQ-3120/3140 P/A
                ECC Mode                 : Enabled
                SMC HW Revision          : Product 300W Active CS

        Cores
                Total No of Active Cores : 57
                Voltage                  : 1039000 uV
                Frequency                : 1100000 kHz

After Upgrade

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC>micinfo
MicInfo Utility Log
Created Fri Jun 16 08:42:07 2017

        System Info
                HOST OS                 : Windows
                OS Version              : Microsoft Windows 7 Professional
                Driver Version          : 3.8.2.4191
                MPSS Version            : 3.8.2.4191
                Host Physical Memory    : 32709 MB

Device No: 0, Device Name: mic0

        Version
                Flash Version            : NotAvailable
                SMC Firmware Version     : NotAvailable
                SMC Boot Loader Version  : NotAvailable
                Coprocessor OS Version   : NotAvailable
                Device Serial Number     : NotAvailable

        Board
                Vendor ID                : 0x8086
                Device ID                : 0x225d
                Subsystem ID             : 0x3608
                Coprocessor Stepping ID  : 2
                PCIe Width               : x16
                PCIe Speed               : 5 GT/s
                PCIe Max payload size    : 256 bytes
                PCIe Max read req size   : 512 bytes
                Coprocessor Model        : 0x01
                Coprocessor Model Ext    : 0x00
                Coprocessor Type         : 0x00
                Coprocessor Family       : 0x0b
                Coprocessor Family Ext   : 0x00
                Coprocessor Stepping     : C0
                Board SKU                : C0PRQ-3120/3140 P/A
                ECC Mode                 : NotAvailable
                SMC HW Revision          : NotAvailable

        Cores
                Total No of Active Cores : NotAvailable
                Voltage                  : NotAvailable
                Frequency                : NotAvailable

        Thermal
                Fan Speed Control        : NotAvailable
                Fan RPM                  : NotAvailable
                Fan PWM                  : NotAvailable
                Die Temp                 : NotAvailable

        GDDR
                GDDR Vendor              : NotAvailable
                GDDR Version             : NotAvailable
                GDDR Density             : NotAvailable
                GDDR Size                : NotAvailable
                GDDR Technology          : NotAvailable
                GDDR Speed               : NotAvailable
                GDDR Frequency           : NotAvailable
                GDDR Voltage             : NotAvailable

[JJK]

From reading the micinfo output I'd say the board is still there but for some reason the mpss driver is not liking it much. Is the mpss service running on Windows? Did it report anything at startup?  Also, did you do a cold restart of the box after the micflash update (if not, try that first) ?

[AndrewC]

I reverted to the last version supposedly supported on Windows 7 SP1  - 3.7.2 and reflashed.

Still no luck , the PHI is in an unusable state. I have tried shutting down the machine and restarting.

Where would I find any errors reported on MPSS startup on Windows. I actually don't see a "service" called MPSS - or any Xeon PHI related service apart from an "erorr reporting" service MicRAS that fails to start anyway.

[AndrewC]

OK, what does this mean?

Error starting device.
Input string was not in a correct format. while parsing file list, line 34 of mic0_0
   at MpssLib.cpio.CreateCPIOFromFilelist(String RootPath, String RootName, String Filelist, Boolean Recreate)
   at MpssLib.cpio.CreateRootFilesystem(MIC current, String InstallPath)
   at MpssLib.MIC.GenFS()
   at MpssLib.Host.GenFS(Int32 node)
   at MpssLib.Host.BootNode(Int32 node)
   at MpssLib.Host.BootAll(List`1 BootList)

[AndrewC]

So when looking at the various files

C:\Program Files\Intel\MPSS\filesystem\mic0.filelist has the line 34

file /root/.ssh/authorized_keys root/.ssh/authorized_keys 600 0 0dir /home/acu 750 502 502

This looks messed up.

So I fixed the format of the line as it looks like a <CR> is needed - and  the PHI started up.

file /root/.ssh/authorized_keys root/.ssh/authorized_keys 600 0 0

dir /home/acu 750 502 502

So it appears the installer did something wrong and changed these files.

[Loc_N_Intel]

Hi,

It look like you tried to upgrade from MPSS 3.3 to 3.8 . There are many changes introduced in these versions, and that may cause something wrong when upgrading.

When you fail to upgrade MPSS, here is one way to handle it:

1. Uninstall the current Windows cross-SDK  (mpss-essentials) in the "Program and Features" window.(from the Control Panel).

2. Uninstall the current Windows MPSS software stack.

3. Delete the folder c:\Program Files\Intel\MPSS completely.

Then you start install the new MPSS stack as usual.

[jimdempseyatthecove]

You might want to add a reboot after 3.

[AndrewC]

That looks like that might have been the best idea. I did "uninstall" but apparently that is not enough and I should have deleted the complete c:\Program Files\Intel\MPSS directory. I was concerned about losing access to the PHI due to some settings made by the OEM.

[Loc_N_Intel]

Yes, a reboot always helps.

Note that when you delete the folder c:\Program Files\Intel\MPSS, all the configuration files that you created previously for the coprocessor’s file system are gone (including root and user credentials on the coprocessor). You have to re-create them if you need to.