SMC bootloader flash fails

john_p_2 · ‎10-24-2014

We have an Intel Xeon Phi 3120A for which we can't update the SMC Flash and SMC Bootloader flash. Any suggestions are greatly appreciated !

First of all I'd like to mention that the blue LED is not blinking at any time during the procedures described below. It is always solid blue. Not sure if it's supposed to be like that.

When we first installed the board in the computer it had flash version 3.72 so we ran a flash update on it to bring it up to the current version (3.90).

We ran:

micflash -update -v -device all -log micflash.log

and the log file shows the following:

Fri Oct 24 16:48:27 2014: micflash: Tool version: 3.4-1
Fri Oct 24 16:48:27 2014: mic0: Flash image: /usr/share/mpss/flash/EXT_HP2_B1_0390-02.rom.smc
Fri Oct 24 16:48:27 2014: mic0: Flash update started...
Fri Oct 24 16:48:29 2014: mic0: Flash update done
Fri Oct 24 16:48:29 2014: mic0: SMC update started...
Fri Oct 24 16:48:29 2014: micflash: mic0: SMC update failed: SMC buffer size exceeded (0x1)

Fri Oct 24 16:48:29 2014: mic0: SMC update: 0% was completed
Fri Oct 24 16:48:30 2014: mic0: Transitioning to ready state...
Fri Oct 24 16:48:39 2014: mic0: Exit status: 10

Then, we followed instructions provided in other posts and we power cycled the computer and then, immediately after reboot, without starting the mpss service, we ran the following command:

micflash -update -v -smcbootloader -device all

but unfortunately we get the same message:

Fri Oct 24 16:53:55 2014: micflash: Tool version: 3.4-1
Fri Oct 24 16:53:55 2014: mic0: Flash image: /usr/share/mpss/flash/EXT_HP2_B1_0390-02.rom.smc
Fri Oct 24 16:53:55 2014: mic0: SMC boot-loader image: /usr/share/mpss/flash/EXT_HP2_SMC_Bootloader_1_8_4326.css_ab
Fri Oct 24 16:53:55 2014: mic0: SMC boot-loader update started...
Fri Oct 24 16:53:55 2014: micflash: mic0: SMC update failed: SMC buffer size exceeded (0x1)

Fri Oct 24 16:53:55 2014: mic0: SMC boot-loader update: 0% was completed
Fri Oct 24 16:53:56 2014: mic0: Transitioning to ready state...
Fri Oct 24 16:54:05 2014: mic0: Exit status: 8

The result of micinfo is this:

    System Info
       HOST OS           : Linux
       OS Version       : 3.10.0-123.el7.x86_64
       Driver Version       : 3.4-1
       MPSS Version       : 3.4

Host Physical Memory : 32682 MB

Device No: 0, Device Name: mic0

   Version
       Flash Version        : NotAvailable
       SMC Firmware Version   : NotAvailable
       SMC Boot Loader Version   : NotAvailable
       uOS Version        : NotAvailable
       Device Serial Number    : NotAvailable

   Board
       Vendor ID        : 0x8086
       Device ID        : 0x225d
       Subsystem ID        : 0x2500
       Coprocessor Stepping ID   : 3
       PCIe Width        : x16
       PCIe Speed        : 5 GT/s
       PCIe Max payload size   : 128 bytes
       PCIe Max read req size   : 512 bytes
       Coprocessor Model   : 0x01
       Coprocessor Model Ext   : 0x00
       Coprocessor Type   : 0x00
       Coprocessor Family   : 0x0b
       Coprocessor Family Ext   : 0x00
       Coprocessor Stepping    : B1
       Board SKU        : B1QS-3115A
       ECC Mode        : NotAvailable
       SMC HW Revision    : NotAvailable

   Cores
       Total No of Active Cores : NotAvailable
       Voltage        : NotAvailable
       Frequency        : NotAvailable

   Thermal
       Fan Speed Control    : NotAvailable
       Fan RPM        : NotAvailable
       Fan PWM        : NotAvailable
       Die Temp       : NotAvailable

   GDDR
       GDDR Vendor       : NotAvailable
       GDDR Version       : NotAvailable
       GDDR Density       : NotAvailable
       GDDR Size       : NotAvailable
       GDDR Technology       : NotAvailable
       GDDR Speed       : NotAvailable
       GDDR Frequency       : NotAvailable
       GDDR Voltage       : NotAvailable

Thank you !

Frances_R_Intel · ‎10-27-2014

In looking at other posts about flash/smc update, you have probably seen that flash versions prior to 3.75 are a problem. You said that the flash version was 3.72. Did you find that out from running micinfo prior to the update attempt? What does 'micctrl -s' say - has the card returned to the ready state?

If the card will not come back to the ready state, there is little I can recommend other than for you to contact your supplier and see if there is anything they can do for you.

The micinfo command will not show all the information unless the card is booted. If the card has come back to the ready state, you can try booting it and rerunning micinfo. If it does boot, could you post the micinfo information you get at that point? It would be good to know what the card thinks its flash and smc version are now. It may think it has an up-to-date flash and an out-of-date smc, which would be a bad thing.

At this point, I don't believe I can get my hands on an early enough version of the MPSS to guarantee that you will be able to update the flash and smc. I didn't have much luck when I looked for one a year ago. The earliest version that is still publicly available is in the MPSS 2.1. However, even this version wants the flash to be 3.75 or later.

So - I guess my recommendation would be -

If the card will not come back to the ready state, talk to the folks who supplied the card.

If it will come back to the ready state, see if it will boot and provide the full micinfo information.

And in the meantime, I will seek advice from those more knowledgeable to see if there is anything that can be done to get this card to a known state and update it.

john_p_2 · ‎10-27-2014

Thank you, Frances !

Per your instructions, micctrl -s indicated that it was ready, so I've tried to boot and I've got the following:

micctrl -b -vv
   [Info] mic0: [Parse] /etc/mpss/mic0.conf
   [Info] mic0: [Parse] Configuration version 1.1
   [Info] mic0: [Parse] /etc/mpss/default.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/sep.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/itt.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/amplxe.conf
   [Info] mic0: Command line: quiet root=ramfs console=hvc0 cgroup_disable=memory highres=off
          mic0: booting /usr/share/mpss/boot/bzImage-knightscorner
[Error] mic0 failed to boot: card state online

Then, I've tried to reboot it with the following 2 commands:

micctrl -r -vv
mic0: resetting

micctrl -b -vv
   [Info] mic0: [Parse] /etc/mpss/mic0.conf
   [Info] mic0: [Parse] Configuration version 1.1
   [Info] mic0: [Parse] /etc/mpss/default.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/sep.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/itt.conf
   [Info] mic0: [Parse] /etc/mpss/conf.d/amplxe.conf
   [Info] mic0: Command line: quiet root=ramfs console=hvc0 cgroup_disable=memory highres=off
          mic0: booting /usr/share/mpss/boot/bzImage-knightscorner

At this point it seems to be booted (because it doesn't say that it failed to boot) so I run micinfo but it still doesn't show the flash version:

micinfo
MicInfo Utility Log
Created Mon Oct 27 15:31:19 2014

   System Info
       HOST OS           : Linux
       OS Version       : 3.10.0-123.el7.x86_64
       Driver Version       : 3.4-1
       MPSS Version       : 3.4

Host Physical Memory : 32682 MB

Device No: 0, Device Name: mic0

   Version
       Flash Version        : NotAvailable
       SMC Firmware Version   : NotAvailable
       SMC Boot Loader Version   : NotAvailable
       uOS Version        : NotAvailable
       Device Serial Number    : NotAvailable

   Board
       Vendor ID        : 0x8086
       Device ID        : 0x225d
       Subsystem ID        : 0x2500
       Coprocessor Stepping ID   : 3
       PCIe Width        : x16
       PCIe Speed        : 5 GT/s
       PCIe Max payload size   : 128 bytes
       PCIe Max read req size   : 512 bytes
       Coprocessor Model   : 0x01
       Coprocessor Model Ext   : 0x00
       Coprocessor Type   : 0x00
       Coprocessor Family   : 0x0b
       Coprocessor Family Ext   : 0x00
       Coprocessor Stepping    : B1
       Board SKU        : B1QS-3115A
       ECC Mode        : NotAvailable
       SMC HW Revision    : NotAvailable

   Cores
       Total No of Active Cores : NotAvailable
       Voltage        : NotAvailable
       Frequency        : NotAvailable

   Thermal
       Fan Speed Control    : NotAvailable
       Fan RPM        : NotAvailable
       Fan PWM        : NotAvailable
       Die Temp       : NotAvailable

   GDDR
       GDDR Vendor       : NotAvailable
       GDDR Version       : NotAvailable
       GDDR Density       : NotAvailable
       GDDR Size       : NotAvailable
       GDDR Technology       : NotAvailable
       GDDR Speed       : NotAvailable
       GDDR Frequency       : NotAvailable
       GDDR Voltage       : NotAvailable

Frances_R_Intel · ‎10-27-2014

It can take a little while between when you see:

mic0: booting /usr/share/mpss/boot/bzImage-knightscorner

and when the coprocessor is actually booted. (If you add a '-w' to the micctrl command, the command will not terminate until the boot processes completes.) So, I am not so sure that your card was booted when you ran the micinfo command.

At this point, the best thing might be to put the testing aside until I can get more information.

Frances_R_Intel · ‎11-21-2014

For all those who have a B1 card with a flash earlier than 375, we now have a 375 flash that has been tested and can be used to move from your present flash to 375 and from there to the current 390 flash in MPSS 3.3.x and 3.4.x. These instructions are ONLY for B1 cards and ONLY if your flash is (or was when you started trying to update the MPSS) pre-375. (C0 cards shouldn't have this problem and if your flash is 375 or later, you can follow the directions in the readme.txt exactly as given.) Some people have been successful just going straight from the a pre-375 flash to the the current flash, but not everyone, as shown by your problem, John.

If you have already attempted to update straight to the current version of flash and, when you run micinfo as root, your flash and smc bootloader versions match the version numbers given in the readme.txt, DO NOT follow the instructions below.

If your card is NOT a B1 stepping, as shown by micinfo, DO NOT follow the instructions below.

Follow the directions in the readme.txt for the version of MPSS you are upgrading to, until you get to the line telling you to update the flash and at that point STOP.

Download the attached file and unzip it in a directory on your host system - for this example, I will use /tmp/oldflash/.

Execute (substituting your directory name, if different):

micflash -update /tmp/oldflash -device all smcbootloader

Reboot your host system
Return to the directions in the readme.txt and continue from where you stopped, updating your flash to the current version.