mpss restart fails before booting completes

[William_Howell]

After recently upgrading to mpss 3.6, I've encountered the following behavior and hope someone has some insights or can help debug. When either starting or restarting the mpss service, the procedure fails. Immediately following the service start, the cards show:

[root@host ~]# micctrl -s
mic0: boot failed
mic1: boot failed
mic2: boot failed
mic3: boot failed

However, after several (on the order of 10) seconds, the cards appear to successfully complete boot:

[root@host ~]# micctrl -s
mic0: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic1: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic2: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic3: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)

I've tried watching boot over serial connection to the mics with no useful information, and also don't see anything obvious in the logs on either the host or the mics. Once the μOS boots, all appears well with regards to connectivity and configuration. Does anyone have any further debugging steps we could try?

Thanks in advance 

Will

[William_Howell]

I should mention, the timeout is visible when rebooting any of the mics on the host. The boot initially appears to fail, but after a short time the mic os comes up clean.  It appears the boot is taking longer then the monitoring allows for.

Interestingly, this does not occur on another node with the exact same configuration.