mpssd segfaults with NFS root after upgrading to mpss_gold_update_3-2.1.6720

[Pierre-Emmanuel_L_]

Hello,

I experience the following issue with mpss_gold_update_3-2.1.6720 on a Scientific Linux 6.3. When the root device is a NFS export, mpssd fails with this message :

[plain]

Tue May 14 17:28:23 2013: <<<<<<<< mpssd: segmentation violation - dumping stack >>>>>>>>
Tue May 14 17:28:23 2013: /usr/sbin/mpssd(segv_handler+0x34) [0x402ca3]
Tue May 14 17:28:23 2013: /lib64/libpthread.so.0(+0xf500) [0x7f73ab217500]
Tue May 14 17:28:23 2013: /lib64/libc.so.6(+0x13259f) [0x7f73aab9559f]
Tue May 14 17:28:23 2013: /usr/lib64/libmpssconfig.so.0(+0x7487) [0x7f73aadfd487]
Tue May 14 17:28:23 2013: /usr/lib64/libmpssconfig.so.0(mpss_set_cmdline+0xbe) [0x7f73aadfd6f0]
Tue May 14 17:28:23 2013: /usr/sbin/mpssd(start_daemon+0x268) [0x4040fb]
Tue May 14 17:28:23 2013: /usr/sbin/mpssd(main+0x160) [0x402674]
Tue May 14 17:28:23 2013: /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f73aaa81cdd]
Tue May 14 17:28:23 2013: /usr/sbin/mpssd() [0x402459]
Tue May 14 17:28:23 2013: <<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

[/plain]

The configuration files were created with the following commands: micctrl --cleanconfig; micctrl --initdefaults; micctrl --rootdev=NFS --target=/srv/nfs/mic0 .

The same settings used to work using KNC_gold_update_2-2.1.5889-16.

Any hints? Thanks,

Pierre-Emmanuel

[Frances_R_Intel]

But the system boots fine if you use the default ramfs?

According to the MPSS_Boot_Config_Guide.pdf, the command should be:

micctrl --rootdev=NFS --target=<location> --server=<name> -d –c [mic card list]

but according to 'micctrl --help', your 'micctrl --rootdev=NFS --target=/srv/nfs/mic0' should be sufficient. I'll look around and see if there are known problems but in the meantime, could you check to see that your root filesystem is up-to-date with the version that was delivered with the new MPSS, then try setting the root device by editting the mic0.conf file and running 'micctrl --resetconfig' to see if you get a different result.

[Pierre-Emmanuel_L_]

Hello,

But the system boots fine if you use the default ramfs?

Yes, it does.

According to the MPSS_Boot_Config_Guide.pdf, the command should be: micctrl --rootdev=NFS --target=<location> --server=<name> -d –c [mic card list] but according to 'micctrl --help', your 'micctrl --rootdev=NFS --target=/srv/nfs/mic0' should be sufficient.

The --server option is ignored with this message 'mic0: Netork static pair ignoring server parameter'. With the '-d' option, the directory /srv/nfs/mic0 is deleted and a new one is created but mpssd still fails.

could you check to see that your root filesystem is up-to-date with the version that was delivered with the new MPSS

Yes, it is. After updating to the new MPSS, the problem appeared so I removed everything (intel-mic*.rpm, /srv/nfs/mic0, /opt/intel/mic, /etc/sysconfig/mic) and then I reinstalled everything.

try setting the root device by editting the mic0.conf file and running 'micctrl --resetconfig' to see if you get a different result.

After 'micctrl --resetconfig' I get the same result.

The mic0.conf file seems fine:

[plain]Version 0 6

# Include configuration common to all MIC cards
Include default.conf

# Include all additional functionality configuration files by default
Include "conf.d/*.conf"

# Startup services from base package
Service fileperms 1 99 on
Service network 21 79 on
Service sshd 80 20 on
Service pm 90 10 on
Service blcr 95 5 off
Service mictune 99 1 on

# Unique per card files for embedded Linux file system
MicDir /opt/intel/mic/filesystem/mic0 /opt/intel/mic/filesystem/mic0.filelist

# Hostname to assign to MIC card
Hostname "phiscale-mic0"

MacAddrs Serial

Network StaticPair 172.31.1.1 172.31.1.254 yes 24

# MIC OS Verbose messages to console
VerboseLogging Disabled

# MIC OS image
OSimage /lib/firmware/mic/uos.img

# Boot MIC card when MPSS stack is started
BootOnStart Enabled

# MIC User Authentication
# This parameter depricates the UserIDs paramter.  If both are
# specified UserAuthentication will override UserIDs
UserAuthentication Local 500 65000

# Control card power state setting
# cpufreq: P state
# corec6: Core C6 state
# pc3: Package C3 state
# pc6: Package C6 state
PowerManagement "cpufreq_on;corec6_off;pc3_on;pc6_on"

# Root device for MIC card
RootDevice NFS 172.31.1.254:/srv/nfs/mic0

[/plain]

According to the error message, the problem seems to be in the function 'mpss_set_cmdline' from /usr/lib64/libmpssconfig.so.0. I suspect a bug either during configuration file parsing or command line string creation.

Thank you.

[Frances_R_Intel]

Could you submit an issue in Premier (https://premier.intel.com/) against Intel(R) MPSS for Linux*? I will do it if you cannot but if you do it, you will get notified when a fix is available. And in the meantime, try using a bridge instead of a static pair for the network.

[bash]

micctrl --addbridge=br0 --type=internal --ip=172.31.1.254
micctrl --network=static --bridge=br0 --ip=172.31.1.1

[/bash]

[Pierre-Emmanuel_L_]

Hello,

using a bridge instead of a static pair does not fix the issue.

Please, submit the issue for me as I do not have access to Premier.

Thank you for your help.

[Pierre-Emmanuel_L_]

Hello,

friday afternoon I realized that mpssd and libmpssconfig are open source. I looked at the source code and found the bug.

mpssd calls the mpss_set_cmdline function from libmpssconfig. This function calls get_rootdev. In the case of a NFS root, get_rootdev accesses to mic->config.net.micMac but this field is not initialized by the config_mac function when MacAddrs is set to Random or Serial (default value).

I modified mic0.conf to use static mac addresses and now the card boots with the NFS root.

[Frances_R_Intel]

Thanks for going to this effort. I added your comment to the premier issue - so the developers now have your fix.

[Frances_R_Intel]

This issue was fixed in MPSS 3.1.