miccheck -ssh fail unexpected

[Pierpaolo_M_]

Hi,

i am using mpss 3.2.1 and i have configured an internal bridge in this way:

micctrl --addbridge=br0 --type=internal --ip=172.31.1.254
micctrl --network=static --bridge=br0 --ip=172.31.1.1

Everything went fine and after restarting mpssd i am able to login on both mic cards.

But when i try to check using miccheck tool, i obtain this failure:

miccheck --ssh --ping
MicCheck 3.2.1-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
Executing default tests for device: 1
  Test 7 (mic1): Check device is in online state and its postcode is FF ... pass
  Test 8 (mic1): Check ras daemon is available in device ... pass
  Test 9 (mic1): Check running flash version is correct ... pass
Executing optional tests for device: 0
  Test 10 (mic0): Check device can be pinged over its network interface ... pass
  Test 11 (mic0): Check device can be accessed through ssh ... fail
    interface mic0 could not be accessed through ssh
Executing optional tests for device: 1
  Test 12 (mic1): Check device can be pinged over its network interface ... pass
  Test 13 (mic1): Check device can be accessed through ssh ... fail
    interface mic1 could not be accessed through ssh

Status: FAIL
Failure: An optional device test failed

Is it normal?

As i said before i am able to login on both cards and i am also able to login successfully from on card to the other. So i don't understand why i receive this failure.

Thanks in advance

[Frances_R_Intel]

Did miccheck pass the ssh test before you set up the bridge? You say you can still log into the cards; are you using ssh to log in? What about your authentication methods - are you using LDAP or NIS?

Actually, let me back up a bit. Usually, when the miccheck fails the ssh test it is because the ssh keys either did not get properly generated or did not get copied over to the home directory on the coprocessor. So, let's go back to the question: are you using ssh to log in? If not, can you?

[Pierpaolo_M_]

Hi,

miccheck didn't pass the ssh before also before i set up the bridge. To log into the cards i use simply ssh (no LDAP, no NIS). After micctrl --initdefaults i generated keys with command: ssh-keygen -t dsa and everything went fine. Only the first time i had to say 'yes' because it was a new connection, but after that i login on both cards without using any password(user root was added using instruction in readme.txt):

[pierpaolo@imip1 ~]$ ssh mic0
pierpaolo@imip1-mic0:~$ exit
logout
Connection to mic0 closed.
[pierpaolo@imip1 ~]$ ssh mic1
pierpaolo@imip1-mic1:~$ exit
logout
Connection to mic1 closed.
[pierpaolo@imip1 ~]$

[root@imip1 ~]# ssh mic0
[root@imip1-mic0 ~]# exit
logout
Connection to mic0 closed.
[root@imip1 ~]# ssh mic1
[root@imip1-mic1 ~]# exit
logout
Connection to mic1 closed.
[root@imip1 ~]#

I have also successfully configured NFS (if you have a firewall for CentOS 6.5 procedure is a little more complicated then instruction written in MPSS 3.2.1 user guide) and now i see /home from host on both cards in position /home, but miccheck -ssh continue to fail. I hope that my answer is compete.

Pierpaolo

[Frances_R_Intel]

I dug around in the source code for miccheck and found the exact command that it runs:

/usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes -oStrictHostKeyChecking=no mic0 echo hello

Could you see if this fails for you? Then try omitting options to see if one of those options is causing the error. Maybe that will point us in the right direction for finding the problem.

[Pierpaolo_M_]

Hi,

i don't know if it is good but i tried that command and it works fine also without options. But miccheck continues to fail.

pierpaolo@imip1 ~]$ miccheck --ssh 
MicCheck 3.2.1-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
Executing default tests for device: 1
  Test 7 (mic1): Check device is in online state and its postcode is FF ... pass
  Test 8 (mic1): Check ras daemon is available in device ... pass
  Test 9 (mic1): Check running flash version is correct ... pass
Executing optional tests for device: 0
  Test 10 (mic0): Check device can be accessed through ssh ... fail
    interface mic0 could not be accessed through ssh
Executing optional tests for device: 1
  Test 11 (mic1): Check device can be accessed through ssh ... fail
    interface mic1 could not be accessed through ssh

Status: FAIL
Failure: An optional device test failed
[pierpaolo@imip1 ~]$ /usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes -oStrictHostKeyChecking=no mic0 echo hello
hello
[pierpaolo@imip1 ~]$ /usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes  mic0 echo hello
hello
[pierpaolo@imip1 ~]$ /usr/bin/ssh -oConnectTimeout=3 -oStrictHostKeyChecking=no mic0 echo hello
hello
[pierpaolo@imip1 ~]$ /usr/bin/ssh -oBatchMode=yes -oStrictHostKeyChecking=no mic0 echo hello
hello
[pierpaolo@imip1 ~]$ /usr/bin/ssh mic0 echo hello
hello
[pierpaolo@imip1 ~]$

Thanks again for your support

[Taras_Shapovalov]

Hi Frances,

I can confirm that this check does not work (and have never worked for us since the first version of MPSS3). We use bridges.

I also disappointed that in MPSS3 the developers removed almost half of the healthchecks from miccheck utility. Basically only 5 remains (device related) where 1 does not work.

Best regards,

Taras

[Frances_R_Intel]

It frustrates me when the code says something should be working and yet it isn't. Since I can't reproduce the problem on the machines I have access to, could one of you do me a favor and check the exit status for the command I gave you. I am wondering if, even though the command is able to run echo on the coprocessor and print out hello, it is returning a status other than 0. Meanwhile I will see if I can find anything in the code that could be getting set then causes the command to fail.

@Taras - I hadn't noticed that some of the miccheck tests have been made optional and disabled by default. I can put in a ticket to the MPSS developers and ask them to either re-enable the options that have been disabled by default then add an option that will let someone choose to generate a short report or leave the tests disabled by default then add a single option that will enable all options. Do you think that will help?

[Pierpaolo_M_]

Hi,

sorry but in my case, i have always exit status 0:

[pierpaolo@imip1 Lesson1]$ miccheck --ssh
MicCheck 3.2.1-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
Executing default tests for device: 1
  Test 7 (mic1): Check device is in online state and its postcode is FF ... pass
  Test 8 (mic1): Check ras daemon is available in device ... pass
  Test 9 (mic1): Check running flash version is correct ... pass
Executing optional tests for device: 0
  Test 10 (mic0): Check device can be accessed through ssh ... fail
    interface mic0 could not be accessed through ssh
Executing optional tests for device: 1
  Test 11 (mic1): Check device can be accessed through ssh ... fail
    interface mic1 could not be accessed through ssh

Status: FAIL
Failure: An optional device test failed
[pierpaolo@imip1 Lesson1]$ 
[pierpaolo@imip1 Lesson1]$ echo $?
1
[pierpaolo@imip1 Lesson1]$ /usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes -oStrictHostKeyChecking=no mic0 echo hello
hello
[pierpaolo@imip1 Lesson1]$ echo $?
0
[pierpaolo@imip1 Lesson1]$ /usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes  mic0 echo hello
hello
[pierpaolo@imip1 Lesson1]$ echo $?
0
[pierpaolo@imip1 Lesson1]$ /usr/bin/ssh -oConnectTimeout=3 -oBatchMode=yes  mic0 echo hello
hello
[pierpaolo@imip1 Lesson1]$ echo $?
0
[pierpaolo@imip1 Lesson1]$ /usr/bin/ssh -oBatchMode=yes -oStrictHostKeyChecking=no mic0 echo hello
hello
[pierpaolo@imip1 Lesson1]$ echo $?
0
[pierpaolo@imip1 Lesson1]$ /usr/bin/ssh mic0 echo hello
hello
[pierpaolo@imip1 Lesson1]$ echo $?
0
[pierpaolo@imip1 Lesson1]$

Best Regards 

Pierpaolo

[Frances_R_Intel]

I am conceding defeat and have submitted a bug report to have the developers look at this. I'll let you know what they say.

Also, I submitted a feature request for Taras to get the missing tests returned to miccheck even if only as an option.

Finally, I would like to recommend that you move to MPSS 3.2.3 because it has a number of important fixes, although I don't believe there is anything in there to address this problem.

[Frances_R_Intel]

The developers have not been able to reproduce the bug, but then, CentOS is not one of the systems they normally test against. They found something similar in testing RHEL 7.0 and have a fix for that in MPSS 3.3. When that comes out in a few weeks, could you please retest to see if the ssh test is working?

And on the issue of the tests that were removed from miccheck with the 3.x releases, there was a major rewrite of miccheck with that release and it is not so much that those tests were removed as that they were not added in. The decision of what tests to put in was based on what the developers believed was still relevant. I have asked that they at least add an option to allow you to enable all the tests without having to explicitly specify each of the disabled ones by name - they will think about it.

[Pierpaolo_M_]

Hi,

i have tried with MPSS 3.2.3 and it continues to fail. I wait for MPSS 3.3 and in the meantime, thanks for your support.

Pierpaolo

[Frances_R_Intel]

Pierpaolo,

Have you had a chance to check out MPSS 3.3?

Frances

[Pierpaolo_M_]

Hi,

thanks for your suggestion, but i didn't have a chance yet and today it is my first day of vacation. I will try as soon as possible and then i write here results.

thanks again

Pierpaolo