how to set passwordless authentication in a cluster

[JS1]

Hi,

I am configuring MIC boards on a Rocks cluster. I am encountering an issue about accessing MICs with passwordless authentication.

Basically, my home directory is mounted to all machines of the cluster, but my RSA public key is based on the head node of the cluster. Therefore, this key is only useful in the head node; however, in all other nodes, I have to type the password when ssh to a MIC. How could I fix this issue?

Thanks!

[Andrey_Vladimirov]

Hi JS,

Do I understand you correctly: you are able to SSH without a password from the head node to all compute nodes (because /home/*/.ssh/id_rsa.pub is present in the respective /home/*/.ssh/authorized_keys), and you are able to SSH without a password from the head node to all MICs, but not from compute nodes to MICs?

If this is the case, then the question is: do you NFS-share /home with the MICs? If you DO, then something is wrong with MPSS configuration, and you can troubleshoot by monitoring /var/log/messages on MICs while you are logging in (you can log in to MICs as root, right?)

If you do NOT NFS-share /home with MICs, then you need to make sure that on compute nodes, /var/mpss/mic*/home/*/.ssh/authorized_keys contains the public SSH key of the respective user. If it does not, you probably did not have /home NFS-mounted on compute nodes when you ran "micctrl --initdefaults". You can either add the keys to /var/mpss/mic*/home/*/.ssh/authorized_keys manually and restart MPSS, or you can re-run the MPSS configuration commands with /home mounted on compute nodes.

Andrey

[JS1]

Hi Andrey,

Thanks for your reply! I think I don't have NFS-share /home with MICs. How could I set NFS with --initdefaults? Also, I tried to put authorized_keys to /var/mpss/mic*/home/*/.ssh/ and restarted MPSS service. But it looks like that the file I put doesn't show up in MIC:/home/*/.ssh/ directory. Did I do anything wrong?

Thanks again!

Andrey Vladimirov wrote:

Hi JS,

Do I understand you correctly: you are able to SSH without a password from the head node to all compute nodes (because /home/*/.ssh/id_rsa.pub is present in the respective /home/*/.ssh/authorized_keys), and you are able to SSH without a password from the head node to all MICs, but not from compute nodes to MICs?

If this is the case, then the question is: do you NFS-share /home with the MICs? If you DO, then something is wrong with MPSS configuration, and you can troubleshoot by monitoring /var/log/messages on MICs while you are logging in (you can log in to MICs as root, right?)

If you do NOT NFS-share /home with MICs, then you need to make sure that on compute nodes, /var/mpss/mic*/home/*/.ssh/authorized_keys contains the public SSH key of the respective user. If it does not, you probably did not have /home NFS-mounted on compute nodes when you ran "micctrl --initdefaults". You can either add the keys to /var/mpss/mic*/home/*/.ssh/authorized_keys manually and restart MPSS, or you can re-run the MPSS configuration commands with /home mounted on compute nodes.

Andrey

[Andrey_Vladimirov]

What I meant by "re-run MPSS configuration commands with /home mounted on compute nodes" is that you should first mount /home on compute nodes as usual, and then run "micctrl --cleanconfig" and "micctrl --initdefaults" on each compute node. This will put the SSH keys into the MIC filesystems. If you go this route, note that "micctrl --cleanconfig" will remove any MPSS customizations that you had, like networking and users on MIC. After that, "micctrl --initdefaults" will re-create this configuration it from scratch. Schematically, the configuration looks like this:

[bash]

cat /home/myuser/.ssh/id_rsa.pub >> /home/myuser/.ssh/authorized_keys

chmod 400 /home/myuser/.ssh/authorized_keys

su

mount /home [if not already done]

service mpss stop

micctrl --cleanconfig

micctrl --initdefaults

service mpss start

su myuser

ssh mic0

# you should be logged in without a password now

[/bash]

And regarding the other method, adding "authorized_keys" to /var/mpss/mic*/home/.ssh/, I forgot to tell you that you also have to modify /var/mpss/mic*.filelist to add a line with this filename (if it is not there yet). However, this route is not good if you don't have the correct file in there, because it means that something had failed earlier on in the configuration procedure. So it is better to do the re-configuration of MPSS from scratch as shown above, rather than mess with the MIC filesystem.

[JS1]

Hi Andrey,

I did try both methods (reconfiguring and modifying filelist) you mentioned, but neither of them worked. I think it is because we have different machine names in different compute nodes. Let me elaborate.

In the host machines, I have my home directory /home/myuser mounted on every nodes. In /home/myuser/.ssh, I have my ssh key generated by "ssh-keygen -t rsa" in the head node, therefore I can ssh to any node without password. However, the id_rsa.pub file has the machine name of the head node, i.e. I have "myuser@headnodename" at the end of id_rsa.pub.

This public key enables me to log onto MIC cards in head node without password, but since it contains the machine name of head node, it won't enable me to ssh to MIC cards in compute nodes ("micctrl --initdefaults" returns warning of "mic0: User 'myuser' does not have either rsa or dsa keys created"). To confirm this, I deleted the rsa key generated in the head node, instead generated a rsa key in a compute node. By doing that, the machine name in the public key became this compute node. Then I did "micctrl --initdefaults", and was able to ssh to the MICs in this compute machine without password.

Hope this makes sense to you. How should I deal with this issue then?

Thanks!

[Andrey_Vladimirov]

Hi JS,

the username and hostname at the end of the public SSH key should not matter, it is just a comment, and can be set to anything. On my headless cluster, node c001-n001 plays the role of the head node, and node c001-n002 mounts /home from c001-n001. The SSH key situation looks like this:

[bash]

[cfxuser@c001-n002 ~]$ cat ~/.ssh/authorized_keys # authorized_keys on host
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAve1JVz+NWkHZoDUU3VmHi4GJC1SuxU3o5KaKkZl3Wgv+dE46OmzusMIHt14g2dIFh8ayI1STqN7hy9HUR+0Uk4R+NW5CpXXX1aIXCqjoBF4AKnE/vspzFf5g5yInxqQxKUPNFLxr2lF28Eogl4RB0DxPQmdRwupjyDSWbSKlvUfdPWOLesgpWR3K0np4a8JWEg5wkQHcBxptjiP2dbrIAFAu2fcDdFEGQxAjacUqyM4gaj685s+3NvEngNyunDRqlniTzsVZvf/q5vsLAhFDASC3YBOH9mHpfrVk5+fknHAOiNe6OegSdek/9p/in3P9+5sGpP0XJsT4Cy59ZvBSGQ== cfxuser@c001-n001
[cfxuser@c001-n002 ~]$
[cfxuser@c001-n002 ~]$ ssh mic0 cat ~/.ssh/authorized_keys # authorized_keys on mic0
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAve1JVz+NWkHZoDUU3VmHi4GJC1SuxU3o5KaKkZl3Wgv+dE46OmzusMIHt14g2dIFh8ayI1STqN7hy9HUR+0Uk4R+NW5CpXXX1aIXCqjoBF4AKnE/vspzFf5g5yInxqQxKUPNFLxr2lF28Eogl4RB0DxPQmdRwupjyDSWbSKlvUfdPWOLesgpWR3K0np4a8JWEg5wkQHcBxptjiP2dbrIAFAu2fcDdFEGQxAjacUqyM4gaj685s+3NvEngNyunDRqlniTzsVZvf/q5vsLAhFDASC3YBOH9mHpfrVk5+fknHAOiNe6OegSdek/9p/in3P9+5sGpP0XJsT4Cy59ZvBSGQ== cfxuser@c001-n001
[cfxuser@c001-n002 ~]$
[cfxuser@c001-n002 ~]$ ssh mic0 # passwordless login
cfxuser@c001-n002-mic0:~$

[/bash]

As you can see, the comment "cfxuser@c001-n001" does not prevent passwordless login in my case.

However, I think, the clues are the warning "mic0: User 'myuser' does not have ... keys created" and the fact that if you generate a key on the compute node, passwordless SSH to mic0 works. My guess is that there is something wrong with NFS. Either the home directory is not actually NFS-shared with the compute node (sorry if it is a silly assumption, but it would explain your observations, and I don't know how how carefully you checked if mounting /home succeeds), or perhaps something is wrong with NFS settings which causes permission problems on the keys.

Could you check what "ls -lha ~/.ssh/*" reports on the head node and on the compute node? Do usernames show up correctly on the compute node, or do you get "nobody" for user and group? Are time stamps on the head node and on compute node identical on the file ~/.ssh/id_rsa? Are all files inside ~/.ssh, and the directory itself, read/writable only by the owner?

Andrey

[JS1]

Hi Andrey,

Thanks for double checking for me.

I am pretty sure that my NFS works fine. The time stamps are all the same across nodes, there's no "nobody" appears, etc. I am using Rocks to manage the cluster, which mounts the users' home directories in /etc/auto.home by default (-nfsvers=3). I am not sure whether it could be a nfs setting/configuring issue, buy at least it works well except this MIC login issue.

Regarding the access permission, must we set all files inside ~/.ssh, and the directory itself be read/writable only by the owner? I set that, it still didn't work though.

[JS1]

Hi Andrey,

I think I figured out the reason. basically, when doing "micctrl --initdefaults", "authorized_keys", "id_rsa" and "id_rsa.pub" must be accessible by root account of the local machine, otherwise they won't be copied to /var/mpss/mic*/home/myuser/.ssh. However, after setting, we have to chmod 600 id_rsa, otherwise it causes a warning and prevents keyless login.

Thanks a lot for your help!

[Andrey_Vladimirov]

Edit: missed your earlier post. Thanks for the update!

[Andrey_Vladimirov]

Was it the "root_squash" setting in /etc/exports that caused this problem?
 

[JS1]

We set "no_root_squash" in the corresponding entry of /etc/exports.

Andrey Vladimirov wrote:

Was it the "root_squash" setting in /etc/exports that caused this problem?
 

[Johnnie_P_Intel]

Populating the ssh keys is a tricky business and before I could make any real comments I would need to know the particular release of the MPSS software you are trying to use.  But here are some comments

Before the --initdefaults option can populate users on the system it needs a source of user information.  The 3.1 and previous releases scour the hosts /etc/passwd file and the the /home directory to find them.  If they are in the /etc/passwd file it has much more information than looking at home.  When copying ssh keys it looks in the users home directory for the .ssh directory and copies very specific files (id_rsa, id_rsa,pub, id_dsa, and id_dsa,pub).  It also look for the authorized_keys file and adds the contents of the .pub files if they are not already in it.  So if it cannot find keys it can't copy them.

That mechanism is not nearly good enough with a wide group of customers.  In the upcoming 3.2 release this code has seen many changes.

First, you will now be able to change the way the initial user list is created.  Some sites will want to create the cards initial user list with only the minimum set (root, ssh, micuser, nobody and nfsnobody) and this will be dome with the command "micctrl --initdefaults --users=none".  If you want to use the host as a source of users then use "micctrl --initdefaults --users=overlay", which is the default if --users is not used, but be away it now only looks at the hosts /etc/passwd file and no longer scrounges the /home directory.

If you which to change your selection for users then use "micctrl --userupdate=<mode>.  If you set mode to none it will delete all but the base users out of the cards /etc/passwd file.  If you set it to overwrite it will recreate the passwd file as if you called initdefaults again.  Or you can set mode to merge and it will merge any users in the hosts /etc/passwd file it does not know about into the cards /etc/passwd file.

In 3.2 ssh keys file processing has been changed to not just look for rsa and dsa keys.  It now copies all files in the users .ssh directory to the card and injects all .pub file contents into the authorized_keys file.

3.2 has also introduced the "micctrl --sshkey" option to copy ssh keys after the fact.  When a user updates ssh keys for what ever reason they can populate them to the cards file system using this command.

It should also be noted the "filelist" file for setting permissions and owners is no longer a part of 3.2 for both the MicDir and CommonDir parameters.  It is required for Windows hosts but was only a complication on Linux hosts and has been removed.

The level of agressiveness checking file permissions in the .ssh directory can be specified in the OpenSSH build process.  I am guessing you are still on the 2.1 releases since this issue showed up.  In the 3.x series this has been backed off and I just tested 3.2 code that shows the file permissions can be set to 777 for the id_* files and it still lets me in.  The authorized_keys file still must not be accessable to other than the user.  It should be noted that the non .pub part of ssh keys (id_rsa instead of id_rsa.pub) should be tightly controlled by a user and basically given to nobody.  This is the reason stricter compiles of OpenSSH enforce this file being 600.  Your users should have this set to 600 in their .ssh directory.

[Johnnie_P_Intel]

I have a correction to my last blog posting.

The reason I was able to use looser permissions on the ssh keys files is because I have a ".ssh/config" file with the contents:

Host *
      StrictHostKeyChecking no

In the upcoming 3.2 release code where I ran the test, the ".ssh/config" file on my host was also mirrored to the cards file system during the --initdefaults command.

It is still recommended to not have your non public ssh keys file with permissions others can get access to it.

[JS1]

Hi Johnnie,

Thanks for replying. I am using MPSS-3.1.2. Do you mean that it's recommended to upgrade to 3.2 version?

[Johnnie_P_Intel]

Yes, upgrade.  Each and every release of the MPSS software brings bug fixes, more functionality and other quality improvements.  The mpssd/micctrl code is no longer changing existing functionality very much but it is adding new functionality.

Not upgrading means you miss the existing fixes and makes it much harder for us to help you.

[Johnnie_P_Intel]

Logging into a Linux system with ssh is controlled by the files found in a users $HOME/.ssh directory.  There are a number of different encryption schemes that can be used but for this example I will use the "rsa" type.  Running the ssh-keygen without any arguments generates these types and produces an id_rsa and an id_rsa.pub file in your .ssh directory.  You should never ever give the id_rsa key file to anybody.

When you want to auto log in to a Linux box, you place the contents of your id_rsa.pub file into the authorized_keys file in you .ssh directory on the target system.  When you attempt to login, ssh will determine if you are allowed to log in by the contents of the authorized_keys file on the target and your id_rsa file on the host your are logging in from. If it gets a match it lets you in.  If not it then asks for a password.

When micctrl --initdefaults runs it adds users it finds in the passwd file or they can be added later with micctrl --adduser.  In both cases it looks in the users $HOME/.ssh directory on the host for ssh keys.  If found it copies them to the area where the file system for the MIC card is being prepared.  It iwll also look for .pub files and place them in the authorized_keys file.

If the users ID did not exist or did not have ssh key files generated when the initdefaults command was ran then they did could not be copied to the cards file system.  You will have to do it yourself.  The upcoming 3.2 release has a new miccctrl --sshkeys command to help you with that.  In previous releases you must do this yourself.

If the system admin has not changed away from the default locations, then your login would have been generated in the /var/mpss/mic0/home/<yourname> directory.  There you will find the .ssh directory.  Make sure it has valid authorized_keys and id_rsa.pub files.  If you want to log back into the host automatically then the id_rsa file should also be there.

Make sure the privileges are correct.  The authorized_keys and id_rsa.pub file should have 644 and the id_rsa file 600.