Authentication problems (NIS/YP and HostBased)

Jess · ‎09-11-2014

I have tried to set up NIS/YP authentication as described in the MPSS 3.3 manual as after we upgraded it was found that user's SSH keys and accounts weren't being copied over any longer. Presumably someone decided that copying /etc/passwd and /etc/shadow across rather than using the list of users with SSH keys in /home was a better idea. This failed, whereupon out of desperation we thought we would try using Hostbased authentication. This has failed too, and rather obscurely. The most relevant part seems to be this:

..
debug3: authmethod_is_enabled hostbased
debug1: Next authentication method: hostbased
get_socket_address: getnameinfo 8 failed: Name or service not known
debug2: userauth_hostbased: chost yellow12.
debug2: ssh_keysign called
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug1: permanently_drop_suid: 23537
get_socket_address: getnameinfo 8 failed: Name or service not known
cannot get sockname for fd
ssh_keysign: no reply
key_sign failed
debug2: we did not send a packet, disable method
..

In the process it seems that there is no real way to get the Phis to forget their previous config (various invocations of micctrl have all failed to do everything the --help output claims they do). Something is always left behind (and tricky to track down), though even the hosts are diskless and everything is rebuilt when they reboot. This is frustrating, to say the least, and I can't help wondering if the error mentioned above is due to this problem. The instructions for NIS/YP authentication seem to also talk about LDAP in places, which we don't use, so although I did try following the instructions to the letter I later also attempted to just set up NIS as has already been done on the host. It doesn't work using either method. Only root can log in and users don't seem to be recognised. It seems that regardless of whether we use the FQDN or the IP for the NIS servers, the Phis don't seem to know how to do any routing to them. I wonder if this is to do with the domainname, since the NIS domain is NOT the same as the DNS domain used for the cluster. Really I would very much like to get NIS to work correctly, since that is what we are using elsewhere, so if anyone has any advice on how to sort this out I would appreciate it.

JOHNNIE_P_Intel1 · ‎10-01-2014

Hello,

The code scrounging the /home directory for users could only be at its best a nasty hack. It was created during the very first phases of getting anything to configure and really was a hack to get one internal cluster here working. It had to make way to many assumptions.

So when the code was added to allow LDAP and NIS configurations it was removed. In the mean time micctrl also has a --useradd option to allow the system admin to add any user wished to the cards file system. If the user has a directory on the host where micctrl is ran then it is searched for .ssh keys at the time --useradd is executed. If not the user himself may add his ssh keys to the MIC card filesystem with the --sshkeys option to micctrl.

I will take a look at getting the ssh host based authentication to work. I had tried in the 3.1 time frame but PAM support on the cards file system did not allow it at that time. That should have been corrected by now. Give me a day or two to figure it out and back to this blog.

In the mean time, please give a much fuller description of your system configuration. What it your host OS? How are you configuring the NIS and YP servers? etc.

Johnnie Peters

Jess · ‎10-02-2014

I'm not saying I think the nasty hack was a good idea, but removing it without highlighting the change has caused us a lot of grief. NIS has always been preferable from our point of view I don't want to add people manually. We have a *lot* of users, with access controlled via AD groups. We don't have LDAP unfortunately, only NIS (political reasons), and we are trying to use IBoIP. The manual is pretty poor at talking about whether the external bridge can be used with IPoIB. What we have had to do is configure IPoIB and not the external bridge, as we can't control which traffic flows over which network otherwise and the MIC cards keep trying to send MPI traffic over the ethernet network. There are two ethernet networks and one InfiniBand one on our machine. Neither ethernet network should be used for MPI as they are dedicated to other things, and actually giving IPs on those to the MICs is not something we want to do. An external bridge with the IB interface doesn't seem to work at all. Bizarrely, ping and other simple tools seem to work fine, but ypbind is unable to contact the NIS servers or resolve their IPs from the FQDNs (and we have tried using the IPs rather than the FQDNs too). Unfortunately I don't have any control over the AD side of things, but the rest of the cluster works fine using NIS. We are running RHEL 6.3 with security updates and Mellanox OFED 2.1 (we have tried 2.0 and 2.2 too).

JOHNNIE_P_Intel1 · ‎10-02-2014

Here is a shell script that pretty much recreates what initdefaults used to do:

for USER in `ls /home`
do
	if [ "$USER" = "lost+found" ]
	then
		continue
	fi

	GROUP=`ls -ld /home/$USER  | cut -d' ' -f4`
	UUID=`id -u $USER 2> /dev/null`

	if [ "$?" != "0" ]
	then
		continue
	fi

	UGID=`id -g $USER`
	echo "Add user $USER uid $UUID gid $UGID"
	micctrl --useradd=$USER --uid=$UUID --gid=$UGID --home=/home/$USER --comment=$USER
	echo "Add group $GROUP gid $UGID"
	micctrl --groupadd=$GROUP --gid=$UGID
done

I am looking into the ssh host based authentication. At the moment we are missing, I think, a support RPM for PAM on the card. As soon as I figure it out I will post here again.

I saw a bug report opened for the NIS issue. I am sure somebody will get back to you soon.

Jess · ‎10-06-2014

Thanks, I'll try that for now.

Jess · ‎10-07-2014

I've tried the workaround on two of our nodes, and it doesn't really work reliably enough I'm afraid. It is very hit and miss as to whether or not a user account is created. It will run, and several will have been created but several others won't have, and there is no way to predict which users will have been affected. More seem to fail than succeed. Occasionally you see error messages stating that micctrl failed to contact mpssd for one or other mic. However, this does not necessarily correlate with a failure to create the user account. (It is also extremely slow, but that had been expected)

JOHNNIE_P_Intel1 · ‎10-07-2014

This should have been pretty much what micctrl did before. Could you send me the output from "ls -l /home" and the script? Lets figure out what the problem is.

Could you also confirm you are using the IPoIB network in your cluster. The information I have is this type of connection will not bridge. so it looks like we need to get your user creation issues solved.

JOHNNIE_P_Intel1 · ‎10-07-2014

I though I had already posted this but since I do not see it here it is.

I have ssh host based authentication working. It solves the issue of user ssh keys but it does not solve your list of users issues as they need to be in the passwd file for it to work.

I also found as a prerequisite you must also have a fully functioning name server entry for the MIC card and revers name look up for its IP address must also work. I found I had misconfigured the reverse look up part in my mini cluster and had to resolve that first. This is particular would be the source of the "get_socket_address: getnameinfo 8 failed: Name or service not known" message you are seeing.

Once I had that work I set host based authentication with the MIC card as the server and the host as the client since I wanted to ssh from host to client.

First the server part. The micctrl utility does not manipulate the cards /etc/ssh/sshd_config file so I booted the card and scp'd it to the host under /var/mpss/mic0/etc/ssh/sshd_config. Then I edited it and uncommented the HostbasedAuthentication entry and set it to yes. I also uncommented the IgnoreRhosts entry and made sure it is set to yes. Here is the resulting file:

#	$OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options change a
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile	.ssh/authorized_keys

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication yes
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication, account processing, 
# and session processing. If this is enabled, PAM authentication will 
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
UsePrivilegeSeparation yes
#PermitUserEnvironment no
Compression no
ClientAliveInterval 15
ClientAliveCountMax 4
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	/usr/libexec/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	ForceCommand cvs server

Then I added a /var/mpss/mic0/etc/ssh/ssh_known_hosts with the host rsa ssh key information and my host name and IP of eagles.music.local and 192.168.5.4. Again here is the resulting file:

eagles.music.local,192.168.5.4 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw1H6iU7U5McrSihq99edHazc7kmdsiyAMhg5i38N3KSYZ24iLNzb2sL8zt4D3Vkg2uzgwG6+gcqD9BSkkx/NbimXGgvW1h/JjGl+fcfAZWsWqaS1o4M0xd53e1EObSZLS2/JJUZ6AI29+zuB/EhPolAOIZ5g/v0XngkW/xDYsa8KQNKzPvbM569oL+pmhge4uqhfzCbpim/B6ZMGQvbaaAD+gtK/jXhal3+2sScR1aN5Ahqlf4D6KrX4D29YRFgcQLGkzKTXFlWf9N/KNjkUeZ04r8ERbup2c4O4u5BmdniNu6W5RN1eFt4xYjRKwm6gKy5q1lOntgeTSXlh9Ee1hw==

And last for the server configuration is to create the /var/mpss/mic0/etc/hosts.equiv file for the card with the hosts name in it. Mine looks like:

eagles.music.local

Now reboot the card for the settings to become effective.

Second configure the host to be a ssh host based authentication client connection. Change the hosts /etc/ssh/ssh_config file to enable host authentication. Edit the file and under the "Host *" configuration box add "HostbasedAuthentication yes" and "EnableSSHKeysign yes". Here is the resulting file:

/etc/ssh/ssh_config
-------------------------------------------------------------
#	$OpenBSD: ssh_config,v 1.25 2009/02/17 01:28:32 djm Exp $

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

# Host *
#   ForwardAgent no
#   ForwardX11 no
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication yes
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
Host *
	GSSAPIAuthentication yes
	HostbasedAuthentication yes
	EnableSSHKeysign yes

# If this option is set to yes then remote X11 clients will have full access
# to the original X11 display. As virtually no X11 client supports the untrusted
# mode correctly we set this to yes.
	ForwardX11Trusted yes
# Send locale-related environment variables
	SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES 
	SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT 
	SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
	SendEnv XMODIFIERS