Remote VTune execution in devcloud

[ixiotidi]

Hi,

I'm trying to execute VTune in devcloud by following the instructions in this link:

https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/using-vtune-server-with-vs-code-intel-devcloud.html

the problem I'm facing is that I can easily tunnel ssh into the login node, however the second ssh command into the computing node (which is active in another terminal window) where I'm running vtune isn't working. Basically, the command executes but I'm not actually getting into the node. My bash stays in the login node. When I then execute $ps -x | grep ssh to check that the tunnel is working I don't get any results back. This can also be verified that when I try to load vtune from my browser I get an error. Also, when looking into my terminal where I did the ssh tunnels I get an error saying:

channel 2: open failed: connect failed: Connection refused

I can't be seeming to find any instructions on how to solve this issue. I would have expected that since the qsub -I command keeps the computing node working, the tunnel should be active and hence trivial to access port 55001

Any recommendations?

Thanks,

--Ioannis

[yuzhang3_intel]

Let me check first and then give you update.

[Jennifer_D_Intel]

I've seen issues with using port 55001. Can you try something like 55009? The reason we used 55001 is because it's not typically used by anything else, but it can be any available port.  

[ixiotidi]

Hi Jennifer,

thank you for your reply. I'm afraid I don't think it's the port, as I just tried a different one and the symptoms are the same. I believe there is something going on with the ssh tunnel. The reason I'm suspecting this is because when I execute from my local shell the following command:

ssh -L 127.0.0.1:55009:127.0.0.1:55009 devcloud

I hop into the devcloud machine but then at the devcloud machine the same command for the working node:

ssh -L 127.0.0.1:55009:127.0.0.1:55009 s001-n067

doesn't take me to the computing node (please see image attached).

Can it be that there is something in my account that I haven't setup properly?

Thanks,

--Ioannis

[ixiotidi]

Hi,

is there any update regarding this?

Thanks,
—Ioannis

[Jennifer_D_Intel]

I ran some tests and wasn't able to reproduce the problem, so it might be something in your devcloud ssh configuration. Can you check your local ssh config file? It should look something like this:

################################################################################################
# oneAPI DevCloud SSH config
################################################################################################
Host devcloud
User uxxxx
IdentityFile ~/.ssh/devcloud-access-key-xxxx.txt
ProxyCommand ssh -T -i ~/.ssh/devcloud-access-key-xxxx.txt guest@ssh.devcloud.intel.com

Host devcloud.proxy
User uxxxx
Port 4022
IdentityFile ~/.ssh/devcloud-access-key-xxxx.txt
ProxyCommand ssh -T devcloud-via-proxy

# If you must route outgoing SSH connection via a corporate proxy,
# replace PROXY_HOSTNAME and PORT below with the values provided by
# your network administrator.
Host devcloud-via-proxy
User guest
Hostname ssh.devcloud.intel.com
IdentityFile ~/.ssh/devcloud-access-key-xxxx.txt
LocalForward 4022 c009:22
ProxyCommand nc -x your.proxy.server.com:port %h %p
################################################################################################

################################################################################################
# DevCloud VSCode config
################################################################################################
Host devcloud-vscode
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
Hostname localhost
User uxxxx
Port 5022
IdentityFile ~/.ssh/devcloud-access-key-xxxx.txt
################################################################################################

################################################################################################
# SSH Tunnel config
################################################################################################
Host *.aidevcloud
User uxxxx
IdentityFile ~/.ssh/devcloud-access-key-xxxx.txt
ProxyCommand ssh -T devcloud nc %h %p
LocalForward 5022 localhost:22
LocalForward 5901 localhost:5901
################################################################################################

[ixiotidi]

Hi Jennifer,

thank you for your reply. Please see below my ssh config (for which I used the automatic script provided by Intel).

################################################################################################
# oneAPI DevCloud SSH config
################################################################################################
Host devcloud
User uXXXXX
IdentityFile ~/.ssh/devcloud-access-key-212375.txt
ProxyCommand ssh -T -i ~/.ssh/devcloud-access-key-212375.txt guest@ssh.devcloud.intel.com

Host devcloud.proxy
User uXXXXX
Port 4022
IdentityFile ~/.ssh/devcloud-access-key-212375.txt
ProxyCommand ssh -T devcloud-via-proxy

# If you must route outgoing SSH connection via a corporate proxy,
# replace PROXY_HOSTNAME and PORT below with the values provided by
# your network administrator.
Host devcloud-via-proxy
User guest
Hostname ssh.devcloud.intel.com
IdentityFile ~/.ssh/devcloud-access-key-212375.txt
LocalForward 4022 c009:22
ProxyCommand nc -x PROXY_HOSTNAME:PORT %h %p
################################################################################################

################################################################################################
# DevCloud VSCode config
################################################################################################
Host devcloud-vscode
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
Hostname localhost
User uXXXXX
Port 5022
IdentityFile ~/.ssh/devcloud-access-key-212375.txt
################################################################################################

################################################################################################
# SSH Tunnel config
################################################################################################
Host *.aidevcloud
User uXXXXX
IdentityFile ~/.ssh/devcloud-access-key-212375.txt
ProxyCommand ssh -T devcloud nc %h %p
LocalForward 5022 localhost:22
LocalForward 5901 localhost:5901
################################################################################################

it looks rather identical to me, to what you've posted. One extra bit to add on the table. If execute VTune from the login node (of course I get some memory issue, however then the webpage becomes accessible (of course not fully loaded). So that makes me genuinely think that either there is some short of constraint within my account or something that I'm not getting when I try to tunnel from my login node to the computing node. Additionally, I know 2x more people that are facing the same exactly problem.

Thanks,

--Ioannis

[ixiotidi]

Hi Jennifer,

don't know if you have any more updates for me about this. I did a little bit more investigation. Basically when I qsub -I in a computing node (e.g. s001-n058), I open a new terminal and I ssh tunnel with the command from above. Subsequently I try to tunnel into the active computing node with ssh -L 127.0.0.1:55001:127.0.0.1:55001 -v s001-n057 clearly in the debug output I see that my tunneling isn't really working from login to computing node.

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /home/uXXXXXX/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to s001-n058 [10.9.1.58] port 22.
debug1: Connection established.
debug1: identity file /home/uXXXXXX/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/uXXXXXX/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.11 pat OpenSSH* compat 0x04000000
debug1: Authenticating to s001-n058:22 as 'uXXXXXX'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:w9YpGXPNBARHJeFgysYdAPNuP9PqfAWR4Ba1rKoYXe0
Warning: Permanently added 's001-n058,10.9.1.58' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:M2GDEOEcnKhPp/HODa4c9OvBigJpZ6j5Xlp5cf/mf/Y /home/uXXXXXX/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/uXXXXXX/.ssh/id_dsa
debug1: Trying private key: /home/uXXXXXX/.ssh/id_ecdsa
debug1: Trying private key: /home/uXXXXXX/.ssh/id_ed25519
debug1: No more authentication methods to try.
uXXXXXX@s001-n058: Permission denied (publickey).

Thanks any more help on this is highly appreciated.

--Ioannis

[Jennifer_D_Intel]

I'm still working with the DevCloud admins to figure out the problem. There was a mention of an update last week for an issue with port forwarding, but I'm not sure if it was completed.

[ixiotidi]

Hi Jennifer,

is there any update regarding the VTune GUI running on the DevCloud?

Thanks,

--Ioannis

[Jennifer_D_Intel]

I have pinged the DevCloud admin again and will let you know his response. There still seem to be some issues with VTune for some reason.

[Jennifer_D_Intel]

I apologize for taking so long to resolve this. I heard back from the devcloud admin, who responded that this is known to fail when connecting from a Windows host unless you are using Windows Subsystem for Linux. But I've been unable to reproduce the error using cygwin or putty, until today.  I was running some more tests with a colleague and we managed to reproduce the problem by closing one of the ssh terminals with the VTune browser window open. Here is what we did:

1. Open a cygwin terminal and connect with ssh -L 127.0.0.1:55009:127.0.0.1:55009 devcloud
2. Connect to a compute node with qsub -I
3. Open a second cygwin terminal and connect with ssh -L 127.0.0.1:55009:127.0.0.1:55009 devcloud
4. From the login node, run ssh -L 127.0.0.1:55009:127.0.0.1:55009 s001-nxx
5. In the second cygwin terminal, run vtune-backend --enable-server-profiling --web-port=55009
6. In a browser, go to https://127.0.0.1:55009 and continue to the VTune server

Once we closed the first terminal, the second terminal started displaying the channel 2 error messages you are getting. Closing the browser ends the errors. It seems that somehow the node stops listening on port 55009. 

I wonder if you are able to connect to VTune server without the tunneling. You would connect to the compute node without specifying the localhost, and then run the following command to start the VTune server:

vtune-backend --allow-remote-access --enable-server-profiling --web-port=55009

The URL will contain the node's IP address instead of localhost. 

[ixiotidi]

Hi @Jennifer_D_Intel 

thank you very much for coming back to me. I'm afraid what I'm trying to do isn't in a Windows environment but rather to a native Linux. However, I did try your instructions in both a Windows machine and a Linux machine and I'm getting the same error. I still believe that the issue is on the tunnel between the login node and the computing node, if for example I run vtune on the login node (of course I'm running out of memory) but I see can access a "corrupted" version of the website GUI. However, when I try to ssh into the compute node I seem to get an error that authentication has failed. So I'm puzzled about it. Is it possible to send me the ssh tunnel command with the -v option enabled when you try to move from a login node to the compute node? Cause I'm getting "Permission denied".

Thanks,

--Ioannis