VTune backend doesn't detect agent connection

[maleadt]

I'm trying to use VTune to remotely profile a system. I'm using the Web UI (launched with

--allow-remote-access), which successfully manages to connect to my remote system over SSH, as well as deploy vtune-agent. However, it hangs at:

VTune Profiler Agent is being deployed to the target system. It may take a couple of minutes.

This despite the vtune-agent having been successfully deployed & launched on my remote system. In fact, if I look at how vtune-agent is launched and replicate that myself, it successfully connects to my the server:

❯ ./vtune-agent --uuid=d9ba52ad-2b11-456f-899e-98ee615b2f63 --admission-key-path=/home/tim/admission.key --deployment-user=passphrase_authenticated_user --owner=passphrase_authenticated_user
Intel(R) VTune(TM) Profiler Agent
Connecting to server https://my-vtune-server
Agent fingerprint: 55:e2:8d:72:b7:68:86:c1:10:f2:4c:f1:ed:3f:de:8d:ab:3d:39:d4
Connected to the server
Configuration complete. The Agent is ready for data collection.

Here's the server's debug log when connecting the agent: https://gist.github.com/maleadt/66e437026b74224d7a7d51bbe9c6fae8

FWIW, the server is running in a docker container behind a reverse proxy, but I've correctly set the base-url (which is set in the agent config, as shown above by the agent successfully connecting). I'm also seeing the same behavior when running outside of a docker container.

[Rahila_T_Intel]

Hi,

Thank you for posting in Intel communities.

There are some issues with the method which you are following to deploying agent, so the preferred method is running Vtune via remote ssh.

Could you please try out the below steps:

1. Set up password less ssh by following the below link:

https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/configuring-ssh-access-for-remote-collection.html

2. Next step is to do port forwarding while connecting to the system:

Example : ssh -L 127.0.0.1:7000:127.0.0.1:7000 username@ip

Note: Here, we are port forwarding to the port 7000

3. Run Vtune backend using the below given command:

vtune-backend --allow-remote-access --enable-server-profiling --web-port 7000

4. Now, Paste that link into the browser to get VTune GUI

5. Click on configure analysis and in "where" pane select "Remote Linux (SSH)" option.

6. Fill the details about the remote server and deploy the agent.

7. Now select the analysis type and click on run.

if this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

 Thanks

[maleadt]

Can you explain what you information you are trying to get with those instructions? I already have a fully working set-up where I can reach a remote VTune GUI, whose back-end in turn can successfully deploy an agent and get it to connect back, so I don't understand why you are sending generic set-up instructions.

[bannu]

The preferred method is running Vtune via remote ssh (I think she wants to convey that). I can see some old threads where users are facing issues while running the Vtune agent.

Before doing the above process, you need to use passwordless ssh to connect (from the host machine to the target machine).

Run the below command after doing passwordless ssh (to profile the application on the target machine).

vtune -target-system ssh:root@<Target name> -target-install-dir=/tmp/vtune_profiler -collect hotspots -app-working-dir /usr/bin <Path to the application in the Target container> 

You want to report an issue about the vtune backend not detecting the vtune agent while using --allow-remote-access?

are you facing same issue without using --allow-remote-access too ?

[maleadt]

I already have vtune-backend successfully connected to the target system using password-based SSH, where it properly deploys the agent, launches it, and has it connect back. It's the back-end/GUI that somehow doesn't detect that the agent connected back, so I fail to see what SSH has to do here, as the connect-back happens over the web port AFAICT?

In any case, I did try out the suggestion (connecting over passwordless SSH instead), and that did not help. The problem is identical: The WebUI hangs, telling me it's waiting for the agent to deploy, even though it's successfully deployed and connected.

Using the vtune command directly, with -target-system as you mentioned, does work. AFAIU, that works differently, as I see it running amplxe directly without involvement of vtune-agent. And that's what I'd like to get working, so that I can use the web UI. Or is it possible to use the web UI/vtune-backend without having it require vtune-agent?

[bannu]

Let me explain what @Rahila_T_Intel is trying to say. Basically, you are using the below method and deploying agent as shown in the below screenshot.

There are some issues with the above process, so the workaround she gave was running Vtune via remote ssh. The command line command I gave above is ntg, but remote Linux ssh (-target-system)

If you do password-less ssh perfectly, you can click on deploy and start analysis as shown in the below screenshot.

This above process is similar to the below command, but you are running it in a GUI:

vtune -target-system ssh:root@<Target name> -target-install-dir=/tmp/vtune_profiler -collect hotspots -app-working-dir /usr/bin <Path to the application in the Target container> 

Another method is running above command via terminal and view/open results on vtune GUI.

If this workaround works, you can accept @Rahila_T_Intel post as a solution so that can help others with similar issue. Thank you.

[maleadt]

I see; thanks for spelling it out. I'm pretty new to VTune, so didn't realize that the remote target mode, which uses SSH,  is different from the remote SSH mode, which requires --enable-server-profiling. Pretty confusing.

Although this is a valid workaround, the cluster I'm trying to connect to doesn't allow me to add arbitrary SSH keys to set-up passwordless SSH, so this isn't an option.

It's too bad the remote agent doesn't work. @Rahila_T_Intel Is this a known bug, or how should I file an issue about this?

[Rahila_T_Intel]

Hi,

We apologize for the issue and confusion caused to you by us. We have informed the concerned team regarding this issue. They are working on this internally.

Meanwhile you can find the below links for your references. 

To manually deploying the agent: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/web-server-ui.html#SECTION_A776AEBFD14D459EB8A4EB9806EFF56A

To running it inside a container: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/run-from-container.html 

Thanks

[Rahila_T_Intel]

Hi,

We got an update from the Development team that VTune agent issue will be fixed in the next update. 

Could you please confirm whether we can close this thread?

Thanks

[maleadt]

I updated to 2023.2 (I was using 2023.1), and the problem persists.

[Rahila_T_Intel]

Hi,

Good day to you.

Thank you for your patience. Our dev team is working on your issue and currently, we don't have any ETA on when the fix will be implemented.

We will notify you once we have an update on this thread.

Sorry for the inconvenience caused.

Thanks

[Rahila_T_Intel]

Hi,

Thank you for your patience.

The issue raised by you is targeted to release in 2024.2version.

Please let me know if we can go ahead and close this thread?

Thanks

[Rahila_T_Intel]

Hi,

We haven't heard back from you. Could you please confirm whether we can go ahead and close the case?

Thanks

[Rahila_T_Intel]

Hi,

We haven't heard back from you. We will be closing this thread from our side, If the issue still persists with new release then create a new thread for us to investigate. 

Thanks