VTune agent: {"error":"Unexpected error"} - Can't upload results from kvm-custom-collector.py

Gust · ‎04-20-2023

I'm getting this error when VTune agent tries to upload data from a KVM VM.

Permissions on files seem to be ok and the agent is capable of creating the cache directory.

I'm running version 2023.1.0 of VTune.

I'm manually running the agent with: ./vtune-agent --uuid=344035cd-f1cc-4fa6-8815-03d84296c7db --admission-key-path=/root/admission.key --deployment-user=passphrase_authenticated_user --shared --log-level trace

Running the kvm collector: python3 /opt/intel/oneapi/vtune/2023.1.0/bin64/kvm-custom-collector.py --kvm-ssh- login=root@10.221.19.228 --vtune-dir-on-kvm=/opt/intel/oneapi/vtune/2023.1.0/

VTune agent log:

Collection finished with the exit code 0
Starting trace files upload
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/systemcollector-24469-dl360g10plus-197.maas.sc"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/perfcmd"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/system-wide.stat.perf"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/sep.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24479.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/map.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24519.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/map.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/perfcmd"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/stdsrc-14077875573749.ftrace"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/system-wide.perf"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8765.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/systemcollector-8750-worker-node-de1.sc"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8780.log"
in:  {"messages":[]}
out: {"messageType":"collector:stdout","messageParameters":{"id":"15","message":"Q29sbGVjdGlvbiBoYXMgc3RvcHBlZC4K","size":"24","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
out: {"messageType":"collector:feedback","messageParameters":{"id":"14","message":"IDxzdGF0ZV9jaGFuZ2VkIHN0YXRlPSI5Ii8+CiA8bm9wLz4KIDxzdGF0ZV9jaGFuZ2VkIHN0YXRlPSIzIiBkYXRhPSIwIi8+CiA8bm9wLz4KPC9mZWVkYmFjaz4K","size":"93","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
out: {"messageType":"collector:feedback","messageParameters":{"id":"15","message":"Lz4K","size":"3","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/system-wide.perf"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/systemcollector-24469-dl360g10plus-197.maas.sc"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/perfcmd"
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/stdsrc-14077875573749.ftrace"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24479.log"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/system-wide.stat.perf"
{"error":"Unexpected error"}{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/map.events"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24519.log"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/sep.events"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/perfcmd"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/map.events"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/systemcollector-8750-worker-node-de1.sc"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8765.log"
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8780.log"
Deleting the result directory: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7"
Finished traces upload

VaradJ_Intel · ‎04-21-2023

Hi,

Good day to you.

Thanks for posting in Intel Communities.

Are you trying to profile an application which is running inside remote server(KVM)?

If yes you can try remote server profiling using the steps given below:

1. Password less ssh.

2. run "vtune -target-system ssh:<user_name>@ip (example user@192.168.1.1) -collect <analysis name> <path to the binary>"

For detailed description of this method you can follow this link: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/remote-linux-target-setup.html

Also, please can you provide us the the below details:

1. Processor details

2. OS details

3. Kernel version

4. Exact steps to reproduce the issue(if possible screenshots)

Thank You!

Regards,

Varad

Gust · ‎04-21-2023

Hi Varad,

I'm trying to "Profile system" not a single application.

Steps:

1. Start VTune webserver in host machine: ./vtune-server --allow-remote-access

2. Start agent in host machine: ./vtune-agent --uuid=344035cd-f1cc-4fa6-8815-03d84296c7db --admission-key-path=/root/admission.key --deployment-user=passphrase_authenticated_user --shared --log-level trace

3. Run the following:

I've tried without the kvm collector and I get the same error in the agent logs.

There's one thing that might be worth mentioning. I can't run the agent automatically, only if I manually run does it connect.

CPU: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz

OS: Rocky Linux release 8.6 (Green Obsidian)

Kernel: 4.18.0-372.19.1.el8_6.x86_64

Gustavo

VaradJ_Intel · ‎04-28-2023

Hi,

Good day to you.

I'm trying to "Profile system" not a single application

The above suggestion which we provided is also applicable to "Profile System".

Please can you try using vtune-backend instead of vtune-server as it is causing issue.

Here are detailed step for vtune-backend method:

1. Set up password less ssh by following the below link:

https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/configuring-ssh-access-for-remote-collection.html

2. Next step is to do port forwarding while connecting to the system:

Example : ssh -L 127.0.0.1:7000:127.0.0.1:7000 username@ip

Note: Here, we are port forwarding to the port 7000

3. Run Vtune backend using the below given command:

vtune-backend --allow-remote-access --enable-server-profiling --web-port 7000

4. Now, Paste that link into the browser to get VTune GUI

5. Click on configure analysis and in "where" pane select "Remote Linux (SSH)" option.

6. Fill the details about the remote server and deploy the agent.

7. After that in "what" pane select "Profile System" and fill the fields which are required in the "advanced" section

8. Now select the analysis type and click on run.

Please do let us know if you still face any issues.

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

Thank You!

Regards,

Varad

Gust · ‎05-02-2023

Hi Varad,

Sorry for the late reply.

vtune-backend works fine, nothing else needed.

Is server deprecated or something?

Gustavo

VaradJ_Intel · ‎05-09-2023

Hi,

Good day to you.

Is server deprecated or something?

Yes, vtune-server is now changed to vtune-backend. You could use vtune-backend for remote profiling from now onwards.

Glad to know that your issue is resolved.

Could you please confirm if we can close this thread as your issue is resolved?

Thank You!

Regards,

Varad

VaradJ_Intel · ‎05-17-2023

Hi,

Good day to you.

We have not heard back from you. Could you please confirm if we can close this thread?

Thank You!

Regards,

Varad

Gust · ‎05-17-2023

Hi Varad,

Sorry, missed the other email.

The issue is closed, with vtune-backend never faced any issues.

Gustavo

VaradJ_Intel · ‎05-24-2023

Hi,

Good day to you.

Thanks for confirming.

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thank You!

Regards,

Varad