Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)
5025 Discussions

VTune agent: {"error":"Unexpected error"} - Can't upload results from kvm-custom-collector.py

Gust
Beginner
1,626 Views

I'm getting this error when VTune agent tries to upload data from a KVM VM.

 

Permissions on files seem to be ok and the agent is capable of creating the cache directory.

I'm running version 2023.1.0 of VTune.

I'm manually running the agent with: ./vtune-agent --uuid=344035cd-f1cc-4fa6-8815-03d84296c7db --admission-key-path=/root/admission.key --deployment-user=passphrase_authenticated_user --shared --log-level trace

Running the kvm collector: python3 /opt/intel/oneapi/vtune/2023.1.0/bin64/kvm-custom-collector.py --kvm-ssh- login=root@10.221.19.228 --vtune-dir-on-kvm=/opt/intel/oneapi/vtune/2023.1.0/

 

VTune agent log:

 

Collection finished with the exit code 0
Starting trace files upload
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/systemcollector-24469-dl360g10plus-197.maas.sc"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/perfcmd"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/system-wide.stat.perf"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/sep.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24479.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/map.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24519.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/map.events"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/perfcmd"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/stdsrc-14077875573749.ftrace"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/system-wide.perf"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8765.log"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/systemcollector-8750-worker-node-de1.sc"
Uploading trace file: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8780.log"
in:  {"messages":[]}
out: {"messageType":"collector:stdout","messageParameters":{"id":"15","message":"Q29sbGVjdGlvbiBoYXMgc3RvcHBlZC4K","size":"24","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
out: {"messageType":"collector:feedback","messageParameters":{"id":"14","message":"IDxzdGF0ZV9jaGFuZ2VkIHN0YXRlPSI5Ii8+CiA8bm9wLz4KIDxzdGF0ZV9jaGFuZ2VkIHN0YXRlPSIzIiBkYXRhPSIwIi8+CiA8bm9wLz4KPC9mZWVkYmFjaz4K","size":"93","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
out: {"messageType":"collector:feedback","messageParameters":{"id":"15","message":"Lz4K","size":"3","agentSignature":"4mC+tpFjzAwsiagDwd/n+Ej5meVdfRhuo+4p5gmJmZ8yoIX5/OqxM5gkqJ2HYxLXERRYtqbVDzNbC3y9rAN1c7rsM32QkUJbH20iSDhcvcE5+of6Y464TIW+J9oCvoIMYtD89qydxPVs+FoyNg+N9ESKkEdmkYLr3lV/OBjRAQlXvBZ7bgJUSf2kCoYsy/athVx+2ifdeDjx/0yxacEPnggn0rEIypGTVoPTtiZ0vpUJ7DmuRjytlfMMguUOwgjwpxNZboosyUHgn38WnYxwH+NUQw1SEBwsfhzR2g9/5IwOuU4xYXQl7auoPZGVTVQTTfjeQ+ZKAXPvY+YCTUh1PAic1po5yDlK+lfzZyvZ4h6H8D6KU3Nms9DrME9afq1p3VxZUg4lLkezgWZONkqYLEKJPMGmX0cgvDHDUhSSlHIDtieP5kQA+aFVMlD66ipdmiDk3DtS+GelxtlKooveTNtwxMZnsHp2L7rCHUT1NMe0ah5Igyf5DEFo4fdECL/VhmI8DebIfQz6H7ZpyuC4CI3mztoUagaHCuhoDVQl2ExCYZmL8JKigmE1P9ZomYYe0krqQMouPAkAgyGCEPa625vXaLcc8KXlkECef7YNjViFX+4L50qYcnZqtOdMxPO3yu66JOKcD4KB+AxqunZtajRF3Q94NKIUGCZRQYmtE50=","agentUuid":"344035cd-f1cc-4fa6-8815-03d84296c7db"}}
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/system-wide.perf"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/systemcollector-24469-dl360g10plus-197.maas.sc"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/perfcmd"
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/stdsrc-14077875573749.ftrace"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24479.log"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/system-wide.stat.perf"
{"error":"Unexpected error"}{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/map.events"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/amplxe-perf.24519.log"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/sep.events"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/perfcmd"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/map.events"
{"error":"Unexpected error"}{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/systemcollector-8750-worker-node-de1.sc"
Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8765.log"
{"error":"Unexpected error"}Finished uploading "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7/data.0/KVM/amplxe-perf.8780.log"
Deleting the result directory: "/tmp/vtune-agent/./cache/72a4-3982-04c6-c4c7"
Finished traces upload

 

 

0 Kudos
8 Replies
VaradJ_Intel
Moderator
1,587 Views

Hi,


Good day to you.

                       

Thanks for posting in Intel Communities.


Are you trying to profile an application which is running inside remote server(KVM)?


If yes you can try remote server profiling using the steps given below:


1. Password less ssh.


2. run "vtune -target-system ssh:<user_name>@ip (example user@192.168.1.1) -collect <analysis name> <path to the binary>"


For detailed description of this method you can follow this link: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/remote-linux-target-setup.html


Also, please can you provide us the the below details:


1. Processor details 

2. OS details 

3. Kernel version 

4. Exact steps to reproduce the issue(if possible screenshots) 


Thank You!


Regards,

Varad


0 Kudos
Gust
Beginner
1,578 Views

Hi Varad,

 

I'm trying to "Profile system" not a single application.

 

Steps:

1. Start VTune webserver in host machine: ./vtune-server --allow-remote-access

2. Start agent in host machine: ./vtune-agent --uuid=344035cd-f1cc-4fa6-8815-03d84296c7db --admission-key-path=/root/admission.key --deployment-user=passphrase_authenticated_user --shared --log-level trace

3. Run the following:

Gust_0-1682079726321.png

 

I've tried without the kvm collector and I get the same error in the agent logs.

 

There's one thing that might be worth mentioning. I can't run the agent automatically, only if I manually run does it connect.

 

CPU: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz

OS: Rocky Linux release 8.6 (Green Obsidian)

Kernel: 4.18.0-372.19.1.el8_6.x86_64

 

Gustavo

0 Kudos
VaradJ_Intel
Moderator
1,486 Views

Hi,

 

Good day to you.

 

I'm trying to "Profile system" not a single application

 

The above suggestion which we provided is also applicable to "Profile System".

 

Please can you try using vtune-backend instead of vtune-server as it is causing issue.

 

Here are detailed step for vtune-backend method:

 

1. Set up password less ssh by following the below link:

 

https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/configuring-ssh-access-for-remote-collection.html

 

2. Next step is to do port forwarding while connecting to the system:

 

Example : ssh -L 127.0.0.1:7000:127.0.0.1:7000 username@ip

 

Note: Here, we are port forwarding to the port 7000

 

3. Run Vtune backend using the below given command:

 

vtune-backend --allow-remote-access --enable-server-profiling --web-port 7000

 

4. Now, Paste that link into the browser to get VTune GUI

 

5. Click on configure analysis and in "where" pane select "Remote Linux (SSH)" option.

 

VaradJ_Intel_0-1682690589080.png

 

6. Fill the details about the remote server and deploy the agent.

 

7. After that in "what" pane select "Profile System" and fill the fields which are required in the "advanced" section

 

VaradJ_Intel_1-1682690735104.png

 

VaradJ_Intel_0-1682692537510.png

 

8. Now select the analysis type and click on run.

 

Please do let us know if you still face any issues.

 

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

 

Thank You!

 

Regards,

Varad

 

0 Kudos
Gust
Beginner
1,406 Views

Hi Varad,

 

Sorry for the late reply.

 

vtune-backend works fine, nothing else needed.

Is server deprecated or something?

 

Gustavo

0 Kudos
VaradJ_Intel
Moderator
1,275 Views

Hi,


Good day to you.


Is server deprecated or something?


Yes, vtune-server is now changed to vtune-backend. You could use vtune-backend for remote profiling from now onwards.


Glad to know that your issue is resolved. 


Could you please confirm if we can close this thread as your issue is resolved?


Thank You!


Regards,

Varad


0 Kudos
VaradJ_Intel
Moderator
1,203 Views

Hi,


Good day to you.


We have not heard back from you. Could you please confirm if we can close this thread?


Thank You!


Regards,

Varad



0 Kudos
Gust
Beginner
1,198 Views

Hi Varad,

 

Sorry, missed the other email.

 

The issue is closed, with vtune-backend never faced any issues.

 

Gustavo

0 Kudos
VaradJ_Intel
Moderator
918 Views

Hi,


Good day to you.


Thanks for confirming.


If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.


Thank You!


Regards,

Varad


0 Kudos
Reply