Re: Intel VTune very long time for finalizing results.

tsocha_intel · ‎08-30-2023

Hi,
I try to use VTune to profile a program on a remote server.
I can successfully run and collect results but after that finalizing takes forever.
Most of the time it hangs on locating files on the remote system.

Do you know how I can speed-up this process?

AlekhyaV_Intel · ‎08-31-2023

Hi,

Thank you for posting in Intel Communities.

To debug your issue further, please let us know the below details:

VTune version. Is it the latest i.e. 2023.2.0?
System details like OS version, processor, etc.
Please try a different analysis and let us know if the issue persists.
Try to profile via VTune CLI and see if the VTune finalization still hangs or not?

Meanwhile, could you please try the below suggestions?

Reboot your target machine and see if this issue persists.
Try selecting Fast finalization or No finalization and see if you still see the delay/hang. Please follow this documentation(https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-1/finalization-mode.html)

Regards,

Alekhya

tsocha_intel · ‎08-31-2023

Ad. 1
VTune version: 2023.2.0 build: 626047

Ad. 2
Host system: Windows 11 Enterprise version: 10.0.22621 Build 22621
CPU: i7-1185G7
RAM: 32GB

Remote system: Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-67-generic x86_64)
CPU: Intel(R) Xeon(R) Gold 6342 CPU
RAM: 125 GB

Ad. 3
Performance snapshot is fast but collecting hotspots take forever.
(Current run is in progress for 3 hours already, it shows 45% on progress bar).
The machine has been rebooted.

Ad. 4 I have never used it via CLI, should I try to run it on host or remote system?
=============================================================
Ad. 1 Reboot does not help, I think that when driver sampling is disabled(user was lacking vtune group) it's faster but I switched to sampling driver for more information.
Ad. 2 I will try that.

tsocha_intel · ‎08-31-2023

I forgot to add logs from current Hotspots analysis:
Local time: 13:51

Finalizing results
Aug 31 2023 10:43:59
The database has been cleared, elapsed time is 0.584 seconds.
Raw data has been loaded to the database, elapsed time is 3631.073 seconds.
Data transformations have been finished, elapsed time is 0.072 seconds.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\mtrand.cpython-310-x86_64-linux-gnu.so\628424a4536048312d0cf611fb94b3ed\mtrand.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_stats.cpython-310-x86_64-linux-gnu.so\4e519117c26d5e057d611f4ee124a696\_stats.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libigdgmm.so.12\07d0f94d140aa785178515b90f4b1ba2\libigdgmm.so.12'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libstdc++.so.6\26b3d4ec128b9b6716dd4b8f1c205f9f\libstdc++.so.6'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libtpsstool.so\84168aec286b27cd2e395bff999f1adb\libtpsstool.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libc.so.6\3d7240354d70ebbd11911187f1acd6e8\libc.so.6'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libopencl-clang.so.10\4ab1a6727dd21f7e5804de9b78fd3135\libopencl-clang.so.10'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_generator.cpython-310-x86_64-linux-gnu.so\65a8be145b30683a9581b43237036d77\_generator.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_bspl.cpython-310-x86_64-linux-gnu.so\ad87e86a85c2f4fd1ad45a061532ae67\_bspl.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libgcc_s.so.1\9f5ce810d5a4e1ab84dd9ae019e8fb35\libgcc_s.so.1'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_multiarray_tests.cpython-310-x86_64-linux-gnu.so\f660d63399ee756d1cd420e621c9563f\_multiarray_tests.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_distance_pybind.cpython-310-x86_64-linux-gnu.so\623da653e96e25de4d768427565f6b7a\_distance_pybind.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_pcg64.cpython-310-x86_64-linux-gnu.so\874a7482e6dd439e6932e1e95f8743ad\_pcg64.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_rotation.cpython-310-x86_64-linux-gnu.so\b699eb3cb6c01c0c52ff0be335a4a949\_rotation.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libze_loader.so.1\37d43042e7bd05c24f2f42f7b5240446\libze_loader.so.1'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_cythonized_array_utils.cpython-310-x86_64-linux-gnu.so\223f7d82893b90d1cd39bed01fd34544\_cythonized_array_utils.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libigdrcl.so\de10e7347cf0249842f650004a12a01b\libigdrcl.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_ckdtree.cpython-310-x86_64-linux-gnu.so\6a4011a87aba87446349e2041596fe75\_ckdtree.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_minpack2.cpython-310-x86_64-linux-gnu.so\19709904c98f4f775434e559e432ad33\_minpack2.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libze_intel_gpu.so.1\fa6e5c0264b20ca633f0d1a481df71f8\libze_intel_gpu.so.1'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\lscpu\eed5b34700c7eab9e3e072ea040d0542\lscpu'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libmkl_intel_ilp64.so.2\beafc60e2ca1c8db7b52e9fc43e4cceb\libmkl_intel_ilp64.so.2'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\_multiarray_umath.cpython-310-x86_64-linux-gnu.so\f4cc82a8d2295878eb400c9f5cc9eabd\_multiarray_umath.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libigdfcl.so.1\0c040e7bd09af6d9eb8541ba8d6f11ad\libigdfcl.so.1'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libopenblas64_p-r0-5007b62f.3.23.dev.so\a3eb0052d354386f8271ca7bbef44d2f\libopenblas64_p-r0-5007b62f.3.23.dev.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\bit_generator.cpython-310-x86_64-linux-gnu.so\e2376df28829ab3de9b8728d39be7a93\bit_generator.cpython-310-x86_64-linux-gnu.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libopenblasp-r0-23e5df77.3.21.dev.so\559a0ff7cac141f6e4a227f83b98573d\libopenblasp-r0-23e5df77.3.21.dev.so'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\ld-linux-x86-64.so.2\1489a4c7bedc2ff50ad13ad1bb43a225\ld-linux-x86-64.so.2'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\libigc.so.1\a1a3f966603c1584b9201a78a77ff78d\libigc.so.1'.
Cannot locate debugging information for file `C:\Users\tsocha\AppData\Local\Temp\amplxe-tmp-tsocha\modules.ssh_tsocha@10.211.177.232\python3.10\fc51071a3dfb6ba6beb4a3f5f5b51532\python3.10'.
Resolving information for `python3.10'

tsocha_intel · ‎08-31-2023

Fast finalization was selected by default.
None finalization was fast but now I don't know how to display any results.
Analysis Configuration tab have inactive re-finalize button:

AlekhyaV_Intel · ‎09-07-2023

Hi,

Based on this, and our internal discussion about creating a priority ticket if needed, I am going ahead and closing this case for now.

Please feel free to contact us further if you need any assistance from our side.

Regards,

Alekhya