ResNet50 Workload - Cannot locate debugging symbols

[Anonymous102]

Hi,

I am trying to get the VTune hotspots for a ResNet-50 workload written in Python. I am running the latest VTune (2020 version) on Ubuntu 18.04.

A few of the warnings seen are as follows:

vtune: Warning: Cannot locate file `<frozen importlib._bootstrap>'.
vtune: Warning: Cannot locate file `<frozen importlib._bootstrap_external>'.
vtune: Warning: Cannot locate file `<string>.ko'.
vtune: Warning: Cannot locate file `/tmp/tmpy52up4de.py'.
vtune: Warning: Cannot locate file `/tmp/tmpbbbewbwv.py'.
vtune: Warning: Cannot locate debugging information for file `/mnt/nvme_nfs_share/AIBench_shared_disk/Extra/vtune_profiler_2020.0.0.605129/bin64/pinbin'.

vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/h5py/h5.cpython-36m-x86_64-linux-gnu.so'.
vtune: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libpthread.so.0'.

vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so'.

vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/pandas/_libs/tslibs/conversion.cpython-36m-x86_64-linux-gnu.so'.

vtune: Warning: Cannot locate debugging information for file `/lib/x86_64-linux-gnu/libdl.so.2'.
Cannot match the module with the symbol file `/lib/x86_64-linux-gnu/libdl-2.27.so'. Make sure to specify the correct path to the symbol file in the Binary/Symbol Search list of directories.
vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/pandas/_libs/tslibs/timezones.cpython-36m-x86_64-linux-gnu.so'.

vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/pandas/_libs/tslibs/c_timestamp.cpython-36m-x86_64-linux-gnu.so'.

vtune: Warning: Cannot locate debugging information for file `/mnt/nvme_nfs_share/AIBench_shared_disk/Extra/vtune_profiler_2020.0.0.605129/lib64/libtpsstool.so'.

vtune: Warning: Cannot locate debugging information for file `/usr/local/lib/python3.6/dist-packages/_solib_k8/_U@mkl_Ulinux_S_S_Cmkl_Ulibs_Ulinux___Uexternal_Smkl_Ulinux_Slib/libiomp5.so'.
 

Please find a screenshot of the hotspots on the VTune GUI attached.

As seen form the screenshot, the most time consuming hotspot is not detected.

Please can I get some help fixing this.

[Anonymous102]

Hi,

Just an update: I installed the debug version of Python 3.6 and then ran VTune. I still see these warnings.

Please help.

[ArunJ_Intel]

If kernel pointers information was explicitly hidden by setting the kptr_restrict to a non-zero value, hardware event-based analysis results may not contain functions from kernel modules. As a result, you may see the CPU time associated with the [Outside any known module] item. To workaround this problem for the current session, set the contents of the /proc/sys/kernel/kptr_restrict file to 0 before starting the VTune Profiler as follows:

sysctl -w kernel.kptr_restrict=0

[Anonymous102]

I have already set the contents of the /proc/sys/kernel/kptr_restrict file to 0. Still I am seeing CPU time associated with [Outside any known module].

Is there anything else to fix?

[Anonymous102]

Just adding more details:

Ubuntu 18.04

Kernel Version: 5.4.13

I am unable to find the kernel compressed image with debug symbols for this kernel version.

[ArunJ_Intel]

We are forwarding your case for SME to check

Arun Jose

[Mayank_T_Intel]

Hi Neha,

Can you confirm the collector type? If it is the perf collector, then you need to set /sys/kernel/perf_event_paranoid to <=1 to profile kernel.

Thanks,

Mayank

[DrAmarpal_K_Intel]

Hi Neha,

Having not received your response we are assuming that the issue has been fixed and we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only.