Stock Python 3.8.12 and 3.9.7 cannot be profiled by VTune 2021.8.0

[HJ77MyGs]

This is a follow-up to my previous thread, where I was eventually able to report some limited success on a toy script using a recent version of VTune. However, I have recently downloaded the newest version of VTune (2021.8), and Python profiling once again does not function.

I have migrated to a CentOS 8 environment, which could possibly a factor here, although I don't think so (see below). I have installed the most recent version of the oneAPI base and HPC toolkits, and manually upgraded VTune to 2021.8.0.  I am using the example script from the previous thread:

import numpy as np

def do_work(n):
    arr = np.random.random(size=(n, n))
    for i in range(100):
        arr = np.dot(arr, arr)

def main():
    do_work(2000)

if __name__ == '__main__':
    main()

I have tried to profile this script within clean virtualenvs based on Python 3.8.12 and 3.9.7, both built from standard tars available from the cpython github. Neither of these work at all, under any circumstances. I can always successfully profile the underlying binary, but Python source is never attributed and displayed. Note that in the previous thread, it was explicitly claimed that Python 3.8 at least is supported for current version of VTune, so I would have every reason to expect this to work.

As a control, I can confirm that I can profile this script using the binary Intel Python from oneAPI Base 2021.4.0, without any other changes to settings etc. -- which strongly suggests to me that the change in Linux distribution is not at fault here, and also that my settings are sensible. Note also that I'm not doing anything different here to that which produced the partial success that I reported at the end of the last thread.

As a comment: if anybody from the dev team wants to reach out to discuss this away from the forum, I am happy to be contacted directly at the email address linked to this account. I would like very much to help work this out, and I doubt that I'm the only person encountering these problems.

[HJ77MyGs]

As a further comment: just out of curiosity, I rebuilt Python 3.7.9 from scratch, to see if it also failed here. It does not fail -- rather, it runs as it did on my previous Ubuntu installation. This means that it can successfully profile the above script (with Python attribution), and also hangs and crashes for no clear reason, and leaves various random processes running in the background when it does.

[JaideepK_Intel]

Hi,

Thank you for posting in Intel forums.

We're working on this internally. We'll get back to you soon with an update.

Thanks,

Jaideep

[HJ77MyGs]

As this post is now two weeks old, I would like to politely enquire if there has been any progress? Can you replicate this failure? Is there any further information I can provide in order to assist?

Thanks!

[JaideepK_Intel]

Hi,

Sorry for the delay, we have tried the above code in VTune 2021.8.0 and found that there is an issue with importing numpy using python 3.8.12 and 3.9.7 on centos8. This issue is resolved in VTune 2021.9.0. We have verified that the above code is working with python 3.8.12 and python 3.9.7 in centos8 using VTune 2021.9.0 .

Please try using VTune 2021.9.0 and let us know if the issue persists.

Thanks,

Jaideep

[JaideepK_Intel]

Hi,

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue. Thank you!

Thanks,

Jaideep

[HJ77MyGs]

Thanks for the update. I have installed VTune 2021.9.0, and unfortunately this does not resolve the issue when tested with Python 3.9.7. (I have not tested with 3.8.x.) As before, VTune can execute the given script and profile it at a C level, but does not attribute anything to Python source.

If this is working for you, could you please give a precise description of exactly what settings you are using to execute and profile the script? Perhaps I am making an elementary configuration error here?

[JaideepK_Intel]

Hi,

As mentioned in my previous response, while trying to profile the given python script, I had faced an issue issue while importing numpy when profiling the python script with vtune2021.8(centos 8), this issue was resolved in 2021.9 I was just pointing that out. Sorry for the confusion caused.

I have verified the issue with source code attribution from my end, I could observe the python source is not getting displayed. This has been reported to the development team, we will keep you posted on the updates

Thanks,

Jaideep

[HJ77MyGs]

No worries, thanks for the clarification and feeding this back to the dev team.

Just to make sure that we're understanding each other properly, do I understand you correctly as saying that you can also reproduce the bug in 2021.9, and that the numpy problem you mentioned was a completely separate issue?

[JaideepK_Intel]

Hi HJ77MyGs,

Yes, both are different issues

Thanks,

Jaideep

[JaideepK_Intel]

Hi,

With the latest version of Vtune profiler(2022.3), we can view both assembly and Python source code. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Download link:

Download the Intel® VTune™ Profiler

Thanks,

Jaideep