Conditions for Python working/stack collection?

[ndevenish]

Over the past few years I've found the VTune support for mixed python/C++ profiling to be invaluable, but have had some frustration getting it running with more recent distributions of VTune and python.

I normally launch via a bash script as the "Application" that sets up the environment, because we have quite a lot of environment variables set to point to various libraries, python paths to run our application. This mostly worked well with python 3.6 in VTune 2019.

However, I have had significant problems getting things working with VTune 2020,21, for both 3.6 and >3.6, mainly version from conda-forge. All of 3.6+ either python throws SEGV (even with test files, so not coming from the application), or I get the "Stack size provided to sigaltstack is too small" error. When I've attached with the --no-altstack knob I've sometimes gotten 3.6 to run in VTune 2020, but not managed to gather python call stacks. The only version higher than 3.6 I've ever gotten to work with VT2020+ is the IDP==3.7, which isn't what we normally run under and I'd prefer to profile on the version/build of python we actually use (mainly 3.8).

What are the conditions under which this is supposed to work? The documentation describes the VTune python analysis as for "Python distributions of 2.6 and later".Are non-IDP distributions of python supposed to be supported? Does anyone have any suggestions on diagnostics/debugging to try, or am I likely using a Python compiled with some incompatible flags?

[ArunJ_Intel]

Hi Nicholas,

Could you provide us the below details

1)Operating system

2)Vtune analysis type you are running(please share the command line)

We tried vtune user based sampling on an ubuntu machine and the collection works without any issues for stock python(3.8.5). We used vtune 2021(oneapi gold release). We installed the python from anaconda channel. Below are the steps that did work for us

conda create –name pythonenv38

conda activate pythonenv38

conda install -c anaconda python=3.8

vtune -collect hotspots -- python samplecode.py

Please try out python from anaconda channel and with oneapi gold version of vtune and see if this works for you.

Thanks

Arun

[Denis_P_Intel · Accepted solution]

Hi Nicholas,

This is Denis from VTune team. Thank you for contacting us with your problem. Let me outline the scope of VTune's Python code profiling support. VTune does support the stock python 3.8 and IDP 3.7 for Launch mode only. Attach mode may cause the problems. The User-Mode Hotspots, Threading and Memory Consumption analysis are supported. You will not see Python user code if you run the Hardware event-based Hotspots analysis.

I will repeat Arun's questions about the operating system you are using and the type of analysis. We recently fixed an issue with similar symptoms on CentOS 8.

Thanks.

[ArunJ_Intel]

Hi,

As we have not heard back from you for sometime now we assume your issue is resolved, we wouldn't be monitoring this thread further. Please raise a new thread in case of further issues.

Thanks

Arun Jose

[ndevenish]

Hi,

Apologies - I hadn't managed to loop back round to this task yet. Knowing that it's supposed to work is the main thing - it means it's worth putting in the time to try and dissect what in my live environment is causing the problem.

Thanks