vtune crashing as of 2022.2.0: Failed to start threads: File exists

[emilberg]

Hello!

I'm having problem with vtune crashing in version 2022.2.0. I did not have these problems with version 2022.1.0.

vtune command:

vtune -collect hotspots -knob sampling-mode=hw -result-dir /tmp/logs/vtune <executable>

cat /etc/*-release output:

NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp3"
DOCUMENTATION_URL="https://documentation.suse.com/"

cat /proc/version output:

Linux version 4.18.0-305.19.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1-1) (GCC)) #1 SMP Tue Sep 7 07:07:31 EDT 2021

 Error log:

vtune: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel module symbols.
vtune: Warning: To analyze modules at the kernel level in this configuration, load the Intel sampling driver and set an unlimited (0) value for the Stack size option (if you require stack collection). Alternatively, enable access to kernel-mode monitoring by setting the /proc/sys/kernel/perf_event_paranoid value to 1 or lower.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
vtune: Warning: Consider reducing possible collection overhead by setting the /proc/sys/kernel/perf_event_paranoid value to 0 (or less).
vtune: Error: amplxe-perf:
threads_spec: cpu
Using CPUID GenuineIntel-6-55-7
path: /tmp/logs/vtune/data.0
dir-data-file: 1667.perf
dir-reuse: true
Compression enabled, disabling build id collection at the end of the session.
nr_threads: 112
thread_masks[0]: 0x558fd7f36310: maps mask[112]: 0
thread_masks[0]: 0x558fd7f36320: affinity mask[112]: 0
thread_masks[1]: 0x558fd7f36330: maps mask[112]: 1
thread_masks[1]: 0x558fd7f36340: affinity mask[112]: 1
thread_masks[2]: 0x558fd7f36350: maps mask[112]: 2
thread_masks[2]: 0x558fd7f36360: affinity mask[112]: 2
thread_masks[3]: 0x558fd7f36370: maps mask[112]: 3
thread_masks[3]: 0x558fd7f36380: affinity mask[112]: 3
thread_masks[4]: 0x558fd7f36390: maps mask[112]: 4
thread_masks[4]: 0x558fd7f363a0: affinity mask[112]: 4
thread_masks[5]: 0x558fd7f363b0: maps mask[112]: 5
thread_masks[5]: 0x558fd7f363c0: affinity mask[112]: 5
thread_masks[6]: 0x558fd7f363d0: maps mask[112]: 6
thread_masks[6]: 0x558fd7f363e0: affinity mask[112]: 6
thread_masks[7]: 0x558fd7f363f0: maps mask[112]: 7
thread_masks[7]: 0x558fd7f36400: affinity mask[112]: 7
thread_masks[8]: 0x558fd7f36410: maps mask[112]: 8
thread_masks[8]: 0x558fd7f36420: affinity mask[112]: 8
thread_masks[9]: 0x558fd7f36430: maps mask[112]: 9
thread_masks[9]: 0x558fd7f36440: affinity mask[112]: 9
thread_masks[10]: 0x558fd7f36450: maps mask[112]: 10
thread_masks[10]: 0x558fd7f36460: affinity mask[112]: 10
thread_masks[11]: 0x558fd7f36470: maps mask[112]: 11
thread_masks[11]: 0x558fd7f36480: affinity mask[112]: 11
thread_masks[12]: 0x558fd7f36490: maps mask[112]: 12
thread_masks[12]: 0x558fd7f364a0: affinity mask[112]: 12
thread_masks[13]: 0x558fd7f364b0: maps mask[112]: 13
thread_masks[13]: 0x558fd7f364c0: affinity mask[112]: 13
thread_masks[14]: 0x558fd7f364d0: maps mask[112]: 14
thread_masks[14]: 0x558fd7f364e0: affinity mask[112]: 14
thread_masks[15]: 0x558fd7f364f0: maps mask[112]: 15
thread_masks[15]: 0x558fd7f36500: affinity mask[112]: 15
thread_masks[16]: 0x558fd7f36510: maps mask[112]: 16
thread_masks[16]: 0x558fd7f36520: affinity mask[112]: 16
thread_masks[17]: 0x558fd7f36530: maps mask[112]: 17
thread_masks[17]: 0x558fd7f36540: affinity mask[112]: 17
thread_masks[18]: 0x558fd7f36550: maps mask[112]: 18
thread_masks[18]: 0x558fd7f36560: affinity mask[112]: 18
thread_masks[19]: 0x558fd7f36570: maps mask[112]: 19
thread_masks[19]: 0x558fd7f36580: affinity mask[112]: 19
thread_masks[20]: 0x558fd7f36590: maps mask[112]: 20
thread_masks[20]: 0x558fd7f365a0: affinity mask[112]: 20
thread_masks[21]: 0x558fd7f365b0: maps mask[112]: 21
thread_masks[21]: 0x558fd7f365c0: affinity mask[112]: 21
thread_masks[22]: 0x558fd7f365d0: maps mask[112]: 22
thread_masks[22]: 0x558fd7f365e0: affinity mask[112]: 22
thread_masks[23]: 0x558fd7f365f0: maps mask[112]: 23
thread_masks[23]: 0x558fd7f36600: affinity mask[112]: 23
thread_masks[24]: 0x558fd7f36610: maps mask[112]: 24
thread_masks[24]: 0x558fd7f36620: affinity mask[112]: 24
thread_masks[25]: 0x558fd7f36630: maps mask[112]: 25
thread_masks[25]: 0x558fd7f36640: affinity mask[112]: 25
thread_masks[26]: 0x558fd7f36650: maps mask[112]: 26
thread_masks[26]: 0x558fd7f36660: affinity mask[112]: 26
thread_masks[27]: 0x558fd7f36670: maps mask[112]: 27
thread_masks[27]: 0x558fd7f36680: affinity mask[112]: 27
thread_masks[28]: 0x558fd7f36690: maps mask[112]: 28
thread_masks[28]: 0x558fd7f366a0: affinity mask[112]: 28
thread_masks[29]: 0x558fd7f366b0: maps mask[112]: 29
thread_masks[29]: 0x558fd7f366c0: affinity mask[112]: 29
thread_masks[30]: 0x558fd7f366d0: maps mask[112]: 30
thread_masks[30]: 0x558fd7f366e0: affinity mask[112]: 30
thread_masks[31]: 0x558fd7f366f0: maps mask[112]: 31
thread_masks[31]: 0x558fd7f36700: affinity mask[112]: 31
thread_masks[32]: 0x558fd7f36710: maps mask[112]: 32
thread_masks[32]: 0x558fd7f36720: affinity mask[112]: 32
thread_masks[33]: 0x558fd7f36730: maps mask[112]: 33
thread_masks[33]: 0x558fd7f36740: affinity mask[112]: 33
thread_masks[34]: 0x558fd7f36750: maps mask[112]: 34
thread_masks[34]: 0x558fd7f36760: affinity mask[112]: 34
thread_masks[35]: 0x558fd7f36770: maps mask[112]: 35
thread_masks[35]: 0x558fd7f36780: affinity mask[112]: 35
thread_masks[36]: 0x558fd7f36790: maps mask[112]: 36
thread_masks[36]: 0x558fd7f367a0: affinity mask[112]: 36
thread_masks[37]: 0x558fd7f367b0: maps mask[112]: 37
thread_masks[37]: 0x558fd7f367c0: affinity mask[112]: 37
thread_masks[38]: 0x558fd7f367d0: maps mask[112]: 38
thread_masks[38]: 0x558fd7f367e0: affinity mask[112]: 38
thread_masks[39]: 0x558fd7f367f0: maps mask[112]: 39
thread_masks[39]: 0x558fd7f36800: affinity mask[112]: 39
thread_masks[40]: 0x558fd7f36810: maps mask[112]: 40
thread_masks[40]: 0x558fd7f36820: affinity mask[112]: 40
thread_masks[41]: 0x558fd7f36830: maps mask[112]: 41
thread_masks[41]: 0x558fd7f36840: affinity mask[112]: 41
thread_masks[42]: 0x558fd7f36850: maps mask[112]: 42
thread_masks[42]: 0x558fd7f36860: affinity mask[112]: 42
thread_masks[43]: 0x558fd7f36870: maps mask[112]: 43
thread_masks[43]: 0x558fd7f36880: affinity mask[112]: 43
thread_masks[44]: 0x558fd7f36890: maps mask[112]: 44
thread_masks[44]: 0x558fd7f368a0: affinity mask[112]: 44
thread_masks[45]: 0x558fd7f368b0: maps mask[112]: 45
thread_masks[45]: 0x558fd7f368c0: affinity mask[112]: 45
thread_masks[46]: 0x558fd7f368d0: maps mask[112]: 46
thread_masks[46]: 0x558fd7f368e0: affinity mask[112]: 46
thread_masks[47]: 0x558fd7f368f0: maps mask[112]: 47
thread_masks[47]: 0x558fd7f36900: affinity mask[112]: 47
thread_masks[48]: 0x558fd7f36910: maps mask[112]: 48
thread_masks[48]: 0x558fd7f36920: affinity mask[112]: 48
thread_masks[49]: 0x558fd7f36930: maps mask[112]: 49
thread_masks[49]: 0x558fd7f36940: affinity mask[112]: 49
thread_masks[50]: 0x558fd7f36950: maps mask[112]: 50
thread_masks[50]: 0x558fd7f36960: affinity mask[112]: 50
thread_masks[51]: 0x558fd7f36970: maps mask[112]: 51
thread_masks[51]: 0x558fd7f36980: affinity mask[112]: 51
thread_masks[52]: 0x558fd7f36990: maps mask[112]: 52
thread_masks[52]: 0x558fd7f369a0: affinity mask[112]: 52
thread_masks[53]: 0x558fd7f369b0: maps mask[112]: 53
thread_masks[53]: 0x558fd7f369c0: affinity mask[112]: 53
thread_masks[54]: 0x558fd7f369d0: maps mask[112]: 54
thread_masks[54]: 0x558fd7f369e0: affinity mask[112]: 54
thread_masks[55]: 0x558fd7f369f0: maps mask[112]: 55
thread_masks[55]: 0x558fd7f36a00: affinity mask[112]: 55
thread_masks[56]: 0x558fd7f36a10: maps mask[112]: 56
thread_masks[56]: 0x558fd7f36a20: affinity mask[112]: 56
thread_masks[57]: 0x558fd7f36a30: maps mask[112]: 57
thread_masks[57]: 0x558fd7f36a40: affinity mask[112]: 57
thread_masks[58]: 0x558fd7f36a50: maps mask[112]: 58
thread_masks[58]: 0x558fd7f36a60: affinity mask[112]: 58
thread_masks[59]: 0x558fd7f36a70: maps mask[112]: 59
thread_masks[59]: 0x558fd7f36a80: affinity mask[112]: 59
thread_masks[60]: 0x558fd7f36a90: maps mask[112]: 60
thread_masks[60]: 0x558fd7f36aa0: affinity mask[112]: 60
thread_masks[61]: 0x558fd7f36ab0: maps mask[112]: 61
thread_masks[61]: 0x558fd7f36ac0: affinity mask[112]: 61
thread_masks[62]: 0x558fd7f36ad0: maps mask[112]: 62
thread_masks[62]: 0x558fd7f36ae0: affinity mask[112]: 62
thread_masks[63]: 0x558fd7f36af0: maps mask[112]: 63
thread_masks[63]: 0x558fd7f36b00: affinity mask[112]: 63
thread_masks[64]: 0x558fd7f36b10: maps mask[112]: 64
thread_masks[64]: 0x558fd7f36b20: affinity mask[112]: 64
thread_masks[65]: 0x558fd7f36b30: maps mask[112]: 65
thread_masks[65]: 0x558fd7f36b40: affinity mask[112]: 65
thread_masks[66]: 0x558fd7f36b50: maps mask[112]: 66
thread_masks[66]: 0x558fd7f36b60: affinity mask[112]: 66
thread_masks[67]: 0x558fd7f36b70: maps mask[112]: 67
thread_masks[67]: 0x558fd7f36b80: affinity mask[112]: 67
thread_masks[68]: 0x558fd7f36b90: maps mask[112]: 68
thread_masks[68]: 0x558fd7f36ba0: affinity mask[112]: 68
thread_masks[69]: 0x558fd7f36bb0: maps mask[112]: 69
thread_masks[69]: 0x558fd7f36bc0: affinity mask[112]: 69
thread_masks[70]: 0x558fd7f36bd0: maps mask[112]: 70
thread_masks[70]: 0x558fd7f36be0: affinity mask[112]: 70
thread_masks[71]: 0x558fd7f36bf0: maps mask[112]: 71
thread_masks[71]: 0x558fd7f36c00: affinity mask[112]: 71
thread_masks[72]: 0x558fd7f36c10: maps mask[112]: 72
thread_masks[72]: 0x558fd7f36c20: affinity mask[112]: 72
thread_masks[73]: 0x558fd7f36c30: maps mask[112]: 73
thread_masks[73]: 0x558fd7f36c40: affinity mask[112]: 73
thread_masks[74]: 0x558fd7f36c50: maps mask[112]: 74
thread_masks[74]: 0x558fd7f36c60: affinity mask[112]: 74
thread_masks[75]: 0x558fd7f36c70: maps mask[112]: 75
thread_masks[75]: 0x558fd7f36c80: affinity mask[112]: 75
thread_masks[76]: 0x558fd7f36c90: maps mask[112]: 76
thread_masks[76]: 0x558fd7f36ca0: affinity mask[112]: 76
thread_masks[77]: 0x558fd7f36cb0: maps mask[112]: 77
thread_masks[77]: 0x558fd7f36cc0: affinity mask[112]: 77
thread_masks[78]: 0x558fd7f36cd0: maps mask[112]: 78
thread_masks[78]: 0x558fd7f36ce0: affinity mask[112]: 78
thread_masks[79]: 0x558fd7f36cf0: maps mask[112]: 79
thread_masks[79]: 0x558fd7f36d00: affinity mask[112]: 79
thread_masks[80]: 0x558fd7f36d10: maps mask[112]: 80
thread_masks[80]: 0x558fd7f36d20: affinity mask[112]: 80
thread_masks[81]: 0x558fd7f36d30: maps mask[112]: 81
thread_masks[81]: 0x558fd7f36d40: affinity mask[112]: 81
thread_masks[82]: 0x558fd7f36d50: maps mask[112]: 82
thread_masks[82]: 0x558fd7f36d60: affinity mask[112]: 82
thread_masks[83]: 0x558fd7f36d70: maps mask[112]: 83
thread_masks[83]: 0x558fd7f36d80: affinity mask[112]: 83
thread_masks[84]: 0x558fd7f36d90: maps mask[112]: 84
thread_masks[84]: 0x558fd7f36da0: affinity mask[112]: 84
thread_masks[85]: 0x558fd7f36db0: maps mask[112]: 85
thread_masks[85]: 0x558fd7f36dc0: affinity mask[112]: 85
thread_masks[86]: 0x558fd7f36dd0: maps mask[112]: 86
thread_masks[86]: 0x558fd7f36de0: affinity mask[112]: 86
thread_masks[87]: 0x558fd7f36df0: maps mask[112]: 87
thread_masks[87]: 0x558fd7f36e00: affinity mask[112]: 87
thread_masks[88]: 0x558fd7f36e10: maps mask[112]: 88
thread_masks[88]: 0x558fd7f36e20: affinity mask[112]: 88
thread_masks[89]: 0x558fd7f36e30: maps mask[112]: 89
thread_masks[89]: 0x558fd7f36e40: affinity mask[112]: 89
thread_masks[90]: 0x558fd7f36e50: maps mask[112]: 90
thread_masks[90]: 0x558fd7f36e60: affinity mask[112]: 90
thread_masks[91]: 0x558fd7f36e70: maps mask[112]: 91
thread_masks[91]: 0x558fd7f36e80: affinity mask[112]: 91
thread_masks[92]: 0x558fd7f36e90: maps mask[112]: 92
thread_masks[92]: 0x558fd7f36ea0: affinity mask[112]: 92
thread_masks[93]: 0x558fd7f36eb0: maps mask[112]: 93
thread_masks[93]: 0x558fd7f36ec0: affinity mask[112]: 93
thread_masks[94]: 0x558fd7f36ed0: maps mask[112]: 94
thread_masks[94]: 0x558fd7f36ee0: affinity mask[112]: 94
thread_masks[95]: 0x558fd7f36ef0: maps mask[112]: 95
thread_masks[95]: 0x558fd7f36f00: affinity mask[112]: 95
thread_masks[96]: 0x558fd7f36f10: maps mask[112]: 96
thread_masks[96]: 0x558fd7f36f20: affinity mask[112]: 96
thread_masks[97]: 0x558fd7f36f30: maps mask[112]: 97
thread_masks[97]: 0x558fd7f36f40: affinity mask[112]: 97
thread_masks[98]: 0x558fd7f36f50: maps mask[112]: 98
thread_masks[98]: 0x558fd7f36f60: affinity mask[112]: 98
thread_masks[99]: 0x558fd7f36f70: maps mask[112]: 99
thread_masks[99]: 0x558fd7f36f80: affinity mask[112]: 99
thread_masks[100]: 0x558fd7f36f90: maps mask[112]: 100
thread_masks[100]: 0x558fd7f36fa0: affinity mask[112]: 100
thread_masks[101]: 0x558fd7f36fb0: maps mask[112]: 101
thread_masks[101]: 0x558fd7f36fc0: affinity mask[112]: 101
thread_masks[102]: 0x558fd7f36fd0: maps mask[112]: 102
thread_masks[102]: 0x558fd7f36fe0: affinity mask[112]: 102
thread_masks[103]: 0x558fd7f36ff0: maps mask[112]: 103
thread_masks[103]: 0x558fd7f37000: affinity mask[112]: 103
thread_masks[104]: 0x558fd7f37010: maps mask[112]: 104
thread_masks[104]: 0x558fd7f37020: affinity mask[112]: 104
thread_masks[105]: 0x558fd7f37030: maps mask[112]: 105
thread_masks[105]: 0x558fd7f37040: affinity mask[112]: 105
thread_masks[106]: 0x558fd7f37050: maps mask[112]: 106
thread_masks[106]: 0x558fd7f37060: affinity mask[112]: 106
thread_masks[107]: 0x558fd7f37070: maps mask[112]: 107
thread_masks[107]: 0x558fd7f37080: affinity mask[112]: 107
thread_masks[108]: 0x558fd7f37090: maps mask[112]: 108
thread_masks[108]: 0x558fd7f370a0: affinity mask[112]: 108
thread_masks[109]: 0x558fd7f370b0: maps mask[112]: 109
thread_masks[109]: 0x558fd7f370c0: affinity mask[112]: 109
thread_masks[110]: 0x558fd7f370d0: maps mask[112]: 110
thread_masks[110]: 0x558fd7f370e0: affinity mask[112]: 110
thread_masks[111]: 0x558fd7f370f0: maps mask[112]: 111
thread_masks[111]: 0x558fd7f37100: affinity mask[112]: 111
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 1
Problems creating module maps, continuing anyway...
mmap size 528384B
Failed to start threads: File exists
failed to write feature HYBRID_TOPOLOGY
failed to write feature HYBRID_CPU_PMU_CAPS
[ perf record: Captured and wrote 0.023 MB /tmp/logs/vtune/data.0 ]

vtune: Collection failed.
vtune: Internal Error

[JaideepK_Intel]

Hi,

Thank you for posting in Intel Communities.

We ran hardware event-based sampling from our side on RHEL 8 and also in SLES 15.3 with the latest version of Vtune (2022.3) and it was working fine. Please go through the below screenshots.

Could you please try the latest version of vtune and let us know?

download the latest version of Vtune: Download the Intel® VTune™ Profiler

RHEL8:

SLES15.3:

If the issue still persists, please provide the below details.

1. Processor Information

2. Vtune2022.3 self-checker log (latest)

<Vtune_installation_directory\latest\bin64\vtune-self-checker.bat> for windows.
<Vtune_installation_directory\latest\bin64\vtune-self-checker.sh> for linux.

3. Sample reproducer 

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

Thanks,

Jaideep

[emilberg]

I tried 2022.3.0 and I had the same problem. Note that we are running the application in a container.

CPU: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz

Output from self checker is attached.

Sample reproducer (I guess any program will suffice):

#include <unistd.h>

int main() {
    while (1) {
        sleep(1);
    }
}

[JaideepK_Intel]

Hi,

I assume you are running the Vtune profiler and your application inside a Docker container. Please follow the below commands before starting a Docker container.

Run docker image with the below command.

docker run --rm -dti --cap-add CAP_SYS_ADMIN <docker_imagename>

To get the container id.

docker ps

To access the container run the below command.

docker exec -it <container id> bash

We ran hardware event-based sampling from our side inside a Docker container with the latest version of Vtune.

If you want to try container-to-container profiling, check the below link.

https://community.intel.com/t5/Analyzers/Vtune-Stops-the-container-when-generating-hotspots-result-for/m-p/1353002#M21785

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

Thanks,

Jaideep

[emilberg]

I validated that we had the CAP_SYS_ADMIN capability already with

kubectl <pod> -- capsh --print

[JaideepK_Intel]

Hi,

We are working on this internally and will get back to you with an update.

Thanks,

Jaideep

[emilberg]

Please note that the

Failed to start threads: File exists

message may be a bug in perf. 

In perf, look at the changes to tools/perf/builtin-record.c made Feb 10 2022 with:

perf record: Start threads in the beginning of trace streaming

SHA: 3217e9fecf118d5dcabdd68d91e0c6afcb4c3e1b

At line 2014 pthread_create() is run and on line 2017 strerror(errno) is printed. Between line 2014 and 2017 record__terminate_thread() is run.

Most likely record__terminate_thread() run in-between the creation and the error message looks like it may overwrite errno, thus messing up the error message. 

if (pthread_create(&handle, &attrs, record__thread, &thread_data[t])) {
    for (tt = 1; tt < t; tt++)
        record__terminate_thread(&thread_data[t]);
    pr_err("Failed to start threads: %s\n", strerror(errno));

Just so you know the error message may be misleading.

[emilberg]

Any updates here? This problem is still apparent in 2022.4.

[Vinutha_SV]

Hi ,

You are using perf. Can you please install SEP drivers and then try running hardware event based analysis on the same application?

[Vinutha_SV]

Hi,

I hope you were able to resolve the issue with the recommendations provided. As I have not received the response, we shall stop monitoring this thread.