Analyzers
Support for Analyzers (Intel VTune™ Profiler, Intel Advisor, Intel Inspector)
Announcements
This community is designed for sharing of public information. Please do not share Intel or third-party confidential information here.
4676 Discussions

vtune crashing as of 2022.2.0: Failed to start threads: File exists

emilberg
Beginner
310 Views

Hello!

I'm having problem with vtune crashing in version 2022.2.0. I did not have these problems with version 2022.1.0.

vtune command:

vtune -collect hotspots -knob sampling-mode=hw -result-dir /tmp/logs/vtune <executable>

cat /etc/*-release output:

NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp3"
DOCUMENTATION_URL="https://documentation.suse.com/"

cat /proc/version output:

Linux version 4.18.0-305.19.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1-1) (GCC)) #1 SMP Tue Sep 7 07:07:31 EDT 2021

 Error log:

vtune: Warning: Access to /proc/kallsyms file is limited. Consider changing /proc/sys/kernel/kptr_restrict to 0 to enable resolution of OS kernel and kernel module symbols.
vtune: Warning: To analyze modules at the kernel level in this configuration, load the Intel sampling driver and set an unlimited (0) value for the Stack size option (if you require stack collection). Alternatively, enable access to kernel-mode monitoring by setting the /proc/sys/kernel/perf_event_paranoid value to 1 or lower.
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
vtune: Warning: Consider reducing possible collection overhead by setting the /proc/sys/kernel/perf_event_paranoid value to 0 (or less).
vtune: Error: amplxe-perf:
threads_spec: cpu
Using CPUID GenuineIntel-6-55-7
path: /tmp/logs/vtune/data.0
dir-data-file: 1667.perf
dir-reuse: true
Compression enabled, disabling build id collection at the end of the session.
nr_threads: 112
thread_masks[0]: 0x558fd7f36310: maps mask[112]: 0
thread_masks[0]: 0x558fd7f36320: affinity mask[112]: 0
thread_masks[1]: 0x558fd7f36330: maps mask[112]: 1
thread_masks[1]: 0x558fd7f36340: affinity mask[112]: 1
thread_masks[2]: 0x558fd7f36350: maps mask[112]: 2
thread_masks[2]: 0x558fd7f36360: affinity mask[112]: 2
thread_masks[3]: 0x558fd7f36370: maps mask[112]: 3
thread_masks[3]: 0x558fd7f36380: affinity mask[112]: 3
thread_masks[4]: 0x558fd7f36390: maps mask[112]: 4
thread_masks[4]: 0x558fd7f363a0: affinity mask[112]: 4
thread_masks[5]: 0x558fd7f363b0: maps mask[112]: 5
thread_masks[5]: 0x558fd7f363c0: affinity mask[112]: 5
thread_masks[6]: 0x558fd7f363d0: maps mask[112]: 6
thread_masks[6]: 0x558fd7f363e0: affinity mask[112]: 6
thread_masks[7]: 0x558fd7f363f0: maps mask[112]: 7
thread_masks[7]: 0x558fd7f36400: affinity mask[112]: 7
thread_masks[8]: 0x558fd7f36410: maps mask[112]: 8
thread_masks[8]: 0x558fd7f36420: affinity mask[112]: 8
thread_masks[9]: 0x558fd7f36430: maps mask[112]: 9
thread_masks[9]: 0x558fd7f36440: affinity mask[112]: 9
thread_masks[10]: 0x558fd7f36450: maps mask[112]: 10
thread_masks[10]: 0x558fd7f36460: affinity mask[112]: 10
thread_masks[11]: 0x558fd7f36470: maps mask[112]: 11
thread_masks[11]: 0x558fd7f36480: affinity mask[112]: 11
thread_masks[12]: 0x558fd7f36490: maps mask[112]: 12
thread_masks[12]: 0x558fd7f364a0: affinity mask[112]: 12
thread_masks[13]: 0x558fd7f364b0: maps mask[112]: 13
thread_masks[13]: 0x558fd7f364c0: affinity mask[112]: 13
thread_masks[14]: 0x558fd7f364d0: maps mask[112]: 14
thread_masks[14]: 0x558fd7f364e0: affinity mask[112]: 14
thread_masks[15]: 0x558fd7f364f0: maps mask[112]: 15
thread_masks[15]: 0x558fd7f36500: affinity mask[112]: 15
thread_masks[16]: 0x558fd7f36510: maps mask[112]: 16
thread_masks[16]: 0x558fd7f36520: affinity mask[112]: 16
thread_masks[17]: 0x558fd7f36530: maps mask[112]: 17
thread_masks[17]: 0x558fd7f36540: affinity mask[112]: 17
thread_masks[18]: 0x558fd7f36550: maps mask[112]: 18
thread_masks[18]: 0x558fd7f36560: affinity mask[112]: 18
thread_masks[19]: 0x558fd7f36570: maps mask[112]: 19
thread_masks[19]: 0x558fd7f36580: affinity mask[112]: 19
thread_masks[20]: 0x558fd7f36590: maps mask[112]: 20
thread_masks[20]: 0x558fd7f365a0: affinity mask[112]: 20
thread_masks[21]: 0x558fd7f365b0: maps mask[112]: 21
thread_masks[21]: 0x558fd7f365c0: affinity mask[112]: 21
thread_masks[22]: 0x558fd7f365d0: maps mask[112]: 22
thread_masks[22]: 0x558fd7f365e0: affinity mask[112]: 22
thread_masks[23]: 0x558fd7f365f0: maps mask[112]: 23
thread_masks[23]: 0x558fd7f36600: affinity mask[112]: 23
thread_masks[24]: 0x558fd7f36610: maps mask[112]: 24
thread_masks[24]: 0x558fd7f36620: affinity mask[112]: 24
thread_masks[25]: 0x558fd7f36630: maps mask[112]: 25
thread_masks[25]: 0x558fd7f36640: affinity mask[112]: 25
thread_masks[26]: 0x558fd7f36650: maps mask[112]: 26
thread_masks[26]: 0x558fd7f36660: affinity mask[112]: 26
thread_masks[27]: 0x558fd7f36670: maps mask[112]: 27
thread_masks[27]: 0x558fd7f36680: affinity mask[112]: 27
thread_masks[28]: 0x558fd7f36690: maps mask[112]: 28
thread_masks[28]: 0x558fd7f366a0: affinity mask[112]: 28
thread_masks[29]: 0x558fd7f366b0: maps mask[112]: 29
thread_masks[29]: 0x558fd7f366c0: affinity mask[112]: 29
thread_masks[30]: 0x558fd7f366d0: maps mask[112]: 30
thread_masks[30]: 0x558fd7f366e0: affinity mask[112]: 30
thread_masks[31]: 0x558fd7f366f0: maps mask[112]: 31
thread_masks[31]: 0x558fd7f36700: affinity mask[112]: 31
thread_masks[32]: 0x558fd7f36710: maps mask[112]: 32
thread_masks[32]: 0x558fd7f36720: affinity mask[112]: 32
thread_masks[33]: 0x558fd7f36730: maps mask[112]: 33
thread_masks[33]: 0x558fd7f36740: affinity mask[112]: 33
thread_masks[34]: 0x558fd7f36750: maps mask[112]: 34
thread_masks[34]: 0x558fd7f36760: affinity mask[112]: 34
thread_masks[35]: 0x558fd7f36770: maps mask[112]: 35
thread_masks[35]: 0x558fd7f36780: affinity mask[112]: 35
thread_masks[36]: 0x558fd7f36790: maps mask[112]: 36
thread_masks[36]: 0x558fd7f367a0: affinity mask[112]: 36
thread_masks[37]: 0x558fd7f367b0: maps mask[112]: 37
thread_masks[37]: 0x558fd7f367c0: affinity mask[112]: 37
thread_masks[38]: 0x558fd7f367d0: maps mask[112]: 38
thread_masks[38]: 0x558fd7f367e0: affinity mask[112]: 38
thread_masks[39]: 0x558fd7f367f0: maps mask[112]: 39
thread_masks[39]: 0x558fd7f36800: affinity mask[112]: 39
thread_masks[40]: 0x558fd7f36810: maps mask[112]: 40
thread_masks[40]: 0x558fd7f36820: affinity mask[112]: 40
thread_masks[41]: 0x558fd7f36830: maps mask[112]: 41
thread_masks[41]: 0x558fd7f36840: affinity mask[112]: 41
thread_masks[42]: 0x558fd7f36850: maps mask[112]: 42
thread_masks[42]: 0x558fd7f36860: affinity mask[112]: 42
thread_masks[43]: 0x558fd7f36870: maps mask[112]: 43
thread_masks[43]: 0x558fd7f36880: affinity mask[112]: 43
thread_masks[44]: 0x558fd7f36890: maps mask[112]: 44
thread_masks[44]: 0x558fd7f368a0: affinity mask[112]: 44
thread_masks[45]: 0x558fd7f368b0: maps mask[112]: 45
thread_masks[45]: 0x558fd7f368c0: affinity mask[112]: 45
thread_masks[46]: 0x558fd7f368d0: maps mask[112]: 46
thread_masks[46]: 0x558fd7f368e0: affinity mask[112]: 46
thread_masks[47]: 0x558fd7f368f0: maps mask[112]: 47
thread_masks[47]: 0x558fd7f36900: affinity mask[112]: 47
thread_masks[48]: 0x558fd7f36910: maps mask[112]: 48
thread_masks[48]: 0x558fd7f36920: affinity mask[112]: 48
thread_masks[49]: 0x558fd7f36930: maps mask[112]: 49
thread_masks[49]: 0x558fd7f36940: affinity mask[112]: 49
thread_masks[50]: 0x558fd7f36950: maps mask[112]: 50
thread_masks[50]: 0x558fd7f36960: affinity mask[112]: 50
thread_masks[51]: 0x558fd7f36970: maps mask[112]: 51
thread_masks[51]: 0x558fd7f36980: affinity mask[112]: 51
thread_masks[52]: 0x558fd7f36990: maps mask[112]: 52
thread_masks[52]: 0x558fd7f369a0: affinity mask[112]: 52
thread_masks[53]: 0x558fd7f369b0: maps mask[112]: 53
thread_masks[53]: 0x558fd7f369c0: affinity mask[112]: 53
thread_masks[54]: 0x558fd7f369d0: maps mask[112]: 54
thread_masks[54]: 0x558fd7f369e0: affinity mask[112]: 54
thread_masks[55]: 0x558fd7f369f0: maps mask[112]: 55
thread_masks[55]: 0x558fd7f36a00: affinity mask[112]: 55
thread_masks[56]: 0x558fd7f36a10: maps mask[112]: 56
thread_masks[56]: 0x558fd7f36a20: affinity mask[112]: 56
thread_masks[57]: 0x558fd7f36a30: maps mask[112]: 57
thread_masks[57]: 0x558fd7f36a40: affinity mask[112]: 57
thread_masks[58]: 0x558fd7f36a50: maps mask[112]: 58
thread_masks[58]: 0x558fd7f36a60: affinity mask[112]: 58
thread_masks[59]: 0x558fd7f36a70: maps mask[112]: 59
thread_masks[59]: 0x558fd7f36a80: affinity mask[112]: 59
thread_masks[60]: 0x558fd7f36a90: maps mask[112]: 60
thread_masks[60]: 0x558fd7f36aa0: affinity mask[112]: 60
thread_masks[61]: 0x558fd7f36ab0: maps mask[112]: 61
thread_masks[61]: 0x558fd7f36ac0: affinity mask[112]: 61
thread_masks[62]: 0x558fd7f36ad0: maps mask[112]: 62
thread_masks[62]: 0x558fd7f36ae0: affinity mask[112]: 62
thread_masks[63]: 0x558fd7f36af0: maps mask[112]: 63
thread_masks[63]: 0x558fd7f36b00: affinity mask[112]: 63
thread_masks[64]: 0x558fd7f36b10: maps mask[112]: 64
thread_masks[64]: 0x558fd7f36b20: affinity mask[112]: 64
thread_masks[65]: 0x558fd7f36b30: maps mask[112]: 65
thread_masks[65]: 0x558fd7f36b40: affinity mask[112]: 65
thread_masks[66]: 0x558fd7f36b50: maps mask[112]: 66
thread_masks[66]: 0x558fd7f36b60: affinity mask[112]: 66
thread_masks[67]: 0x558fd7f36b70: maps mask[112]: 67
thread_masks[67]: 0x558fd7f36b80: affinity mask[112]: 67
thread_masks[68]: 0x558fd7f36b90: maps mask[112]: 68
thread_masks[68]: 0x558fd7f36ba0: affinity mask[112]: 68
thread_masks[69]: 0x558fd7f36bb0: maps mask[112]: 69
thread_masks[69]: 0x558fd7f36bc0: affinity mask[112]: 69
thread_masks[70]: 0x558fd7f36bd0: maps mask[112]: 70
thread_masks[70]: 0x558fd7f36be0: affinity mask[112]: 70
thread_masks[71]: 0x558fd7f36bf0: maps mask[112]: 71
thread_masks[71]: 0x558fd7f36c00: affinity mask[112]: 71
thread_masks[72]: 0x558fd7f36c10: maps mask[112]: 72
thread_masks[72]: 0x558fd7f36c20: affinity mask[112]: 72
thread_masks[73]: 0x558fd7f36c30: maps mask[112]: 73
thread_masks[73]: 0x558fd7f36c40: affinity mask[112]: 73
thread_masks[74]: 0x558fd7f36c50: maps mask[112]: 74
thread_masks[74]: 0x558fd7f36c60: affinity mask[112]: 74
thread_masks[75]: 0x558fd7f36c70: maps mask[112]: 75
thread_masks[75]: 0x558fd7f36c80: affinity mask[112]: 75
thread_masks[76]: 0x558fd7f36c90: maps mask[112]: 76
thread_masks[76]: 0x558fd7f36ca0: affinity mask[112]: 76
thread_masks[77]: 0x558fd7f36cb0: maps mask[112]: 77
thread_masks[77]: 0x558fd7f36cc0: affinity mask[112]: 77
thread_masks[78]: 0x558fd7f36cd0: maps mask[112]: 78
thread_masks[78]: 0x558fd7f36ce0: affinity mask[112]: 78
thread_masks[79]: 0x558fd7f36cf0: maps mask[112]: 79
thread_masks[79]: 0x558fd7f36d00: affinity mask[112]: 79
thread_masks[80]: 0x558fd7f36d10: maps mask[112]: 80
thread_masks[80]: 0x558fd7f36d20: affinity mask[112]: 80
thread_masks[81]: 0x558fd7f36d30: maps mask[112]: 81
thread_masks[81]: 0x558fd7f36d40: affinity mask[112]: 81
thread_masks[82]: 0x558fd7f36d50: maps mask[112]: 82
thread_masks[82]: 0x558fd7f36d60: affinity mask[112]: 82
thread_masks[83]: 0x558fd7f36d70: maps mask[112]: 83
thread_masks[83]: 0x558fd7f36d80: affinity mask[112]: 83
thread_masks[84]: 0x558fd7f36d90: maps mask[112]: 84
thread_masks[84]: 0x558fd7f36da0: affinity mask[112]: 84
thread_masks[85]: 0x558fd7f36db0: maps mask[112]: 85
thread_masks[85]: 0x558fd7f36dc0: affinity mask[112]: 85
thread_masks[86]: 0x558fd7f36dd0: maps mask[112]: 86
thread_masks[86]: 0x558fd7f36de0: affinity mask[112]: 86
thread_masks[87]: 0x558fd7f36df0: maps mask[112]: 87
thread_masks[87]: 0x558fd7f36e00: affinity mask[112]: 87
thread_masks[88]: 0x558fd7f36e10: maps mask[112]: 88
thread_masks[88]: 0x558fd7f36e20: affinity mask[112]: 88
thread_masks[89]: 0x558fd7f36e30: maps mask[112]: 89
thread_masks[89]: 0x558fd7f36e40: affinity mask[112]: 89
thread_masks[90]: 0x558fd7f36e50: maps mask[112]: 90
thread_masks[90]: 0x558fd7f36e60: affinity mask[112]: 90
thread_masks[91]: 0x558fd7f36e70: maps mask[112]: 91
thread_masks[91]: 0x558fd7f36e80: affinity mask[112]: 91
thread_masks[92]: 0x558fd7f36e90: maps mask[112]: 92
thread_masks[92]: 0x558fd7f36ea0: affinity mask[112]: 92
thread_masks[93]: 0x558fd7f36eb0: maps mask[112]: 93
thread_masks[93]: 0x558fd7f36ec0: affinity mask[112]: 93
thread_masks[94]: 0x558fd7f36ed0: maps mask[112]: 94
thread_masks[94]: 0x558fd7f36ee0: affinity mask[112]: 94
thread_masks[95]: 0x558fd7f36ef0: maps mask[112]: 95
thread_masks[95]: 0x558fd7f36f00: affinity mask[112]: 95
thread_masks[96]: 0x558fd7f36f10: maps mask[112]: 96
thread_masks[96]: 0x558fd7f36f20: affinity mask[112]: 96
thread_masks[97]: 0x558fd7f36f30: maps mask[112]: 97
thread_masks[97]: 0x558fd7f36f40: affinity mask[112]: 97
thread_masks[98]: 0x558fd7f36f50: maps mask[112]: 98
thread_masks[98]: 0x558fd7f36f60: affinity mask[112]: 98
thread_masks[99]: 0x558fd7f36f70: maps mask[112]: 99
thread_masks[99]: 0x558fd7f36f80: affinity mask[112]: 99
thread_masks[100]: 0x558fd7f36f90: maps mask[112]: 100
thread_masks[100]: 0x558fd7f36fa0: affinity mask[112]: 100
thread_masks[101]: 0x558fd7f36fb0: maps mask[112]: 101
thread_masks[101]: 0x558fd7f36fc0: affinity mask[112]: 101
thread_masks[102]: 0x558fd7f36fd0: maps mask[112]: 102
thread_masks[102]: 0x558fd7f36fe0: affinity mask[112]: 102
thread_masks[103]: 0x558fd7f36ff0: maps mask[112]: 103
thread_masks[103]: 0x558fd7f37000: affinity mask[112]: 103
thread_masks[104]: 0x558fd7f37010: maps mask[112]: 104
thread_masks[104]: 0x558fd7f37020: affinity mask[112]: 104
thread_masks[105]: 0x558fd7f37030: maps mask[112]: 105
thread_masks[105]: 0x558fd7f37040: affinity mask[112]: 105
thread_masks[106]: 0x558fd7f37050: maps mask[112]: 106
thread_masks[106]: 0x558fd7f37060: affinity mask[112]: 106
thread_masks[107]: 0x558fd7f37070: maps mask[112]: 107
thread_masks[107]: 0x558fd7f37080: affinity mask[112]: 107
thread_masks[108]: 0x558fd7f37090: maps mask[112]: 108
thread_masks[108]: 0x558fd7f370a0: affinity mask[112]: 108
thread_masks[109]: 0x558fd7f370b0: maps mask[112]: 109
thread_masks[109]: 0x558fd7f370c0: affinity mask[112]: 109
thread_masks[110]: 0x558fd7f370d0: maps mask[112]: 110
thread_masks[110]: 0x558fd7f370e0: affinity mask[112]: 110
thread_masks[111]: 0x558fd7f370f0: maps mask[112]: 111
thread_masks[111]: 0x558fd7f37100: affinity mask[112]: 111
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 1
Problems creating module maps, continuing anyway...
mmap size 528384B
Failed to start threads: File exists
failed to write feature HYBRID_TOPOLOGY
failed to write feature HYBRID_CPU_PMU_CAPS
[ perf record: Captured and wrote 0.023 MB /tmp/logs/vtune/data.0 ]

vtune: Collection failed.
vtune: Internal Error

 

0 Kudos
6 Replies
JaideepK_Intel
Moderator
287 Views

Hi,

 

Thank you for posting in Intel Communities.

 

We ran hardware event-based sampling from our side on RHEL 8 and also in SLES 15.3 with the latest version of Vtune (2022.3) and it was working fine. Please go through the below screenshots.

Could you please try the latest version of vtune and let us know?

download the latest version of Vtune: Download the Intel® VTune™ Profiler

RHEL8:

JaideepK_Intel_0-1656512105796.jpeg

JaideepK_Intel_1-1656512115662.jpeg

SLES15.3:

JaideepK_Intel_2-1656512131336.jpeg

JaideepK_Intel_3-1656512142605.jpeg

 

If the issue still persists, please provide the below details.

1. Processor Information

2. Vtune2022.3 self-checker log (latest)

<Vtune_installation_directory\latest\bin64\vtune-self-checker.bat> for windows.
<Vtune_installation_directory\latest\bin64\vtune-self-checker.sh> for linux.

3. Sample reproducer 

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

 

Thanks,

Jaideep

 

emilberg
Beginner
266 Views

I tried 2022.3.0 and I had the same problem. Note that we are running the application in a container.

 

CPU: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz

 

Output from self checker is attached.

 

Sample reproducer (I guess any program will suffice):

 

 

#include <unistd.h>

int main() {
    while (1) {
        sleep(1);
    }
}

 

 

JaideepK_Intel
Moderator
252 Views

Hi,

 

I assume you are running the Vtune profiler and your application inside a Docker container. Please follow the below commands before starting a Docker container.

 

Run docker image with the below command.

 

docker run --rm -dti --cap-add CAP_SYS_ADMIN <docker_imagename>

 

To get the container id.

 

docker ps

 

To access the container run the below command.

 

docker exec -it <container id> bash

 

We ran hardware event-based sampling from our side inside a Docker container with the latest version of Vtune.

JaideepK_Intel_0-1656584828677.jpeg

 

JaideepK_Intel_1-1656584846780.jpeg

If you want to try container-to-container profiling, check the below link.

https://community.intel.com/t5/Analyzers/Vtune-Stops-the-container-when-generating-hotspots-result-f...

 

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

 

Thanks,

Jaideep

 

emilberg
Beginner
237 Views

I validated that we had the CAP_SYS_ADMIN capability already with

kubectl <pod> -- capsh --print

 

JaideepK_Intel
Moderator
168 Views

Hi,


We are working on this internally and will get back to you with an update.


Thanks,

Jaideep


emilberg
Beginner
97 Views

Please note that the

Failed to start threads: File exists

message may be a bug in perf. 

In perf, look at the changes to tools/perf/builtin-record.c made Feb 10 2022 with:

perf record: Start threads in the beginning of trace streaming

SHA: 3217e9fecf118d5dcabdd68d91e0c6afcb4c3e1b

 

At line 2014 pthread_create() is run and on line 2017 strerror(errno) is printed. Between line 2014 and 2017 record__terminate_thread() is run.

 

Most likely record__terminate_thread() run in-between the creation and the error message looks like it may overwrite errno, thus messing up the error message. 

 

if (pthread_create(&handle, &attrs, record__thread, &thread_data[t])) {
    for (tt = 1; tt < t; tt++)
        record__terminate_thread(&thread_data[t]);
    pr_err("Failed to start threads: %s\n", strerror(errno));

 

Just so you know the error message may be misleading.

Reply